# Distributed Troubleshooting

DevFeed: [Distributed Troubleshooting](<https://devfeed.tech/articles/distributed-troubleshooting-20407.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/distributed-troubleshooting>)

Author: Target Brands, Inc

Published: 2017-04-05T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [incident](<https://devfeed.tech/topics/incident.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>), [systems](<https://devfeed.tech/topics/systems.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [data](<https://devfeed.tech/tags/data.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [elk](<https://devfeed.tech/tags/elk.md>), [incident](<https://devfeed.tech/tags/incident.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-components](<https://devfeed.tech/tags/open-source-components.md>), [troubleshooting](<https://devfeed.tech/tags/troubleshooting.md>)

## AI overview

The article describes Target's Distributed Troubleshooting Platform for investigating issues across a large open source big data platform. It aggregates service logs and metrics so engineers can inspect information from many machines and services in one place, using open-source components and the Elasticsearch, Logstash, and Kibana stack.

## Source excerpt

Target's open source big data platform contains a vast array of clustered technologies or ecosystems working together. Troubleshooting an issue within a single ecosystem is a difficult task let alone an issue that spans several ecosystems. It is impractical for a single human to individually investigate ecosystems one at a time for potential problems. The house will burn to the ground long before an engineer can find the cause of an issue and resolve it without quick access to aggregated system metrics and logs. The Solution How to identify, troubleshoot and resolve a distributed issue? Fight fire with fire of course! Big data issues must be solved with big data solutions. At Target, we are constantly expanding our Distributed Troubleshooting Platform to encapsulate every log and metric from every service in every ecosystem of our big data platform. Aggregating this data into a single troubleshooting platform enables an engineer to view error logs and system metrics across hundreds of machines and services with a single click. A troubleshooting platform like the one described above is not a new idea. Systems like Splunk have been doing it for years. Splunk however, has restrictions on the amount of data that can be ingested without an enterprise license. The larger we scale; the more money we pay for systems like Splunk. We created our Distributed Troubleshooting Platform from open-source components and without enterprise licenses. This allows us to utilize it on every server in the big data platform without worrying about the volume of data it is processing and re-negotiating enterprise licenses. It becomes a given, not a variable. Our Distributed Troubleshooting Platform is similar to the black box recorder on an aircraft. A majority of the time, the contents are never viewed. When the plane crashes however, the contents of the black box are the only way to reconstruct what happened and learn from the incident. Running a big data platform without enterprise licens