# logstash

Published articles for logstash.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How we improved reporting and monitoring of test automation results

DevFeed: [How we improved reporting and monitoring of test automation results](<https://devfeed.tech/articles/how-we-improved-reporting-and-monitoring-of-test-automation-results-28033.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2023-02-15-how-we-improved-reporting-and-monitoring-of-test-automation-results/>)

Author: Giuseppe Donati Web Test Automation Engineer Not a stereotypical Italian guy; Except

Published: 2023-02-15T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [GitHub Actions](<https://devfeed.tech/topics/github-actions.md>), [Selenium](<https://devfeed.tech/topics/selenium.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Cucumber](<https://devfeed.tech/topics/cucumber.md>)

Tags: [article](<https://devfeed.tech/tags/article.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [quality-assurance](<https://devfeed.tech/tags/quality-assurance.md>), [selenium](<https://devfeed.tech/tags/selenium.md>), [test-automation](<https://devfeed.tech/tags/test-automation.md>)

### AI overview

The article describes trivago's updated system for executing, reporting, and monitoring Selenium-based end-to-end tests. It covers GitHub Actions workflows, custom runners on Google Cloud, cloud report storage, Kafka and Logstash processing, Elasticsearch and Kibana visualization, and Grafana and Slack alerting.

### Source excerpt

Over the last few years, we completely refactored what was described in our previous article about how we use the ELK sta...

## Diagnosing an Elasticsearch field mapping explosion incident

DevFeed: [Diagnosing an Elasticsearch field mapping explosion incident](<https://devfeed.tech/articles/elasticwars-episode-iv-a-new-field-27993.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2020-06-03-elasticwarsepisodeivanewfield/>)

Author: Jorge Luis Betancourt Follow

Published: 2020-06-03T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [incident](<https://devfeed.tech/topics/incident.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [incident](<https://devfeed.tech/tags/incident.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>)

### AI overview

The article describes a production incident in which Logstash consumers fell behind, Elasticsearch ingestion stopped, and client nodes experienced memory pressure and long garbage-collection cycles. Monitoring showed rejected writes, saturated write threads, and queued operations, leading the team to identify a field mapping explosion as the underlying issue.

### Source excerpt

On a normal day, we ingest a lot of data into our ELK clusters (~6TB across all of our data centers). This is mostly operational data (logs) from different components in our infrastructure. This...

## Better URL Search with Elasticsearch

DevFeed: [Better URL Search with Elasticsearch](<https://devfeed.tech/articles/better-url-search-with-elasticsearch-27985.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2020-02-11-betterurlsearchwithelasticsearch/>)

Author: Jorge Luis Betancourt Follow

Published: 2020-02-11T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>)

Tags: [ascii](<https://devfeed.tech/tags/ascii.md>), [cardinality](<https://devfeed.tech/tags/cardinality.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

This article explains how trivago searches URL components stored in Elasticsearch within an ELK-based logging pipeline. It examines how the Standard Analyzer tokenizes URL-like query strings and discusses preprocessing and flattened-field approaches for matching query-parameter key/value pairs, including their tradeoffs.

### Source excerpt

At trivago, we generate a huge amount of logs and we have our own custom setup for shipping logs using mostly

## SAML SSO Authentication for Splunk with G Suite

DevFeed: [SAML SSO Authentication for Splunk with G Suite](<https://devfeed.tech/articles/saml-sso-authentication-for-splunk-with-g-suite-27895.md>)

Original publisher: [Read original article](<https://clevertap.com/blog/saml-sso-authentication-for-splunk-with-g-suite/>)

Author: kishlaya kumar

Published: 2018-04-12T07:00:30Z

Content type: tutorial

Language: en

Sources: [CleverTap](<https://devfeed.tech/sources/clevertap.md>)

Topics: [saml](<https://devfeed.tech/topics/saml.md>), [Single sign-on (SSO)](<https://devfeed.tech/topics/sso.md>), [Authentication](<https://devfeed.tech/topics/authentication.md>), [Google](<https://devfeed.tech/topics/google.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [logstash](<https://devfeed.tech/topics/logstash.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [authorization](<https://devfeed.tech/tags/authorization.md>), [browser](<https://devfeed.tech/tags/browser.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [elk](<https://devfeed.tech/tags/elk.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [google](<https://devfeed.tech/tags/google.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [log-management](<https://devfeed.tech/tags/log-management.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [saml](<https://devfeed.tech/tags/saml.md>), [security](<https://devfeed.tech/tags/security.md>), [sso](<https://devfeed.tech/tags/sso.md>), [technology](<https://devfeed.tech/tags/technology.md>), [technology-engineering](<https://devfeed.tech/tags/technology-engineering.md>)

### AI overview

A step-by-step guide to configuring SAML-based single sign-on for Splunk using Google (G Suite) as the identity provider. It explains the SAML authentication model and flow, following the authors' move from ELK to Splunk for on-premises log management and analytics.

### Source excerpt

From early on, our team used ELK (Elasticsearch-Logstash-Kibana) for log management and analytics. ELK served us well, but as our The post SAML SSO Authentication for Splunk with G Suite first appeared on CleverTap.

## Distributed Troubleshooting

DevFeed: [Distributed Troubleshooting](<https://devfeed.tech/articles/distributed-troubleshooting-20407.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/distributed-troubleshooting>)

Author: Target Brands, Inc

Published: 2017-04-05T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [incident](<https://devfeed.tech/topics/incident.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>), [systems](<https://devfeed.tech/topics/systems.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [data](<https://devfeed.tech/tags/data.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [elk](<https://devfeed.tech/tags/elk.md>), [incident](<https://devfeed.tech/tags/incident.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-components](<https://devfeed.tech/tags/open-source-components.md>), [troubleshooting](<https://devfeed.tech/tags/troubleshooting.md>)

### AI overview

The article describes Target's Distributed Troubleshooting Platform for investigating issues across a large open source big data platform. It aggregates service logs and metrics so engineers can inspect information from many machines and services in one place, using open-source components and the Elasticsearch, Logstash, and Kibana stack.

### Source excerpt

Target's open source big data platform contains a vast array of clustered technologies or ecosystems working together. Troubleshooting an issue within a single ecosystem is a difficult task let alone an issue that spans several ecosystems. It is impractical for a single human to individually investigate ecosystems one at a time for potential problems. The house will burn to the ground long before an engineer can find the cause of an issue and resolve it without quick access to aggregated system metrics and logs. The Solution How to identify, troubleshoot and resolve a distributed issue? Fight fire with fire of course! Big data issues must be solved with big data solutions. At Target, we are constantly expanding our Distributed Troubleshooting Platform to encapsulate every log and metric from every service in every ecosystem of our big data platform. Aggregating this data into a single troubleshooting platform enables an engineer to view error logs and system metrics across hundreds of machines and services with a single click. A troubleshooting platform like the one described above is not a new idea. Systems like Splunk have been doing it for years. Splunk however, has restrictions on the amount of data that can be ingested without an enterprise license. The larger we scale; the more money we pay for systems like Splunk. We created our Distributed Troubleshooting Platform from open-source components and without enterprise licenses. This allows us to utilize it on every server in the big data platform without worrying about the volume of data it is processing and re-negotiating enterprise licenses. It becomes a given, not a variable. Our Distributed Troubleshooting Platform is similar to the black box recorder on an aircraft. A majority of the time, the contents are never viewed. When the plane crashes however, the contents of the black box are the only way to reconstruct what happened and learn from the incident. Running a big data platform without enterprise licens

## Better Log Parsing with Logstash and Google Protocol Buffers

DevFeed: [Better Log Parsing with Logstash and Google Protocol Buffers](<https://devfeed.tech/articles/better-log-parsing-with-logstash-and-google-protocol-buffers-27939.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2016-01-19-logstash_protobuf_codec/>)

Author: Inga Feick Follow

Published: 2016-01-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [logstash](<https://devfeed.tech/topics/logstash.md>), [Zig](<https://devfeed.tech/topics/zig.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Development](<https://devfeed.tech/topics/development.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [data](<https://devfeed.tech/tags/data.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [google](<https://devfeed.tech/tags/google.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [parsing](<https://devfeed.tech/tags/parsing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

This tutorial explains how trivago uses Google Protocol Buffers with Kafka, Elasticsearch, and Logstash for log processing. It describes why protobuf can reduce message size and enforce a stable structure, then outlines creating and using a custom Logstash codec for decoding protobuf messages.

### Source excerpt

At trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process it from there into Elasticsearch using Logstash.

## Elasticsearch and Kibana for Selenium Automation

DevFeed: [Elasticsearch and Kibana for Selenium Automation](<https://devfeed.tech/articles/elasticsearch-and-kibana-for-selenium-automation-27935.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2015-12-02-selenium_with_kibana/>)

Author: Teodor Rupi Follow

Published: 2015-12-02T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [kibana](<https://devfeed.tech/topics/kibana.md>), [Selenium](<https://devfeed.tech/topics/selenium.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [ci](<https://devfeed.tech/topics/ci.md>), [Maven](<https://devfeed.tech/topics/maven.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [automation](<https://devfeed.tech/tags/automation.md>), [backend](<https://devfeed.tech/tags/backend.md>), [chrome](<https://devfeed.tech/tags/chrome.md>), [ci](<https://devfeed.tech/tags/ci.md>), [data-visualization](<https://devfeed.tech/tags/data-visualization.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [firefox](<https://devfeed.tech/tags/firefox.md>), [java](<https://devfeed.tech/tags/java.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [mac](<https://devfeed.tech/tags/mac.md>), [mac-os](<https://devfeed.tech/tags/mac-os.md>), [maven](<https://devfeed.tech/tags/maven.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [selenium](<https://devfeed.tech/tags/selenium.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

This article describes trivago's Selenium-based automated testing infrastructure and its use of Kibana for real-time reporting, filtering, and analysis of test results. It outlines a setup that uses Jenkins, Kafka, Logstash, Elasticsearch, and Kibana, with testing across multiple platforms and browsers.

### Source excerpt

The advances and growth of our Selenium based automated testing infrastructure generated an unexpected number of test results to evaluate. We had to rethink our reporting systems. Combining the power of Selenium with Kibana's graphing and filtering features totally changed our way of working.

## Introducing Gollum: A NxM message multiplexer written in Go

DevFeed: [Introducing Gollum: A NxM message multiplexer written in Go](<https://devfeed.tech/articles/introducing-gollum-a-nxm-message-multiplexer-written-in-go-27930.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2015-06-22-gollum/>)

Author: Arne Claus Follow

Published: 2015-06-22T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [Go Language](<https://devfeed.tech/topics/go-language.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [logstash](<https://devfeed.tech/topics/logstash.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [backend](<https://devfeed.tech/tags/backend.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [github](<https://devfeed.tech/tags/github.md>), [go](<https://devfeed.tech/tags/go.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [services](<https://devfeed.tech/tags/services.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

Trivago introduces Gollum, a Go-based message multiplexer and routing framework. It routes messages from multiple sources to different services and supports a redesigned logging pipeline using Kafka and background workers that can push data to systems such as Elasticsearch.

### Source excerpt

Here at trivago we write a huge number of log messages every day that need to be stored and monitored. To handle all these messages we created Gollum, a tool that enables us to conveniently send messages from multiple sources to different services.

## Log4j2 - dependencies

DevFeed: [Log4j2 - dependencies](<https://devfeed.tech/articles/log4j2-dependencies-31980.md>)

Original publisher: [Read original article](<https://tech.finn.no2014/10/27/log4j2-bringing-together-the-dependencies/>)

Author: mick

Published: 2014-10-27T11:50:00Z

Content type: tutorial

Language: en

Sources: [Finn.no](<https://devfeed.tech/sources/finn-no.md>)

Topics: [log4j](<https://devfeed.tech/topics/log4j.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Library](<https://devfeed.tech/topics/library.md>)

Tags: [dependencies](<https://devfeed.tech/tags/dependencies.md>), [library](<https://devfeed.tech/tags/library.md>), [log4j](<https://devfeed.tech/tags/log4j.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logstash](<https://devfeed.tech/tags/logstash.md>)

### AI overview

The article describes bundling Log4j2, Logstash, and related logging dependencies into one library so codebases can consistently route logging abstractions through Log4j2 and use Logstash configuration.

### Source excerpt

To make it easy for all our codebases, to automatically have all logging abstractions pointing towards log4j2 along with logstash configured, we have bundled together the dependencies via one library. These dependencies look like: <!-- core libraries --> org.apache.logging.log4j:log4j-api:2.0.2 org.apache.logging.log4j:log4j-core:2.0.2 com.lmax:disruptor:3.2.1 <!-- commons-logging --> org.apache.logging.log4j:log4j-jcl:2.0.2 org.apache.logging.log4j:log4j-1.2-api:2.0.2 <!-- slf4j --> org.slf4j:slf4j-api:1.7.7 <!-- JUL routed through slf4j --> org.apache.logging.log4j:log4j-slf4j-impl:2.0.2 <!-- override old log4j with an empty jarfile --> <!-- incase it gets re-introduced transitively --> log4j:log4j:2-empty <!-- logstash --> net.logstash.log4j2:log4j2-logstash-jsonevent-layout:3.0.0-finn-2

## 10x: Logging at Clay.io

DevFeed: [10x: Logging at Clay.io](<https://devfeed.tech/articles/10x-logging-at-clay-io-35612.md>)

Original publisher: [Read original article](<https://zolmeister.com/2014/10/10x-logging-at-clay-io.html>)

Author: Zoli Kahan

Published: 2014-10-25T05:00:00Z

Content type: article

Language: en

Sources: [Zolmeister](<https://devfeed.tech/sources/zolmeister.md>)

Topics: [Logging](<https://devfeed.tech/topics/logging.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>), [Server](<https://devfeed.tech/topics/server.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Docker](<https://devfeed.tech/topics/docker.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-vpc](<https://devfeed.tech/tags/amazon-vpc.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [apply](<https://devfeed.tech/tags/apply.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [complex](<https://devfeed.tech/tags/complex.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [network](<https://devfeed.tech/tags/network.md>), [series](<https://devfeed.tech/tags/series.md>), [server](<https://devfeed.tech/tags/server.md>), [servers](<https://devfeed.tech/tags/servers.md>), [ssh](<https://devfeed.tech/tags/ssh.md>)

### AI overview

This article describes how Clay.io used Logstash to aggregate logs from more than 20 servers, with Elasticsearch and Kibana for analysis. It also discusses log rotation, securing Elasticsearch through Amazon VPC, and open-sourced Docker containers for deploying a distributed logging system.

### Source excerpt

10x: Logging at Clay.io Managing 20+ servers as a small team is no easy task, and when things go wrong (they always do) figuring out what happened quickly is essential. Of course we can't ssh into each machine, that would take ages, so instead we use Logstash to aggregate our logs. This is the second post in my series, and if you missed last episode: Architecture at Clay.io. Logstash overview Logstash deployments have two parts. The aggregate server (or cluster), and the client servers.