# logstash

Logstash is an open-source server-side data processing pipeline that ingests data from multiple sources, transforms it, and sends it to destinations.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How we improved reporting and monitoring of test automation results

DevFeed: [How we improved reporting and monitoring of test automation results](<https://devfeed.tech/articles/how-we-improved-reporting-and-monitoring-of-test-automation-results-28033.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2023-02-15-how-we-improved-reporting-and-monitoring-of-test-automation-results/>)

Author: Giuseppe Donati Web Test Automation Engineer Not a stereotypical Italian guy; Except

Published: 2023-02-15T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [GitHub Actions](<https://devfeed.tech/topics/github-actions.md>), [Selenium](<https://devfeed.tech/topics/selenium.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Cucumber](<https://devfeed.tech/topics/cucumber.md>)

Tags: [article](<https://devfeed.tech/tags/article.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [quality-assurance](<https://devfeed.tech/tags/quality-assurance.md>), [selenium](<https://devfeed.tech/tags/selenium.md>), [test-automation](<https://devfeed.tech/tags/test-automation.md>)

### AI overview

The article describes trivago's updated system for executing, reporting, and monitoring Selenium-based end-to-end tests. It covers GitHub Actions workflows, custom runners on Google Cloud, cloud report storage, Kafka and Logstash processing, Elasticsearch and Kibana visualization, and Grafana and Slack alerting.

### Source excerpt

Over the last few years, we completely refactored what was described in our previous article about how we use the ELK sta...

## Diagnosing an Elasticsearch field mapping explosion incident

DevFeed: [Diagnosing an Elasticsearch field mapping explosion incident](<https://devfeed.tech/articles/elasticwars-episode-iv-a-new-field-27993.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2020-06-03-elasticwarsepisodeivanewfield/>)

Author: Jorge Luis Betancourt Follow

Published: 2020-06-03T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [incident](<https://devfeed.tech/topics/incident.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [incident](<https://devfeed.tech/tags/incident.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>)

### AI overview

The article describes a production incident in which Logstash consumers fell behind, Elasticsearch ingestion stopped, and client nodes experienced memory pressure and long garbage-collection cycles. Monitoring showed rejected writes, saturated write threads, and queued operations, leading the team to identify a field mapping explosion as the underlying issue.

### Source excerpt

On a normal day, we ingest a lot of data into our ELK clusters (~6TB across all of our data centers). This is mostly operational data (logs) from different components in our infrastructure. This...

## Better Log Parsing with Logstash and Google Protocol Buffers

DevFeed: [Better Log Parsing with Logstash and Google Protocol Buffers](<https://devfeed.tech/articles/better-log-parsing-with-logstash-and-google-protocol-buffers-27939.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2016-01-19-logstash_protobuf_codec/>)

Author: Inga Feick Follow

Published: 2016-01-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [logstash](<https://devfeed.tech/topics/logstash.md>), [Zig](<https://devfeed.tech/topics/zig.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Development](<https://devfeed.tech/topics/development.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [data](<https://devfeed.tech/tags/data.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [google](<https://devfeed.tech/tags/google.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [parsing](<https://devfeed.tech/tags/parsing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

This tutorial explains how trivago uses Google Protocol Buffers with Kafka, Elasticsearch, and Logstash for log processing. It describes why protobuf can reduce message size and enforce a stable structure, then outlines creating and using a custom Logstash codec for decoding protobuf messages.

### Source excerpt

At trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process it from there into Elasticsearch using Logstash.

## Elasticsearch and Kibana for Selenium Automation

DevFeed: [Elasticsearch and Kibana for Selenium Automation](<https://devfeed.tech/articles/elasticsearch-and-kibana-for-selenium-automation-27935.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2015-12-02-selenium_with_kibana/>)

Author: Teodor Rupi Follow

Published: 2015-12-02T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [kibana](<https://devfeed.tech/topics/kibana.md>), [Selenium](<https://devfeed.tech/topics/selenium.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [ci](<https://devfeed.tech/topics/ci.md>), [Maven](<https://devfeed.tech/topics/maven.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [automation](<https://devfeed.tech/tags/automation.md>), [backend](<https://devfeed.tech/tags/backend.md>), [chrome](<https://devfeed.tech/tags/chrome.md>), [ci](<https://devfeed.tech/tags/ci.md>), [data-visualization](<https://devfeed.tech/tags/data-visualization.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [firefox](<https://devfeed.tech/tags/firefox.md>), [java](<https://devfeed.tech/tags/java.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [mac](<https://devfeed.tech/tags/mac.md>), [mac-os](<https://devfeed.tech/tags/mac-os.md>), [maven](<https://devfeed.tech/tags/maven.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [selenium](<https://devfeed.tech/tags/selenium.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

This article describes trivago's Selenium-based automated testing infrastructure and its use of Kibana for real-time reporting, filtering, and analysis of test results. It outlines a setup that uses Jenkins, Kafka, Logstash, Elasticsearch, and Kibana, with testing across multiple platforms and browsers.

### Source excerpt

The advances and growth of our Selenium based automated testing infrastructure generated an unexpected number of test results to evaluate. We had to rethink our reporting systems. Combining the power of Selenium with Kibana's graphing and filtering features totally changed our way of working.

## Log4j2 - dependencies

DevFeed: [Log4j2 - dependencies](<https://devfeed.tech/articles/log4j2-dependencies-31980.md>)

Original publisher: [Read original article](<https://tech.finn.no2014/10/27/log4j2-bringing-together-the-dependencies/>)

Author: mick

Published: 2014-10-27T11:50:00Z

Content type: tutorial

Language: en

Sources: [Finn.no](<https://devfeed.tech/sources/finn-no.md>)

Topics: [log4j](<https://devfeed.tech/topics/log4j.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [Library](<https://devfeed.tech/topics/library.md>)

Tags: [dependencies](<https://devfeed.tech/tags/dependencies.md>), [library](<https://devfeed.tech/tags/library.md>), [log4j](<https://devfeed.tech/tags/log4j.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logstash](<https://devfeed.tech/tags/logstash.md>)

### AI overview

The article describes bundling Log4j2, Logstash, and related logging dependencies into one library so codebases can consistently route logging abstractions through Log4j2 and use Logstash configuration.

### Source excerpt

To make it easy for all our codebases, to automatically have all logging abstractions pointing towards log4j2 along with logstash configured, we have bundled together the dependencies via one library. These dependencies look like: <!-- core libraries --> org.apache.logging.log4j:log4j-api:2.0.2 org.apache.logging.log4j:log4j-core:2.0.2 com.lmax:disruptor:3.2.1 <!-- commons-logging --> org.apache.logging.log4j:log4j-jcl:2.0.2 org.apache.logging.log4j:log4j-1.2-api:2.0.2 <!-- slf4j --> org.slf4j:slf4j-api:1.7.7 <!-- JUL routed through slf4j --> org.apache.logging.log4j:log4j-slf4j-impl:2.0.2 <!-- override old log4j with an empty jarfile --> <!-- incase it gets re-introduced transitively --> log4j:log4j:2-empty <!-- logstash --> net.logstash.log4j2:log4j2-logstash-jsonevent-layout:3.0.0-finn-2

## 10x: Logging at Clay.io

DevFeed: [10x: Logging at Clay.io](<https://devfeed.tech/articles/10x-logging-at-clay-io-35612.md>)

Original publisher: [Read original article](<https://zolmeister.com/2014/10/10x-logging-at-clay-io.html>)

Author: Zoli Kahan

Published: 2014-10-25T05:00:00Z

Content type: article

Language: en

Sources: [Zolmeister](<https://devfeed.tech/sources/zolmeister.md>)

Topics: [Logging](<https://devfeed.tech/topics/logging.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [logstash](<https://devfeed.tech/topics/logstash.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>), [Server](<https://devfeed.tech/topics/server.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Docker](<https://devfeed.tech/topics/docker.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-vpc](<https://devfeed.tech/tags/amazon-vpc.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [apply](<https://devfeed.tech/tags/apply.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [complex](<https://devfeed.tech/tags/complex.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [logstash](<https://devfeed.tech/tags/logstash.md>), [network](<https://devfeed.tech/tags/network.md>), [series](<https://devfeed.tech/tags/series.md>), [server](<https://devfeed.tech/tags/server.md>), [servers](<https://devfeed.tech/tags/servers.md>), [ssh](<https://devfeed.tech/tags/ssh.md>)

### AI overview

This article describes how Clay.io used Logstash to aggregate logs from more than 20 servers, with Elasticsearch and Kibana for analysis. It also discusses log rotation, securing Elasticsearch through Amazon VPC, and open-sourced Docker containers for deploying a distributed logging system.

### Source excerpt

10x: Logging at Clay.io Managing 20+ servers as a small team is no easy task, and when things go wrong (they always do) figuring out what happened quickly is essential. Of course we can't ssh into each machine, that would take ages, so instead we use Logstash to aggregate our logs. This is the second post in my series, and if you missed last episode: Architecture at Clay.io. Logstash overview Logstash deployments have two parts. The aggregate server (or cluster), and the client servers.