# Target and Elasticsearch: Maintaining an ELK stack over Peak Season

DevFeed: [Target and Elasticsearch: Maintaining an ELK stack over Peak Season](<https://devfeed.tech/articles/target-and-elasticsearch-maintaining-an-elk-stack-over-peak-season-20410.md>)

Original publisher: [Read original article](<https://target.github.io/logging%20and%20metrics/elasticsearch-cloud>)

Author: Target Brands, Inc

Published: 2017-05-25T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [consul](<https://devfeed.tech/tags/consul.md>), [contribute](<https://devfeed.tech/tags/contribute.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [elk](<https://devfeed.tech/tags/elk.md>), [hashicorp](<https://devfeed.tech/tags/hashicorp.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logging-and-metrics](<https://devfeed.tech/tags/logging-and-metrics.md>), [logs](<https://devfeed.tech/tags/logs.md>), [make](<https://devfeed.tech/tags/make.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [scalability](<https://devfeed.tech/tags/scalability.md>)

## AI overview

This article describes how Target operated and evolved an Elasticsearch-based ELK stack for large-scale log aggregation, search, analytics, and multi-tenant logging during peak production periods. It discusses cloud scalability, operational challenges, and the open-source tools used alongside Elasticsearch.

## Source excerpt

One of the strongest benefits of launching an application into the cloud is the pure on-demand scalability that it provides. I've had the privilege of working with the ELK stack (Elasticsearch, Logstash, Kibana) for purposes of log aggregation for the past two years. When we started at that time, we were pleased with our performance on search and query times with 10's of gigabytes of data in the cluster in production. When Peak time hit, we reveled as our production clusters successfully managed half a terabyte of data(!). During peak, Target hosted 14 Elasticsearch clusters in the cloud containing more than 83 billion documents across nearly 100 terabytes in production environments alone. Consumers of these logs are able to get access to queries in blazing fast times with excellent reliability. It wasn't always that way though, and our team learned much about Elasticsearch in the process. What's The Use Case At Target? In a word, "vast." The many teams that use our platform for log aggregation and search are often times looking for a variety of things. Simple Search This one is easy, and the least resource intensive. Simply doing a match query and searching for fields within our data. Metrics / Analytics This one can be harder to accommodate at times, but some teams use our Elasticsearch clusters for near-realtime monitoring and Analytics using Kibana dashboards. Multi-tenant Logging Not necessarily consumer facing, but an interesting use for Elasticsearch is that we can aggregate many teams and applications into one cluster. In essence, this saves money over individual applications paying for infrastructure to log themselves. Simple search is the least of our concerns here. Queries add marginal load on the cluster, but often they are one-offs or otherwise infrequently used. However, the largest challenge faced here is multi-tenant demand. Different teams have very different needs for logging/metrics; designing a robust and reliable 'one-size-fits-all' platform is