# Aggregating Apache logs with Fluentd and Amazon S3

DevFeed: [Aggregating Apache logs with Fluentd and Amazon S3](<https://devfeed.tech/articles/aggregating-apache-logs-with-fluentd-and-amazon-s3-19993.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2014/10/17/using-fluentd/>)

Published: 2014-10-17T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [servers](<https://devfeed.tech/topics/servers.md>), [data](<https://devfeed.tech/topics/data.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Ubuntu](<https://devfeed.tech/topics/ubuntu.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [cURL](<https://devfeed.tech/topics/curl.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [article](<https://devfeed.tech/tags/article.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [curl](<https://devfeed.tech/tags/curl.md>), [developer](<https://devfeed.tech/tags/developer.md>), [logs](<https://devfeed.tech/tags/logs.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [s3](<https://devfeed.tech/tags/s3.md>), [ubuntu](<https://devfeed.tech/tags/ubuntu.md>)

## AI overview

The article explains how HackerEarth aggregates Apache access and error logs from multiple web servers using Fluentd and stores the parsed, buffered data in Amazon S3. It covers installation with td-agent, configuration for Apache log formats, testing, and subsequent analysis with Elasticsearch and LogStash/Kibana.

## Source excerpt

HackerEarth infrastructure is hosted on Amazon services. At any given point of time many webservers are running concurrently serving thousands of requests. This generates tons of access and error logs on each server separately. The task here was to parse the logs on all these webservers and store them at one place in a format that can further be used to derive meaningful insights from the data. We tried to accomplish this using fluentd and Amazon S3. ####Mechanism Fluentd does the following things: Continuously tails apache log files. Parses incoming entries into meaning fields like ip, address etc and buffers them. Writes the buffered data to Amazon S3 periodically. ####Installation The stable version of fluentd is called td-agent and we are using the same for our purpose here. For ubuntu 12.04 LTS the following shell command will install td-agent on your system. curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh The other supported operating systems and installation methods are listed here. Please note that if you are installing ruby using Ruby Gems, you will have to install Amazon S3 output plugin separately. This can be done by: gem install fluent-plugin-s3 ####Configuration Once td-agent is installed you will find a td-agent.conf file in /etc/td-agent/ directory. For parsing apache access logs you will need to add the following configuration: <source> type tail # for continuosly tailing the log format apache2 # for default format of apache access logs time_format %d/%b/%Y:%H:%M:%S %z # time format in access logs path /var/log/apache2/access.log # path from where log is to be read # if td-agent restarts, it starts reading from the #last position td-agent read before the restart pos_file /var/log/td-agent/apache2.access_log.pos tag s3.apache.access # for identifying the log stream uniquely </source> <match s3.*.*> type s3 # plugin for writing the log to s3 aws_key_id <YOUR AWS KEY ID> aws_sec_key <YOUR AWS SECRET KEY> s3_bucket <