# Uploading EC2 Logs to S3 on Shutdown

DevFeed: [Uploading EC2 Logs to S3 on Shutdown](<https://devfeed.tech/articles/uploading-ec2-logs-to-s3-on-shutdown-26526.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/09/04/ec2-log-upload.html>)

Published: 2018-09-04T08:56:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Bash](<https://devfeed.tech/topics/bash.md>), [Ubuntu](<https://devfeed.tech/topics/ubuntu.md>), [chmod](<https://devfeed.tech/topics/chmod.md>), [cURL](<https://devfeed.tech/topics/curl.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [bash](<https://devfeed.tech/tags/bash.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [chmod](<https://devfeed.tech/tags/chmod.md>), [curl](<https://devfeed.tech/tags/curl.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [s3](<https://devfeed.tech/tags/s3.md>), [script](<https://devfeed.tech/tags/script.md>), [ubuntu](<https://devfeed.tech/tags/ubuntu.md>)

## AI overview

This tutorial shows how to use a Linux shutdown script on Ubuntu EC2 instances to upload selected logs to Amazon S3 before an Auto Scaling Group terminates a failed instance. It is intended to preserve evidence for debugging, including errors from C++ code that may bypass JVM or logging services.

## Source excerpt

If you've ever used an auto scaling group (ASG) on AWS, you've probably had an EC2 instance fail and get removed from the ASG. While great for redundancy (the ASG launches a new instance to start handling requests), it makes debugging the failure difficult since the ASG terminates the bad instance, erasing any evidence of what went wrong. Below, I present a script that will upload relevant files to S3 after an instance is triggered to shutdown but before it terminates. To achieve this, we make use of Linux's runlevel scripts. The instructions below are for Ubuntu, but it should be straight forward to migrate to a different distro. First, we make a script /etc/rc0.d/K01upload-logs. This script will run when the system is shutting down. You should change BUCKET, PATH, and LOG_FILE to match your needs. #!/bin/bash source /etc/environment # get strict after sourcing environment since we don't trust it... set -euo pipefail IFS=$'\n\t' # what logs should I upload and where to? LOG_FILE="/var/log/tomcat7/catalina.out" BUCKET="my-logs-bucket" # below we include the instance id in the path. That way it's easily findable. HOST=$(/usr/bin/curl http://169.254.169.254/latest/meta-data/instance-id) PATH="services/logs/$HOST/" # upload the logs /bin/echo "Uploading logs to s3://$BUCKET/$PATH" | /usr/bin/wall /usr/local/bin/aws s3 cp $LOG_FILE s3://$BUCKET/$PATH wait After installing the script, you need to set the permissions: chown root:root /etc/rc0.d/K01upload-logs chmod +x /etc/rc0.d/K01upload-logs And that's it! The script will upload the logs to your S3 bucket when the ASG terminates an instance. We've found this extremely helpful for our deep learning infrastructure that can often contain errors from C++ code (and thus isn't handled by the jvm or sent to our logging services).