# aws-athena

Published articles for aws-athena.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A Serverless Data Engineering Stack at Teamwork Using Go and AWS

DevFeed: [A Serverless Data Engineering Stack at Teamwork Using Go and AWS](<https://devfeed.tech/articles/the-go-serverless-data-engineering-revolution-at-teamwork-golang-aws-35103.md>)

Original publisher: [Read original article](<https://engineroom.teamwork.com/the-go-serverless-data-engineering-revolution-at-teamwork-golang-aws-f2fd3cb1f563?source=rss----cea4eecd5960---4>)

Author: Joe Minichino

Published: 2021-10-19T13:58:52Z

Content type: article

Language: en

Sources: [Teamwork](<https://devfeed.tech/sources/teamwork.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Go Language](<https://devfeed.tech/topics/go-language.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [GitHub Actions](<https://devfeed.tech/topics/github-actions.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [airflow](<https://devfeed.tech/topics/airflow.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-athena](<https://devfeed.tech/tags/aws-athena.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [docker](<https://devfeed.tech/tags/docker.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [go](<https://devfeed.tech/tags/go.md>), [golang](<https://devfeed.tech/tags/golang.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [s3](<https://devfeed.tech/tags/s3.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-architecture](<https://devfeed.tech/tags/serverless-architecture.md>)

### AI overview

This article describes Teamwork's serverless-oriented data engineering stack and the reasons for choosing it. It discusses AWS services including S3, Athena, Glue, QuickSight, Kinesis, and Lambda, with Go used for Lambda processing and GitHub Actions used for deployment.

### Source excerpt

A real data lake. Traditional Data Engineering relies on products such as Airflow, Hadoop, Spark and Spark-based architectures, or similar technologies. These are still viable solutions for a number of reason, not least the fact that Data Engineers are few and far between, and the vast majority of them will be familiar in the above technologies or similar products/frameworks. Go Serverless I wrote an article about our tech stack, which includes S3, Athena, Glue, Quicksight, Kinesis etc. What is not immediately apparent is the serverless nature of our stack, which was a deliberate choice taken in the context of a Data Analytics department which was started as an experiment and had to pick its battles very wisely. Sysops, cluster / server management, CI/CD were not top of our list. Creating dashboards was. Also before Teamwork I had a nearly 2-year run working with a company that was entirely serverless in their set up AND mentality, and it was a career-changing experience. We worked prevalently with AWS Lambda when Lambda was a new toy, and we loved it. Finally, despite the "buzz" over Functional Programming being seemingly over, I am an arduous fanatic of it and of what it represents philosophically, a way to represent each problem in terms of an input, some transformation, an output and when possible, no side effects. NOTE: some of the tools I mention are not strictly serverless as much as they are fully managed (eg. Kinesis or Github Actions), but they still involve little or no sysops / devops. Serverless Processing: AWS Lambda Once upon a time you would have to settle for Node.js or Python to write Lambda code, but nowadays not only you can use a whole lot of runtimes, you can simply use your own docker image et voila', you're ready to go. In our case we are ready to Go, since at Teamwork Go(lang) is our programming language of choice. It's fast, easy to use, very safe and reliable, and given that it can complete the same operation faster than most other program