# A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage

DevFeed: [A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage](<https://devfeed.tech/articles/a-journey-towards-a-custom-data-warehouse-solution-part-2-we-need-storage-35108.md>)

Original publisher: [Read original article](<https://upday.github.io/blog/dwh-part2-we-need-storage/>)

Author: Robert Bordo (robert@upday.com)

Published: 2017-08-22T04:39:55Z

Content type: article

Language: en

Sources: [Upday](<https://devfeed.tech/sources/upday.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [storage](<https://devfeed.tech/tags/storage.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

This article examines storage choices for a custom data warehouse. It describes application log data, the limitations of time-series databases for additional master and historical data, business intelligence access needs, and the use of Amazon S3 as a data lake while considering other storage options including AWS Redshift.

## Source excerpt

In the beginning we created a cluster. And the cluster was without form, and void; and nulls were upon the face of the storage. As we learned in part 1 of our series, a data warehouse consists of several components. The key component is the storage. All the others group around it. But how can one draw a decision on which storage solution to adopt? What is out there anyway? Preface In a perfect world there would be only one kind of storage that fits all the needs of current DWH development and analysis. But since we are not living in that kind of place, we have several options. And the number of options increase the deeper one dives into the topic. There seem to be solutions for every use case you can think of. That might be a good starting point. What is our most common use case? What are we going to store? And how would we like to access our data in the end? Our major source is a massive amount of log data coming from our app. Everything the user does (e.g swiping through articles, selecting categories, leaving the app) is tracked, enriched with metadata (e.g. the user's location, app version, article identifier) and stored by a third-party service in big, semi-structured log files. Having only this source, a time series database like Graphite or InfluxDB could do the job. But also having slow changing master data, like user profiles, article metadata and maybe even to keep a history of data, this solution would not satisfy our current and future needs. Another thing that comes to my mind is how the data will be accessed by our final consumer (namely: Business Intelligence). Usually they use tools like Jasper Reports or Tableau for generating reports. For analyses we have to pre-aggregate the data to make queries more performant and translate raw information into a digestible format. What else is on the market? Storage good at bad at Example S3/Flat Files scalability, easy to use, data lake querying S3 Time Series DB handling time series data non time series data G