# Big Data Storage

DevFeed: [Big Data Storage](<https://devfeed.tech/articles/big-data-storage-36266.md>)

Original publisher: [Read original article](<https://berthub.eu/articles/posts/big-data-storage/>)

Published: 2022-10-21T17:33:51Z

Content type: article

Language: en

Sources: [Bert Hubert's writings](<https://devfeed.tech/sources/bert-hubert-s-writings.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [CSV](<https://devfeed.tech/topics/csv.md>), [data](<https://devfeed.tech/topics/data.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [vendor lock-in](<https://devfeed.tech/topics/vendor-lock-in.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [compression](<https://devfeed.tech/tags/compression.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [programming](<https://devfeed.tech/tags/programming.md>), [storage](<https://devfeed.tech/tags/storage.md>), [vendor-lock-in](<https://devfeed.tech/tags/vendor-lock-in.md>)

## AI overview

This article examines storage and transfer formats for large-scale data. It presents CSV as easy to generate, widely supported, reasonably efficient, compressible, and resistant to vendor lock-in, while noting risks involving dialects, escaping, numeric precision, locale, field ordering, and inconsistent types.

## Source excerpt

This is a page about some of the mechanics of 'big data', specifically how to store, transfer and process perhaps 100s of millions or billions of rows/events. Although some people think big data starts at 65k rows, at which point Excel used to lose it, I'm talking about gigabytes of data here. Infrastructure for this kind of thing abounds. Data can however come from lots of sources, and go to lots of different places.