# Important Data Systems Problems Understudied by Database Research

DevFeed: [Important Data Systems Problems Understudied by Database Research](<https://devfeed.tech/articles/what-are-important-data-systems-problems-ignored-by-research-25088.md>)

Original publisher: [Read original article](<https://databasearchitects.blogspot.com/2024/12/what-are-important-data-systems.html>)

Author: Viktor Leis (noreply@blogger.com)

Published: 2024-12-13T07:37:00Z

Content type: opinion

Language: en

Sources: [Database Architects](<https://devfeed.tech/sources/database-architects.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Database](<https://devfeed.tech/topics/database.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [compression](<https://devfeed.tech/tags/compression.md>), [data](<https://devfeed.tech/tags/data.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [paper](<https://devfeed.tech/tags/paper.md>), [performance](<https://devfeed.tech/tags/performance.md>), [research](<https://devfeed.tech/tags/research.md>), [standard](<https://devfeed.tech/tags/standard.md>), [storage](<https://devfeed.tech/tags/storage.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

This discussion of database research priorities highlights variable-length string processing, database-specific string compression, unrealistic benchmarks, and the need for more representative analytical workloads. It also notes challenges in distributed query processing.

## Source excerpt

In November, I had the pleasure of attending the Dutch-Belgian DataBase Day, where I moderated a panel on practical challenges often overlooked in database research. Our distinguished panelists included Allison Lee (founding engineer at Snowflake), Andy Pavlo (professor at CMU), and Hannes Mühleisen (co-creator of DuckDB and researcher at CWI), with attendees contributing to the discussion and sharing their perspectives. In this post, I'll attempt to summarize the discussion in the hope that it inspires young (and young-at-heart) researchers to tackle these challenges. Additionally, I'll link to some paper that can serve as motivation and starting points for research in these areas. One significant yet understudied problem raised by multiple panellists is the handling of variable-length strings. Any analysis of real-world analytical queries reveals that strings are ubiquitous. For instance, Amazon Redshift recently reported that around 50% of all columns are strings. Since strings are typically larger than numeric data, this implies that strings are a substantial majority of real-world data. Dealing with strings presents two major challenges. First, query processing is often slow due to the variable size of strings and the (time and space) overhead of dynamic allocation. Second, surprisingly little research has been dedicated to efficient database-specific string compression. Given the importance of strings on real-world query performance and storage consumption, it is surprising how little research there is on the topic (there are some exceptions). Allison highlighted a related issue: standard benchmarks, like TPC-H, are overly simplistic, which may partly explain why string processing is understudied. TPC-H queries involve little complex string processing and don't use strings as join or aggregation keys. Moreover, TPC-H strings have static upper bounds, allowing them to be treated as fixed-size objects. This sidesteps the real challenges of variable-size strings