# big-data Big data refers to extensive datasets characterized by volume, variety, velocity, and/or variability that require scalable architectures for efficient storage, manipulation, and analysis. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Segmented models for personal income estimation at Mercado Libre DevFeed: [Segmented models for personal income estimation at Mercado Libre]() Original publisher: [Read original article]() Author: Samuel Vazquez Published: 2026-01-08T20:44:23Z Content type: article Language: en Sources: [Mercado Libre Tech]() Topics: [data](), [personalization](), [datasets](), [big-data]() Tags: [big-data](), [business](), [classification](), [data](), [dataset](), [estimations](), [personal-income](), [personalization](), [regression]() ### AI overview Mercado Libre describes a "silver shotgun" approach for heterogeneous real-world data: segmenting data across contexts, training specialized models for each segment, and combining their predictions. The article presents personal income estimation as an application supporting more personalized financial products and discusses how geographic scale changes the importance of features such as home GPS location. ### Source excerpt Before we dive in, let's give a huge shout-out to Julio Avila Torreblanca and Alejandro Hernandez Cuevas for their exceptional work in developing this system. Thanks also to Enrique Miranda for reviewing the text and providing valuable feedback. Introduction Real-world data is rarely homogeneous. The same feature can mean different things depending on the city, neighborhood, urban density, or user segment. When we force a single global model to explain all those regimes, we usually pay with lower performance. At Mercado Libre, we take a different approach: instead of looking for a "silver bullet," we use a "silver shotgun." This pattern segments the data in several ways, trains specialized models per segment, and combines their predictions into a single estimate. Motivation: Financial inclusion is a significant opportunity In Mexico, only 63% of adults have a formal bank account, only 37% have a credit product, and less than 16% have a bank credit card [1]. About 45% of employed people have a formal source of income, while the rest work in the informal sector [2]. This gap presents a great opportunity for Mercado Libre to enter a growing market. Hyper-personalization is key to our mission to democratize financial inclusion across Latin America (LatAM). To offer great credit products, such as credit cards, we need a reliable estimate of a person's income. Big data, big problems Imagine we want to build a personal income estimation model for people visiting a very small local park. Every day, our dataset would grow by a few hundred entries for children playing, people exercising, and dog walkers, among other activities. Because people in the park live just a few blocks away, home location would have a limited effect (same neighborhood means low GPS variance). The features with the biggest impact on estimated income would likely be age and occupation. If we scale the problem to an entire city, home GPS location will have a bigger effect on the estimate. Each neighborho ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## Announcing Trino Summit 2024 DevFeed: [Announcing Trino Summit 2024]() Original publisher: [Read original article]() Author: Cole Bowden, Manfred Moser, and Monica Miller Published: 2024-07-11T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [big-data](), [data-engineering](), [Data Science](), [.NET 11]() Tags: [architectures](), [big-data](), [cloud](), [community](), [data-engineering](), [data-science](), [deployment](), [event](), [security](), [sign-up](), [speakers](), [sql](), [streaming](), [summit](), [videos](), [virtual-event]() ### AI overview Trino Summit 2024 is announced as a two-day virtual event on December 11 and 12, featuring talks for engineers, analysts, data scientists, and others interested in Trino. The article also opens a call for beginner to advanced speaker submissions on topics including data architectures, SQL analytics, data engineering, streaming, governance, security, deployment, and custom connectors. ### Source excerpt Fresh off the heels of Trino Fest 2024, where Commander Bun Bun was busy meeting the Trino community in-person, we're already looking forward to another, bigger event to round out the year in Trino. For those who've been here a while, you know that can only mean one thing: Trino Summit 2024. Much like last year, it will be a two-day, fully virtual event, hosting a wide range of talks covering all things Trino on the 11th and 12th of December. Read on for more info, or if you're already convinced... Register to attend! ## Expanded Memory and Compute with Heroku's New Larger Dynos DevFeed: [Expanded Memory and Compute with Heroku's New Larger Dynos]() Original publisher: [Read original article]() Author: Ethan Limchayseng Published: 2024-03-28T02:25:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [cpu](), [Cache](), [big-data](), [Data analysis]() Tags: [big-data](), [cache](), [cli](), [cloud-infrastructure](), [compute](), [cpu](), [data](), [distributed](), [dynos](), [hadoop](), [heroku](), [how-to](), [machine-learning](), [memory](), [news](), [performance-optimization](), [pricing](), [private-spaces](), [real-time](), [spark](), [video]() ### AI overview Heroku introduces nine larger dyno types across its Performance, Private, and Shield tiers, adding higher memory and CPU limits for compute-intensive workloads. The new sizes support use cases including real-time analytics, caching, machine learning, video encoding, and simulations. ### Source excerpt Introduction Heroku is excited to introduce nine new dyno types to our fleets and product offerings. In 2014, we introduced Performance-tier dynos, giving our customers fully dedicated resources to run their most compute-intensive workloads. Now in 2024, today's standards are rapidly increasing as complex applications and growing data volumes consume more memory and carry heavier [...] The post Expanded Memory and Compute with Heroku's New Larger Dynos appeared first on Heroku. ## Polish edition of Trino: The Definitive Guide DevFeed: [Polish edition of Trino: The Definitive Guide]() Original publisher: [Read original article]() Author: Manfred Moser, Martin Traverso, Matt Fuller Published: 2023-04-06T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [big-data](), [Maintainers]() Tags: [announce](), [big-data](), [book](), [community](), [contributors](), [java](), [maintainers](), [summit](), [translation]() ### AI overview The article announces that a Polish translation of the book Trino: The Definitive Guide is available for communities in Poland and beyond. It highlights Trino's global use and Poland's active Trino, Java, and Big Data communities, including contributors and maintainers. ### Source excerpt At this stage Trino is used all around the globe as we know from the community chat and our speakers at Trino Summit 2022. One large community of Trino contributors and maintainers, many employed by Starburst, is located in Poland. Poland also has a very active participation of developers and users in the Java and Big Data communities. ## Journey to Iceberg with Trino DevFeed: [Journey to Iceberg with Trino]() Original publisher: [Read original article]() Author: JaeChang Song, Jennifer Oh, Brian Olsen Published: 2022-12-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [data-engineering](), [big-data](), [dashboards](), [Hadoop]() Tags: [data-engineering](), [engineering](), [hadoop](), [hdfs](), [logs](), [metrics](), [open-source](), [performance](), [post](), [scale](), [speed](), [summit](), [switching]() ### AI overview SK Telecom describes its journey from Hive-based Trino deployments to Iceberg after encountering scaling and performance problems. The company used Trino across Hadoop and HDFS-based data platforms, collected query plans, JMX statistics, system metrics, and logs, and built a dashboard to investigate blocked queries and cluster behavior. ### Source excerpt This post comes from the second half of Trino Summit 2022 session. Our friends JaeChang and Jennifer from SK Telecom traveled across the globe from South Korea to join us in person! SK Telecom recently had some issues scaling Trino on the Hive model, among other issues that come with Hive. While some initial tweaking helped speed things up, it ultimately never solved the problem. After switching to Iceberg, SK Telecom ran initial performance tests with some very impressive results. In this talk, Jennifer and JaeChang describe their journey to Iceberg with Trino. ## Big Data Storage DevFeed: [Big Data Storage]() Original publisher: [Read original article]() Published: 2022-10-21T17:33:51Z Content type: article Language: en Sources: [Bert Hubert's writings]() Topics: [big-data](), [CSV](), [data](), [Data Infrastructure](), [Compression](), [vendor lock-in]() Tags: [big-data](), [compression](), [csv](), [data](), [data-infrastructure](), [programming](), [storage](), [vendor-lock-in]() ### AI overview This article examines storage and transfer formats for large-scale data. It presents CSV as easy to generate, widely supported, reasonably efficient, compressible, and resistant to vendor lock-in, while noting risks involving dialects, escaping, numeric precision, locale, field ordering, and inconsistent types. ### Source excerpt This is a page about some of the mechanics of 'big data', specifically how to store, transfer and process perhaps 100s of millions or billions of rows/events. Although some people think big data starts at 65k rows, at which point Excel used to lose it, I'm talking about gigabytes of data here. Infrastructure for this kind of thing abounds. Data can however come from lots of sources, and go to lots of different places. ## Trino Summit 2022: Federating humans and data DevFeed: [Trino Summit 2022: Federating humans and data]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2022-10-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [AI, ML & Data Engineering](), [big-data](), [datasets](), [Cloud](), [on-prem](), [Open Source](), [SQL](), [Apache Iceberg]() Tags: [architectures](), [big-data](), [blog](), [blog-post](), [book](), [cloud](), [community](), [conference](), [data](), [data-engineering](), [events](), [guide](), [java](), [on-prem](), [open-source](), [speakers](), [sql](), [storage](), [summit](), [us]() ### AI overview The article promotes Trino Summit 2022, whose central theme is query federation. It describes Trino as a query engine that exposes ANSI SQL across multiple data sources and highlights sessions on data mesh, data lakehouse, fault-tolerant execution, and large-scale Trino deployments. The event is presented as a free in-person and virtual conference in San Francisco on November 10, with limited in-person seating. ### Source excerpt Trino has long been the de facto standard to querying large data sets over your cloud or on-prem storage, also known as data lakes. This Trino Summit's theme instead will showcase Trino's other claim to fame: query federation. Trino is a query engine providing an access point that exposes ANSI SQL across multiple data sources. I urge you to join us either in-person or virtually if you are a fan of Trino, big data, open source, data engineering, Java, or all the above! This conference is free and takes place in San Francisco, California on November 10th. ## Happy tenth birthday Trino! DevFeed: [Happy tenth birthday Trino!]() Original publisher: [Read original article]() Author: Brian Olsen, Martin Traverso, Dain Sundstrom, David Phillips, Eric Hwang Published: 2022-08-08T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [big-data](), [SQL](), [data](), [GitHub](), [Slack](), [Development](), [Apache Iceberg]() Tags: [analytics](), [big-data](), [blog-post](), [community](), [data](), [distributed](), [github](), [pull-requests](), [releases](), [scale](), [slack](), [sql](), [youtube]() ### AI overview An anniversary article celebrating Trino's first ten years describes its origins as a community-driven fork of Presto, its role as a distributed big data SQL query engine, and its growth into a system that queries diverse data ecosystems through ANSI SQL. It also highlights learning resources, community milestones, and ways to participate. ### Source excerpt It's inspiring and mindblowing to reflect on the ten year journey that has produced the community around Trino. Trino is the community-driven fork from Presto, the distributed big data SQL query engine created at Facebook in 2012. We are a community of engineers, scientists, analysts, and visionaries that work in a fast paced world where the expectations on the time to insights from our analytics and the scale of the data are ever-increasing. Sometimes words only do so much justice to encompass a journey like this one, so we created a video to let you experience it yourself! Enjoy! ## Why leaving Facebook/Meta was the best thing we could do for the Trino Community DevFeed: [Why leaving Facebook/Meta was the best thing we could do for the Trino Community]() Original publisher: [Read original article]() Author: Martin Traverso, Dain Sundstrom, and David Phillips Published: 2022-08-02T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data analytics](), [big-data](), [Open Source](), [SQL](), [Meta]() Tags: [analytics](), [big-data](), [company](), [leaving](), [linkedin](), [meta](), [netflix](), [open-source](), [sql]() ### AI overview The article explains why leaving Facebook/Meta enabled the creators of Presto to build a broader open-source community around a distributed SQL query engine for big data analytics. It describes how Presto grew beyond Facebook, attracted participation from companies including Airbnb, Dropbox, Netflix, and LinkedIn, and served a wide range of analytics needs. ### Source excerpt It might surprise some that our departure from Facebook was one of the simplest decisions we've ever made. Many posts that discuss leaving a FAANG company focus on leaving some grand sum of money or prestige of working at the company. For us, we were leaving the company where we had launched a project that we knew would quickly outgrow the walls of Facebook, and solve a much larger set of problems in the analytics domain. At the time we didn't quite anticipate that Presto, a distributed SQL query engine for big data analytics, would be adopted around the globe by thousands of companies and an overwhelming number of industries. We appreciate Facebook for serving as the launchpad that inspired others to adopt Presto. Despite the harmonious beginnings, once the needs of the community and Facebook no longer aligned, we had to leave, but we'll get to that part shortly. ## Cinco de Trino recap: Learn how to build an efficient data lake DevFeed: [Cinco de Trino recap: Learn how to build an efficient data lake]() Original publisher: [Read original article]() Author: Brian Olsen, Brian Zhan Published: 2022-05-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [data-engineering](), [big-data](), [Query (disambiguation)](), [datasets](), [Apache Iceberg](), [Stack Overflow](), [X (Twitter)]() Tags: [analytics](), [architecture](), [big-data](), [build](), [clusters](), [community](), [conference](), [cost-savings](), [data](), [data-engineering](), [data-lake](), [etl](), [learn](), [pipelines](), [recap](), [recovery](), [scale](), [tools]() ### AI overview A recap of the Cinco de Trino conference covering Trino's role in data lakehouse architectures, interactive and federated querying, adaptive query planning, and fault-tolerant execution. It highlights Project Tardigrade for autoscaling, spot-instance use, cost savings, and failure recovery, plus a Starburst Galaxy lab for ingesting, cleaning, and analyzing Twitter and Stack Overflow data. ### Source excerpt When Trino (formerly PrestoSQL) arrived on the scene almost 10 years ago, it immediately became known as the much faster alternative to the data warehouse of big data, Apache Hive. The use cases that you, as the community, have built had far exceeded anything we had imagined in complexity. Together we've made Trino not only the fastest way to interactively query large data sets, but also a convenient way to run federated queries across data sources to make moving all the data optional. At Cinco de Trino, we came full circle back to the next iteration of analytics architecture with the data lake. This conference offers advice from industry thought leaders about how to use best lakehouse tools with Trino to manage that data complexity. Hear from industry thought leaders like Martin Traverso (Trino), Dain Sundstrom (Trino), James Campbell (Great Expectations), Jeremy Cohen (DBT Labs), Ryan Blue (Iceberg), Denny Lee (Delta Lake), Vinoth Chandar (Hudi). You can watch the talks on-demand on the Cinco de Trino playlist. In this post, I'd like to cover the key items from each talk you won't want to miss. ## SQL Server on Kubernetes Book Published! DevFeed: [SQL Server on Kubernetes Book Published!]() Original publisher: [Read original article]() Author: Anthony Nocentino Published: 2021-08-04T16:01:08Z Content type: release Language: en Sources: [Kubernetes on Anthony Nocentino's Blog]() Topics: [Kubernetes](), [sql-server](), [Azure](), [big-data]() Tags: [announce](), [azure](), [big-data](), [book](), [kubernetes](), [sql-server](), [training]() ### AI overview The authors announce the publication of SQL Server on Kubernetes - Designing and Building a Modern Data Platform. The book covers Kubernetes fundamentals, cluster management, and deploying and operationalizing SQL Server in on-premises and Azure environments, including high-availability options and Azure Arc-enabled Data Services. ### Source excerpt I'm super proud to announce that Ben E. Weissman and I have published SQL Server on Kubernetes - Designing and Building a Modern Data Platform available now at Apress and your favorite online book sellers! Buy the book now...or keep reading below if you need to be more convinced :) A couple of notes about the book, I saw Kubernetes showcased at a Microsoft event a few years back. I immediately saw the value and dedicated time to learning how Kubernetes and SQL Server on Kubernetes works. (Literally on the flight home I started watching training videos and researching books to read :) Since then, I have been sharing those learnings with the data and Kubernetes communities. This book is the intersection of all of that content. I genuinely believe that this will be how you will be building systems in the near future (or right now)! SQL Server on Kubernetes is the foundation for Big Data Clusters and now Azure Arc-enabled Data Services. I also want to call out my co-author and friend, Ben; you are an incredible writer and thank you for driving this book to completion! ## Обзор плагина Big Data Tools DevFeed: [Обзор плагина Big Data Tools]() Original publisher: [Read original article]() Author: olegchir (JetBrains) Published: 2021-07-28T10:41:36Z Content type: article Language: ru Sources: [JetBrains RU]() Topics: [ide](), [big-data](), [Hadoop](), [Apache Spark](), [data](), [Scala](), [Python]() Tags: [big-data](), [big-data-tools](), [data](), [hadoop](), [ide](), [jetbrains](), [kafka](), [python](), [scala](), [spark](), [tools](), [zeppelin]() ### AI overview This article reviews JetBrains' Big Data Tools plugin for working with cloud file systems, Hadoop, Spark, and Zeppelin directly from an IDE. It explains the plugin's role in data-engineering workflows, including ETL, and notes support for Scala and Python. ### Source excerpt Храните файлы в облачных файловых системах или, может быть, используете Hadoop, Spark и Zeppelin? А пробовали ли вы работать с ними напрямую из IDE? Привет, меня зовут Олег, я из команды плагина Big Data Tools. В этой статье мы поговорим, зачем этот плагин нужен, как применяется и где его достать. За последний год плагин прошёл большой путь и из экспериментального продукта превратился в боевое решение, на которое стоит взглянуть специалистам по Big Data. В JetBrains мы создаем IDE и другие инструменты, которые делают жизнь разработчиков лучше. Big Data Tools -- это очень узкоспециализированный, редкоземельный плагин, который предназначен для конкретного вида разработчиков -- для дата-инженеров. Если вам интересно подробней узнать о мире Big Data и работе дата-инженеров, рекомендую развернутую серию статей Паши Финкельштейна. Здесь мы рассмотрим одну из самых популярных схем. Читать далее ## Capturing Every Change From Shopify's Sharded Monolith DevFeed: [Capturing Every Change From Shopify's Sharded Monolith]() Original publisher: [Read original article]() Author: John Martin Published: 2021-03-12T18:51:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data-platforms](), [Shopify](), [Kafka](), [data](), [big-data](), [Open Source](), [Apache Spark](), [Databases](), [Cloud](), [parquet](), [Amazon Redshift](), [BigQuery]() Tags: [analytics](), [big-data](), [bigquery](), [cloud](), [cloud-storage](), [data](), [data-platforms](), [data-science](), [databases](), [kafka](), [latency](), [low-latency](), [open-source](), [parquet](), [pipelines](), [platforms](), [shopify](), [spark](), [streaming]() ### AI overview Shopify describes how its data architecture evolved from an internal warehouse into a cloud-based data lake and separate platforms for internal workloads and merchant analytics. The article covers batch extraction, streaming change events, Apache Spark and dbt transformations, Kafka-based pipelines, and serving data through systems including BigQuery, Redshift, and Google Bigtable. ### Source excerpt Shopify is moving from existing query-based CDC (Change-Data Capture) tooling (Longboat) to an immutable, append-only, log-based mechanism. This has required a complete re-platforming, involving a shift to new, open source technology such as Kafka Connect and Debezium. ## An overview of end-to-end entity resolution for big data DevFeed: [An overview of end-to-end entity resolution for big data]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-12-14T14:37:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Entity resolution](), [big-data](), [data]() Tags: [algorithms-and-data-structures](), [article](), [big-data](), [blocking](), [clustering](), [comparisons](), [overview](), [uncategorized]() ### AI overview This article summarizes an ACM Computing Surveys survey on end-to-end entity resolution for big data. It explains the main pipeline stages: blocking to reduce comparisons, block processing to remove redundant and superfluous comparisons, matching entity-description pairs, and clustering matches into resolved entities. It also outlines classification dimensions including schema awareness, matching process, and batch or incremental processing. ### Source excerpt An overview of end-to-end entity resolution for big data, Christophides et al., ACM Computing Surveys, Dec. 2020, Article No. 127 The ACM Computing Surveys are always a great way to get a quick orientation in a new subject area, and hot off the press is this survey on the entity resolution (aka record linking) problem. It's an ... Continue reading An overview of end-to-end entity resolution for big data ## Helios: hyperscale indexing for the cloud & edge - part 1 DevFeed: [Helios: hyperscale indexing for the cloud & edge - part 1]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-10-26T20:22:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [data-processing](), [big-data](), [Cloud](), [distributed-systems](), [data](), [Edge](), [Microsoft](), [client](), [servers]() Tags: [architecture](), [big-data](), [cloud](), [data](), [data-processing](), [datastores](), [distributed](), [distributed-computing](), [edge](), [idc](), [latency](), [microsoft](), [paper](), [real-time](), [server](), [streams](), [systems](), [uncategorized]() ### AI overview This first part of a discussion of Helios examines its role as a reference architecture for distributed big-data systems. It describes the pressures driving cloud platforms toward new approaches, including growing data volume, variable arrival rates, lower latency, reduced data-center costs, and stronger privacy considerations. ### Source excerpt Helios: hyperscale indexing for the cloud & edge, Potharaju et al., PVLDB'20 On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, ... Continue reading Helios: hyperscale indexing for the cloud & edge - part 1 ## Hello I'm Brian, Presto Developer Advocate DevFeed: [Hello I'm Brian, Presto Developer Advocate]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2020-10-01T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Open Source](), [data](), [datasets](), [big-data](), [Pull Request](), [Maintainers](), [elasticsearch]() Tags: [big-data](), [code](), [data](), [datasets](), [developer](), [elasticsearch](), [maintainers](), [open-source](), [pull-request](), [slack](), [testing]() ### AI overview A new developer advocate at Starburst describes his path from working with big data and databases to adopting Presto as an alternative to Hive. He explains contributing Elasticsearch array and nested type support, submitting a tested pull request, and helping the Presto community through communication and support. ### Source excerpt Hello, Presto nation! My name is Brian, and I'm a new developer advocate working at Starburst. Let me give you a little background on how I got here, and cover how my role can help the Presto community. ## Beyond LIMIT, Presto meets OFFSET and TIES DevFeed: [Beyond LIMIT, Presto meets OFFSET and TIES]() Original publisher: [Read original article]() Author: Piotr Findeisen, Starburst Data Published: 2020-02-03T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [SQL](), [Databases](), [big-data]() Tags: [big-data](), [relational-databases](), [sql]() ### AI overview The article explains how Presto added ANSI SQL support for limiting query results, including skipping initial rows with OFFSET and retaining all rows tied with the final result using TIES. It contrasts the standard syntax with Presto's existing shorthand and discusses its usefulness for ad-hoc queries over big data sets. ### Source excerpt Presto follows the SQL Standard faithfully. We extend it only when it is well justified, we strive to never break it and we always prefer the standard way of doing things. There was one situation where we stumbled, though. We had a non-standard way of limiting query results with LIMIT n without implementing the standard way of doing that first. We have corrected that, adding ANSI SQL way of limiting query results, discarding initial results and - a hidden gem - retaining initial results in case of ties. ## Improved Support for Replacing or Appending Data DevFeed: [Improved Support for Replacing or Appending Data]() Original publisher: [Read original article]() Author: Javier Álvarez Published: 2019-09-03T00:00:00Z Content type: release Language: en Sources: [Tinybird]() Topics: [data](), [big-data]() Tags: [big-data](), [consistency](), [data](), [product-updates](), [resiliency](), [support]() ### AI overview Tinybird Analytics emphasizes resiliency and consistency when handling frequent or big-data updates, in the context of improved support for replacing or appending data. ### Source excerpt One of the foundational ideas of Tinybird Analytics is resiliency and consistency under high frequency or big data updates. ## A review of the first international Presto Conference, Tel Aviv, April 2019 DevFeed: [A review of the first international Presto Conference, Tel Aviv, April 2019]() Original publisher: [Read original article]() Author: Ori Reshef, VP Product, Varada Published: 2019-05-03T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [ide](), [big-data](), [Software](), [data](), [dataset](), [Algorithm](), [Front end](), [parquet](), [SQL]() Tags: [analytics](), [big-data](), [community](), [conference](), [data](), [front-end](), [ide](), [open](), [parquet](), [review](), [sql](), [technology]() ### AI overview This article reviews the first international Presto Conference in Tel Aviv in April 2019. It covers presentations about the Presto Software Foundation, ORC improvements, pushdown algorithms, Varada's indexed analytics platform for data lakes, and Wix's open-sourced Quix IDE for data discovery, transformation, SQL queries, datasets, and ETL job scheduling. ### Source excerpt Community, noun: "A feeling of fellowship with others, as a result of sharing common attributes, interests, and goals" The fun picture you see here was taken at the first lecture of the First international Presto summit in Israel last month. The atmosphere in the room during the various presentations was unique. It's as if you could physically feel the brainpower of 250 engineers fascinated by technology in one room. We would like to share with you a bit of the content that was discussed during the conference. Enjoy the read and the videos! ## Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup DevFeed: [Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-17T17:02:27Z Content type: article Language: en Sources: [Yahoo]() Topics: [big-data](), [Open Source](), [recommendation systems]() Tags: [github](), [meetup](), [open-source](), [recommender-systems](), [yahoo-engineering]() ### AI overview A report on a Vespa presentation at the SF Big Analytics Meetup. It describes Vespa as an open-source big data processing and serving engine and gives Zedge as an example of its use for search, recommendation systems, and mobile content discovery. ### Source excerpt yahoodevelopers: By Jon Bratseth, Distinguished Architect, Oath I had the wonderful opportunity to present Vespa at the SF Big Analytics Meetup on September 26th, hosted by Amplitude. Several members of the Vespa team (Kim, Frode and Kristian) also attended. We all enjoyed meeting with members of the Big Analytics community to discuss how Vespa could be helpful for their companies. Thank you to Chester Chen, T.J. Bay, and Jin Hao Wan for planning the meetup, and here's our presentation, in case you missed it (slides are also available here): Largely developed by Yahoo engineers, Vespa is our big data processing and serving engine, available as open source on GitHub. It's in use by many products, such as Yahoo News, Yahoo Sports, Yahoo Finance and Oath Ads Platforms. Vespa use is growing even more rapidly; since it is open source under a permissive Apache license, Vespa can power other external third-party apps as well. A great example is Zedge, which uses Vespa for search and recommender systems to support content discovery for personalization of mobile phones (Android, iOS, and Web). Zedge uses Vespa in production to serve millions of monthly active users. Visit https://vespa.ai/ to learn more and download the code. We encourage code contributions and welcome opportunities to collaborate. ## Introducing Oak: an Open Source Scalable Key-Value Map for Big Data Analytics DevFeed: [Introducing Oak: an Open Source Scalable Key-Value Map for Big Data Analytics]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-09-13T16:04:58Z Content type: article Language: en Sources: [Yahoo]() Topics: [Open Source](), [Data structures](), [big-data](), [Scalability](), [data](), [real-time](), [Java](), [Programming](), [systems]() Tags: [big-data](), [data](), [data-structures](), [java](), [open-source](), [programming](), [real-time](), [scalability](), [yahoo-engineering]() ### AI overview The article introduces Oak, an open-source Java project from Oath that implements OakMap, a concurrent ordered key-value map. It presents Oak as a tool for scalable real-time analytics workloads and describes its relationship to concurrent data structures and Druid. ### Source excerpt yahoodevelopers: By Dmitry Basin, Edward Bortnikov, Anastasia Braginsky, Eshcar Hillel, Idit Keidar, Hagar Meir, Gali Sheffi Real-time analytics applications are on the rise. Modern decision support and machine intelligence engines strive to continuously ingest large volumes of data while providing up-to-date insights with minimum delay. For example, in Flurry Analytics, an Oath service which provides mobile developers with rich tools to explore user behavior in real time, it only takes seconds to reflect the events that happened on mobile devices in its numerous dashboards. The scalability demand is immense - as of late 2017, the Flurry SDK was installed on 2.6B devices and monitored 1M+ mobile apps. Mobile data hits the Flurry backend at a huge rate, updates statistics across hundreds of dimensions, and becomes queryable immediately. Flurry harnesses the open-source distributed interactive analytics engine named Druid to ingest data and serve queries at this massive rate. In order to minimize delays before data becomes available for analysis, technologies like Druid should avoid maintaining separate systems for data ingestion and query serving, and instead strive to do both within the same system. Doing so is nontrivial since one cannot compromise on overall correctness when multiple conflicting operations execute in parallel on modern multi-core CPUs. A promising approach is using concurrent data structure (CDS) algorithms which adapt traditional data structures to multiprocessor hardware. CDS implementations are thread-safe - that is, developers can use them exactly as sequential code while maintaining strong theoretical correctness guarantees. In recent years, CDS algorithms enabled dramatic application performance scaling and became popular programming tools. For example, Java programmers can use the ConcurrentNavigableMap JDK implementations for the concurrent ordered key-value map abstraction that is instrumental in systems like Druid. Today, we are excited ## REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack DevFeed: [REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-12-07T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Provisioning](), [Hadoop](), [openstack](), [Orchestration](), [Python](), [Open Source](), [Docker]() Tags: [big-data](), [big-data-infrastructure](), [chef](), [cloud](), [docker](), [druid](), [elasticsearch](), [hadoop](), [open-source](), [openstack](), [orchestration](), [provisioning](), [python]() ### AI overview REDstack is an open-source sandbox tool for Big Data development that provisions kerberized Hadoop clusters on OpenStack. It combines a cookbook for installing and configuring cluster components with a Python orchestration application that manages resource provisioning, Chef deployment, and component installation. ### Source excerpt REDstack is Now Open Source! We are officially open sourcing REDstack, our sandbox tool for Big Data development at Target. What is REDstack? REDstack is a tool for provisioning kerberized clusters on OpenStack. We created it with four goals in mind: Provide a secured environment, with the ability to leverage preconfigured LDAP and Kerberos servers. Out of the box usability, allowing you to log in with preconfigured user accounts. Custom user management utilities to administer the cluster. Provide a fully customizable experience, everything is a configuration option in your build files: Cluster size, node sizes, types of nodes and node roles, Hadoop configurations, heap sizes, and components, All users, passwords, and secure assets. Components REDstack is made up of two major components: hdp-cloud - The cookbook The cookbook is used by the application itself to install components and lay down cluster configuration. The cookbook can be used independently of REDstack to manually provision a cluster. REDstack - The orchestration component REDstack is a python application that performs all of the high-level complexities and timings associated with a full Hadoop installation: Orchestrates the provisioning of resources over OpenStack APIs, Controls and monitors parallel Chef deployment across the cluster, Manages and monitors cluster component install over HTTPS requests. REDstack is bundled with a Docker image, where the configs are set up locally before an installation, and all of the dependencies are updated and configured. How to Get Started Head over to the repository at https://github.com/target/redstack and follow along. The repo has instructions on how to build and configure the clusters using the included Docker image. History of the Project Target's Big Data Platform Team manages multiple Big Data environments, with hundreds of nodes and many PB's of data. As mentioned in our prior blog posts, we depend heavily on Chef as a core part of our CI/CD pipeline. Durin [Next page]()