# big-data Published articles for big-data. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## AI, JD, and other letters of the law DevFeed: [AI, JD, and other letters of the law]() Original publisher: [Read original article]() Author: Phoebe Sajor Published: 2026-09-15T07:40:00Z Content type: article Language: en Sources: [Stack Overflow Blog]() Topics: [Artificial Intelligence](), [AI Strategy](), [data centers]() Tags: [agentic-ai](), [ai](), [big-data](), [data-centers](), [law](), [podcast](), [policy](), [se-stackoverflow](), [se-tech](), [university]() ### AI overview Ryan chats with Kevin Frazier about the legal and social impacts of data centers, workforce disruption, and regulating AI for child safety through existing consumer protection laws. ### Source excerpt Ryan chats with Kevin Frazier, director of the AI Innovation and Law program at the University of Texas School of Law, about the legal and social impacts of data centers, the realities of workforce disruption, and regulating AI for child safety using existing consumer protection laws. ## How INTEGER and INT Produced Different Schemas in Debezium DevFeed: [How INTEGER and INT Produced Different Schemas in Debezium]() Original publisher: [Read original article]() Author: Prathit Malik Published: 2026-09-02T07:02:02Z Content type: article Language: en Sources: [Grofers]() Topics: [MySQL](), [migration](), [Streaming](), [Kafka](), [SQL](), [Databases]() Tags: [big-data](), [blinkit](), [change](), [database](), [debezium](), [jdbc](), [kafka](), [migration](), [mysql](), [open-source](), [schema](), [sql](), [streaming]() ### AI overview This article investigates a Debezium CDC pipeline failure caused by MySQL INT and INTEGER synonyms being treated as different types. The resulting schema mismatch produced an Integer where a downstream consumer expected a Long, causing a ClassCastException on every batch. It explains how streaming and snapshot schema handling differ and how routine migrations exposed the problem. ### Source excerpt A Debezium investigation: how MySQL synonyms INT and INTEGER were treated as different types. One of our CDC pipelines started failing with a ClassCastException on every batch. java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long The pipeline was producing an Integer, but the downstream consumer expected a Long. Every run failed in the same way, which pointed us toward a schema mismatch rather than an issue with individual records. Background: how CDC works To see why a mismatch like that can hide for years, it helps to know how CDC actually works. Most companies replicate their transactional database (MySQL, Postgres, something similar) into a separate data lake for analytics, rather than querying the source directly, and Change Data Capture (CDC) is what keeps that copy in sync: it tails the database's transaction log and replays every insert, update, and delete downstream. Debezium is the most widely used open-source CDC tool for MySQL, and it builds a table's schema in one of two ways that are supposed to agree but do not always. Streaming mode: the first time it sees a CREATE TABLE or ALTER TABLE in the binlog, it parses the raw SQL text and writes the result to its own internal Kafka topic, database.history.kafka.topic. Every restart after that rebuilds the in-memory schema by replaying that topic, not by re-reading the binlog. Snapshot mode: reads the table definition fresh through MySQL's JDBC metadata interface, which normalizes types, every time it runs. Schema Registry sits downstream of both: each connector writes whatever schema it built into the registry, but neither connector reads its own schema back from it. Keep that in mind; it matters later. The trigger: a routine migration With that in mind, here's what actually happened to us. Rewind two years: one of our upstream service teams added a few columns to a source table as part of a standard schema change. ALTER TABLE ADD COLUMN length double NU ## Amazon EC2 R9g and R9gd instances powered by AWS Graviton5 processors are now generally available DevFeed: [Amazon EC2 R9g and R9gd instances powered by AWS Graviton5 processors are now generally available]() Original publisher: [Read original article]() Author: Daniel Abib Published: 2026-08-31T19:53:34Z Content type: release Language: en Sources: [AWS News Blog]() Topics: [Amazon EC2](), [cpu](), [networking]() Tags: [amazon-ec2](), [announcements](), [aws](), [big-data](), [cache](), [compute](), [databases](), [graviton](), [hardware](), [launch](), [memory](), [networking](), [news](), [nitro-isolation-engine](), [nitro-system](), [performance](), [storage]() ### AI overview Amazon EC2 R9g and R9gd memory-optimized instances powered by AWS Graviton5 processors are generally available. The release highlights higher compute performance, faster memory, larger cache, increased network and EBS bandwidth, and local NVMe SSD storage for R9gd. ### Source excerpt Amazon EC2 R9g and R9gd instances powered by AWS Graviton5 are now generally available, delivering up to 25% better compute performance than R8g, ideal for databases, in-memory caches, and real-time analytics. ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs DevFeed: [Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs]() Original publisher: [Read original article]() Author: Sagibhuvana Published: 2026-06-30T11:01:01Z Content type: article Language: en Sources: [Expedia]() Topics: [Apache Spark](), [Large Language Model](), [debugging](), [Optimization](), [MCP]() Tags: [apache-spark](), [big-data](), [data-engineering](), [debugging](), [innovation](), [llm](), [llms](), [mcp](), [mcp-server](), [optimization](), [spark]() ### AI overview This article describes an automated workflow that uses large language models to analyze Spark SQL physical plans, identify performance anti-patterns, and suggest fixes for long-running jobs. It uses an open-source Spark MCP server to provide SQL execution metadata, plans, stage and task metrics, shuffle, spill, and I/O statistics. ### Source excerpt Expedia Group Technology -- InnovationUsing large language models to reveal bottlenecks in Spark SQL execution plansPhoto by Luis del Río If you've ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation. When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don't clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure -- turning optimization into a time-consuming, trial-and-error exercise. So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here's why we built it, how it works, and what we learned after running it on real workloads. Why we even needed this Across teams we kept bumping into the same pain points: Slow or failing jobs that needed deep Spark expertise to debug SQL that didn't behave the way people assumed Skew, spills, or partition issues hiding beneath the surface Tuning efforts that were reactive and inconsistent At some point, the obvious question came up: Can an LLM look at a Spark SQL plan and tell you what's likely to go wrong -- where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk? Turns out: yes How we approached it1. Bring in real context with the Spark MCP server We used an open-source Spark MCP server that exposes: SQL exec ## Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search DevFeed: [Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search]() Original publisher: [Read original article]() Author: Marsan Ma Published: 2026-06-03T23:28:10Z Content type: article Language: en Sources: [Indeed]() Topics: [recommendation systems](), [Embeddings](), [Deep learning](), [Latency](), [Job]() Tags: [big-data](), [data-science](), [deep-learning](), [embeddings](), [engineering](), [feature-store](), [job-search](), [latency](), [machine-learning](), [performance](), [ranking]() ### AI overview Indeed describes a user behavior modeling system for job search that learns from long-term user histories offline, distills them into fixed-length embeddings, and serves them through a feature store for use by online ranking and recommendation models. The approach is designed to preserve rich behavioral signals while meeting latency and cost constraints. ### Source excerpt Authors : Marsan Ma, Nikhil Lopes, Raj Amrit, Hong Lu, Dipankar Biswas, Trent KyonoLeadership: Iris Wang, Madhu Kurup Recommendation and ranking systems power many of the most important experiences on large internet platforms. Yet the models that run in production are rarely the largest models we can train. They are usually compact, latency-sensitive supervised models [...] ## Diagnosing a Deadlock in Trino's Hudi Connector That Stalled Blinkit's Inventory Pipeline DevFeed: [Diagnosing a Deadlock in Trino's Hudi Connector That Stalled Blinkit's Inventory Pipeline]() Original publisher: [Read original article]() Author: Ratul Dawar Published: 2026-05-29T09:26:27Z Content type: article Language: en Sources: [Grofers]() Topics: [Deadlock](), [Concurrent Programming](), [Amazon S3]() Tags: [apache-hudi](), [big-data](), [blinkit](), [bug](), [deadlock](), [open-source](), [s3](), [thread](), [trino](), [trinos](), [yield]() ### AI overview Blinkit describes how a deadlock in Trino's Hudi connector stalled inventory replenishment queries without errors or resource saturation. The issue involved one thread pool handling both file-split production and signalling; the reported fix used cooperative scheduling and was contributed upstream. ### Source excerpt A silent deadlock in our query engine was stalling inventory replenishment jobs with no error, no crash -- just infinite waiting. This is the story of how we found it, traced it to an open-source bug, and fixed it upstream. TL;DRTrino's Hudi connector used a single thread pool for both producing file splits and signalling when there was room for more. Under load, every thread ended up waiting for a signal that had no thread left to run it. The fix was to switch the producer side to a cooperative scheduling pattern: yield the thread when the buffer is full, and resume when space opens. Our inventory replenishment pipeline was frozen. CPU was idle. Memory was fine. There were no errors anywhere. Queries just... stopped moving. The first signal was a long queue on one of our analytics clusters. Queries were piling up. Inventory replenishment jobs -- the jobs that decide how much stock every warehouse and store needs to hold -- were delayed. Blinkit's supply chain was being impacted. Dashboards were turning amber, but nothing was crashing. That was the unsettling part. Investigation: Resources Doing Nothing The affected cluster runs analytical workloads on Trino, reading data stored in Apache Hudi tables on S3. The natural first instinct in a queue build-up is to look at resource saturation -- a CPU spike, memory pressure, network bottleneck. There was none of that. The cluster was sitting largely idle, with CPU barely above baseline and heap usage well within limits. Every new query touching a Hudi table joined the queue and stayed there indefinitely. Queries that were already mid-execution completed fine. Only freshly submitted ones were affected. And crucially, there were no errors. No timeouts, no exceptions in the logs -- just silence and a growing backlog. A thread dump -- a snapshot of what every thread in the process is doing right now -- was our next move. It showed dozens of producer threads all stuck in the same parked state, waiting on the exact same internal signal. ## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines]() Original publisher: [Read original article]() Author: Mahendran Vasagam Published: 2026-05-05T14:00:01Z Content type: article Language: en Sources: [Engineering at Slack]() Topics: [Security](), [OpenSSH](), [Amazon Web Services](), [airflow](), [DataOps](), [Architecture & Design](), [Kubernetes]() Tags: [airflow](), [architecture](), [aws](), [big-data](), [data-analytics](), [data-engineering](), [data-infrastructure](), [kubernetes](), [modernization](), [security](), [ssh](), [uncategorized]() ### AI overview Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection. ### Source excerpt Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security... ## Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam DevFeed: [Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam]() Original publisher: [Read original article]() Author: Yang Yang Published: 2026-02-05T10:39:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Machine Learning & Artificial Intelligence](), [genai](), [AI Research](), [Python](), [Large Language Model](), [Apache Spark](), [SQL](), [Hadoop](), [Reinforcement learning](), [Synthetic Data Generation](), [Transformer]() Tags: [2026](), [ai](), [big-data](), [blog-posts](), [featured](), [genai](), [internship](), [machine-learning](), [ml](), [python](), [research]() ### AI overview Booking.com is recruiting current PhD students in quantitative fields for a three-month GenAI and machine learning research internship in Amsterdam in 2026. Projects include LLM alignment, transformer explainability, embeddings, context engineering, and synthetic data generation. ### Source excerpt At Booking.com, we don't just use Machine Learning -- we use it to solve some of the most complex travel challenges in the world. We're looking for the next generation of researchers to join our Machine Learning community in Amsterdam for a 3-month deep dive into cutting-edge AI. The Program As a Research Intern, you'll be embedded in our teams, working alongside world-class mentors. Your mission? To tackle real-world problems and push the boundaries of the state-of-the-art. Are You the One? We're looking for current PhD students in quantitative fields (CS, Math, AI, Physics) who can conduct independent research and have a solid grip on Python and Big Data tech (SQL, Spark, Hadoop). What's in it for you? You won't just be "an intern". You'll be a contributor to our Machine Learning community. You'll have the opportunity to contribute to the existing efforts of the Machine Learning teams, participate in internal knowledge-sharing sessions, and enjoy the collaborative, high-energy environment of our Amsterdam HQ. Projects Regularized Target Encoding for large real-world datasets Multi-Agent Collaboration Aligning LLMs with user feedback via reinforcement learning Multi-level treatments Interpretable Foundations: Explainability Methods for Transformer Models on Sequential Event Data Scalable and generalisable ID embedding learning Improving property embeddings with better handling of rich and long-context data Utility-aware retrieval for context engineering in travel planning Synthetic Data Generation in Images Requirements We are looking for independent researchers with strong understanding of Machine Learning topics (see requirements for each project in the Linkedin ad), have a track record of peer-reviewed publications and a passion for solving complex problems. Why Booking.com? You'll join a vibrant, diverse community of data scientists and researchers who love to experiment. Beyond the code, you'll experience the unique culture of our Amsterdam headquarters -- a hub ## Segmented models for personal income estimation at Mercado Libre DevFeed: [Segmented models for personal income estimation at Mercado Libre]() Original publisher: [Read original article]() Author: Samuel Vazquez Published: 2026-01-08T20:44:23Z Content type: article Language: en Sources: [Mercado Libre Tech]() Topics: [data](), [personalization](), [datasets](), [big-data]() Tags: [big-data](), [business](), [classification](), [data](), [dataset](), [estimations](), [personal-income](), [personalization](), [regression]() ### AI overview Mercado Libre describes a "silver shotgun" approach for heterogeneous real-world data: segmenting data across contexts, training specialized models for each segment, and combining their predictions. The article presents personal income estimation as an application supporting more personalized financial products and discusses how geographic scale changes the importance of features such as home GPS location. ### Source excerpt Before we dive in, let's give a huge shout-out to Julio Avila Torreblanca and Alejandro Hernandez Cuevas for their exceptional work in developing this system. Thanks also to Enrique Miranda for reviewing the text and providing valuable feedback. Introduction Real-world data is rarely homogeneous. The same feature can mean different things depending on the city, neighborhood, urban density, or user segment. When we force a single global model to explain all those regimes, we usually pay with lower performance. At Mercado Libre, we take a different approach: instead of looking for a "silver bullet," we use a "silver shotgun." This pattern segments the data in several ways, trains specialized models per segment, and combines their predictions into a single estimate. Motivation: Financial inclusion is a significant opportunity In Mexico, only 63% of adults have a formal bank account, only 37% have a credit product, and less than 16% have a bank credit card [1]. About 45% of employed people have a formal source of income, while the rest work in the informal sector [2]. This gap presents a great opportunity for Mercado Libre to enter a growing market. Hyper-personalization is key to our mission to democratize financial inclusion across Latin America (LatAM). To offer great credit products, such as credit cards, we need a reliable estimate of a person's income. Big data, big problems Imagine we want to build a personal income estimation model for people visiting a very small local park. Every day, our dataset would grow by a few hundred entries for children playing, people exercising, and dog walkers, among other activities. Because people in the park live just a few blocks away, home location would have a limited effect (same neighborhood means low GPS variance). The features with the biggest impact on estimated income would likely be age and occupation. If we scale the problem to an entire city, home GPS location will have a bigger effect on the estimate. Each neighborho ## How Data Powers Agent Productivity DevFeed: [How Data Powers Agent Productivity]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2025-10-06T04:22:31Z Content type: article Language: en Sources: [Helpshift]() Topics: [data](), [dashboards]() Tags: [analytics](), [apache-spark](), [big-data](), [customers](), [data](), [data-analysis](), [metrics](), [support]() ### AI overview This article describes how a customer support team developed custom metrics to measure agent productivity more accurately. The metrics aim to distinguish productive work from idle or merely available time and support staffing, scheduling, and performance decisions. ### Source excerpt As a data engineer, I used to see metrics as just numbers on a dashboard -- until I realized they're the lens through which customers view and run their operations. In customer support, for example, agent productivity metrics aren't just figures, they're actionable insights that drive efficiency, shape staffing decisions, and directly impact customer satisfaction. These aren't just charts -- they help customers understand the value we provide, how well things are working, and what decisions to make next. Realizing this changed how I think about building analytics. ➡💡The Question That Shifted Our Perspective In customer support, how well the team works really matters. It affects how much the company spends, how happy the customers are, and how the team feels about their work. Support managers often ask: Are we staffed correctly for the volume we're handling? Are agents spending their time productively? How can we optimize scheduling and performance? When we began our Agent Workforce Management project, we already had a few standard metrics in place like online time, login time, and available time. These told us when agents were present -- but not what they were actually doing. Customers weren't asking "Are our agents online?" They were asking "How productive are our agents?" And truthfully, we didn't have a good answer. There was no visibility into how much time was being spent on real work versus idle time. No way to differentiate between being "available" and being "productive". This made it hard for teams to identify gaps, support high performers, or spot patterns that needed attention. This project was all about answering that question in the right way. 🔍📊 Custom Metrics We Built We designed a set of new metrics that give a clearer picture of how agents spend their time. These metrics give us a deeper understanding of how time is actually being spent, helping us move beyond assumptions and focus on what really drives productivity. Engagement Metrics These show how ## Mastering Entropy: How Mercado Libre captures Millions of Users' Experiences DevFeed: [Mastering Entropy: How Mercado Libre captures Millions of Users' Experiences]() Original publisher: [Read original article]() Author: Franco Bortolazzi Conte Published: 2025-08-14T14:45:10Z Content type: article Language: en Sources: [Mercado Libre Tech]() Topics: [data](), [JSON](), [Artificial Intelligence]() Tags: [analysis](), [architecture](), [big-data](), [data](), [data-analysis](), [data-science](), [governance](), [json](), [mercadolibre](), [technology]() ### AI overview Mercado Libre describes its tracking infrastructure and governance for organizing a large volume of user interaction data. The article explains that its ecosystem generates nearly 50,000 event types and more than 17 billion daily interactions, with tracking records represented in JSON. ### Source excerpt Building a Tracking Infrastructure and Governance for Flexible Information and AI-Driven Data AnalysisGrab a coffee! Let's find order for this chaos together ☕ In the world of e-commerce, every click tells a story. We want to make sure we don't miss a single one. Let's explore the product diversity of Mercado Libre's ecosystem. It is a dynamic and expansive network designed to facilitate and enhance e-commerce & payments across Latin America. At its core is the Mercado Libre Marketplace, a platform that connects millions of buyers and sellers, offering a diverse range of products. Mercado Pago offers an innovative digital payments solution, complementing the ecosystem. To support logistics, Mercado Envíos provides efficient shipping and fulfillment services. Additionally, Mercado Crédito offers financial inclusion by providing loans to businesses and empowering sellers with access to working capital. We can also mention Mercado Play as a content platform, Mercado Ads to connect buyers to sellers or some newer players like the crypto world. Together, these interconnected services create a robust and user-friendly ecosystem, generating a huge amount of data that we collect to enhance our users' experience. The scale is truly remarkable: our platform processes nearly 50,000 distinct event types, culminating in over 17 billion daily interactions -- representing an extraordinary 50% year-over-year growth in the last years. This data deluge presents both unprecedented challenges and opportunities for creating meaningful insights. Then... how do we efficiently organize and manage this diversity and quantity of events?What is an "event" in our context? 🔍 In our ecosystem, we call it a track. We deliberately place the user at the heart of the decision-making process to enrich their experience. This approach directly embodies one of our core cultural principles: "We create value for our users". Technically, the essential form of tracks is a representation in JSON (JavaScript Obj ## Anlytic.com - Big-Data Analytics at the Dutch Police DevFeed: [Anlytic.com - Big-Data Analytics at the Dutch Police]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-05-01T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [dashboards]() Tags: [analytics](), [aws](), [big-data](), [clickhouse](), [dashboards](), [data-analytics](), [data-infrastructure](), [database](), [performance](), [scalability]() ### AI overview An interview with Anlytic.com about using ClickHouse and ClickHouse Cloud on AWS for data analytics, including dashboard production work for the Dutch National Police. The supplied excerpt describes ClickHouse as faster than Snowflake for the stated workloads. ### Source excerpt "If we are looking at high-performance database systems for these types of workloads, the only two options are ClickHouse and Snowflake. ClickHouse is significantly faster." - Martijn Witteveen, CEO ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## Announcing Trino Summit 2024 DevFeed: [Announcing Trino Summit 2024]() Original publisher: [Read original article]() Author: Cole Bowden, Manfred Moser, and Monica Miller Published: 2024-07-11T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [big-data](), [data-engineering](), [Data Science](), [.NET 11]() Tags: [architectures](), [big-data](), [cloud](), [community](), [data-engineering](), [data-science](), [deployment](), [event](), [security](), [sign-up](), [speakers](), [sql](), [streaming](), [summit](), [videos](), [virtual-event]() ### AI overview Trino Summit 2024 is announced as a two-day virtual event on December 11 and 12, featuring talks for engineers, analysts, data scientists, and others interested in Trino. The article also opens a call for beginner to advanced speaker submissions on topics including data architectures, SQL analytics, data engineering, streaming, governance, security, deployment, and custom connectors. ### Source excerpt Fresh off the heels of Trino Fest 2024, where Commander Bun Bun was busy meeting the Trino community in-person, we're already looking forward to another, bigger event to round out the year in Trino. For those who've been here a while, you know that can only mean one thing: Trino Summit 2024. Much like last year, it will be a two-day, fully virtual event, hosting a wide range of talks covering all things Trino on the 11th and 12th of December. Read on for more info, or if you're already convinced... Register to attend! ## Expanded Memory and Compute with Heroku's New Larger Dynos DevFeed: [Expanded Memory and Compute with Heroku's New Larger Dynos]() Original publisher: [Read original article]() Author: Ethan Limchayseng Published: 2024-03-28T02:25:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [cpu](), [Cache](), [big-data](), [Data analysis]() Tags: [big-data](), [cache](), [cli](), [cloud-infrastructure](), [compute](), [cpu](), [data](), [distributed](), [dynos](), [hadoop](), [heroku](), [how-to](), [machine-learning](), [memory](), [news](), [performance-optimization](), [pricing](), [private-spaces](), [real-time](), [spark](), [video]() ### AI overview Heroku introduces nine larger dyno types across its Performance, Private, and Shield tiers, adding higher memory and CPU limits for compute-intensive workloads. The new sizes support use cases including real-time analytics, caching, machine learning, video encoding, and simulations. ### Source excerpt Introduction Heroku is excited to introduce nine new dyno types to our fleets and product offerings. In 2014, we introduced Performance-tier dynos, giving our customers fully dedicated resources to run their most compute-intensive workloads. Now in 2024, today's standards are rapidly increasing as complex applications and growing data volumes consume more memory and carry heavier [...] The post Expanded Memory and Compute with Heroku's New Larger Dynos appeared first on Heroku. ## Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines DevFeed: [Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines]() Original publisher: [Read original article]() Author: Ilay Chen Published: 2024-02-21T16:42:14Z Content type: tutorial Language: en Sources: [PayPal Technology]() Topics: [Apache Spark](), [GPU](), [RAPIDS](), [Cloud](), [Data pipelines](), [migration](), [upgrade](), [Open Source]() Tags: [apache-spark](), [big-data](), [cloud](), [cloud-computing](), [data-pipelines](), [data-processing](), [gpu](), [machine-learning](), [migration](), [rapids](), [upgrade]() ### AI overview A PayPal engineering blog explains how upgrading from Apache Spark 2 to Spark 3 and migrating workloads to GPU clusters with NVIDIA Spark RAPIDS can accelerate selected big-data processing tasks and potentially reduce cloud costs by up to 70%. It covers the migration, parameter tuning, challenges, and reported benefits. ### Source excerpt By Ilay Chen and Tomer Akirav At PayPal, hundreds of thousands of Apache Spark jobs run on an hourly basis, processing petabytes of data and requiring a high volume of resources. To handle the growth of machine learning solutions, PayPal requires scalable environments, cost awareness and constant innovation. This blog explains how Apache Spark 3 and GPUs can help enterprises potentially reduce Apache Spark's jobs cloud costs by up to 70% for big data processing and AI applications. Our journey will begin with a brief introduction of Spark RAPIDS -- Apache Spark's accelerator that leverages GPUs to accelerate processing via the RAPIDS libraries. We will then review PayPal's CPU-based Spark 2 application, our upgrade to Spark 3 and its new capabilities, explore the migration of our Apache Spark application to a GPU cluster, and how we tuned Spark RAPIDS parameters. We will then discuss some challenges we encountered and the benefits of the updates. Libra scales in the cloud, generated by AIBackground GPUs are everywhere, and their parallelism characteristics are perfect for processing AI and graphics applications, among other things. For those unfamiliar: what makes GPUs different from CPUs, computation-wise, is that CPUs have a limited amount of very strong cores, whereas GPUs have thousands, or even tens of thousands or more, relatively weak cores that work together very well. PayPal has been leveraging GPUs to train models for some time now, and so we decided to evaluate if the parallelism of the GPU can be helpful with processing big data applications based on Apache Spark. In our research, we encountered NVIDIA's Spark RAPIDS open-source project. It has many purposes, however we focused on Spark RAPIDS's cost reduction potential, because enterprises like PayPal spend lots of money on running Spark jobs in the cloud. Using Spark with GPUs isn't common in the industry yet, but according to our findings as described in this blog, the potential benefits could be enorm ## Polish edition of Trino: The Definitive Guide DevFeed: [Polish edition of Trino: The Definitive Guide]() Original publisher: [Read original article]() Author: Manfred Moser, Martin Traverso, Matt Fuller Published: 2023-04-06T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [big-data](), [Maintainers]() Tags: [announce](), [big-data](), [book](), [community](), [contributors](), [java](), [maintainers](), [summit](), [translation]() ### AI overview The article announces that a Polish translation of the book Trino: The Definitive Guide is available for communities in Poland and beyond. It highlights Trino's global use and Poland's active Trino, Java, and Big Data communities, including contributors and maintainers. ### Source excerpt At this stage Trino is used all around the globe as we know from the community chat and our speakers at Trino Summit 2022. One large community of Trino contributors and maintainers, many employed by Starburst, is located in Poland. Poland also has a very active participation of developers and users in the Java and Big Data communities. ## Big Data Storage DevFeed: [Big Data Storage]() Original publisher: [Read original article]() Published: 2022-10-21T17:33:51Z Content type: article Language: en Sources: [Bert Hubert's writings]() Topics: [big-data](), [CSV](), [data](), [Data Infrastructure](), [Compression](), [vendor lock-in]() Tags: [big-data](), [compression](), [csv](), [data](), [data-infrastructure](), [programming](), [storage](), [vendor-lock-in]() ### AI overview This article examines storage and transfer formats for large-scale data. It presents CSV as easy to generate, widely supported, reasonably efficient, compressible, and resistant to vendor lock-in, while noting risks involving dialects, escaping, numeric precision, locale, field ordering, and inconsistent types. ### Source excerpt This is a page about some of the mechanics of 'big data', specifically how to store, transfer and process perhaps 100s of millions or billions of rows/events. Although some people think big data starts at 65k rows, at which point Excel used to lose it, I'm talking about gigabytes of data here. Infrastructure for this kind of thing abounds. Data can however come from lots of sources, and go to lots of different places. ## Trino Summit 2022: Federating humans and data DevFeed: [Trino Summit 2022: Federating humans and data]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2022-10-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [AI, ML & Data Engineering](), [big-data](), [datasets](), [Cloud](), [on-prem](), [Open Source](), [SQL](), [Apache Iceberg]() Tags: [architectures](), [big-data](), [blog](), [blog-post](), [book](), [cloud](), [community](), [conference](), [data](), [data-engineering](), [events](), [guide](), [java](), [on-prem](), [open-source](), [speakers](), [sql](), [storage](), [summit](), [us]() ### AI overview The article promotes Trino Summit 2022, whose central theme is query federation. It describes Trino as a query engine that exposes ANSI SQL across multiple data sources and highlights sessions on data mesh, data lakehouse, fault-tolerant execution, and large-scale Trino deployments. The event is presented as a free in-person and virtual conference in San Francisco on November 10, with limited in-person seating. ### Source excerpt Trino has long been the de facto standard to querying large data sets over your cloud or on-prem storage, also known as data lakes. This Trino Summit's theme instead will showcase Trino's other claim to fame: query federation. Trino is a query engine providing an access point that exposes ANSI SQL across multiple data sources. I urge you to join us either in-person or virtually if you are a fan of Trino, big data, open source, data engineering, Java, or all the above! This conference is free and takes place in San Francisco, California on November 10th. ## Happy tenth birthday Trino! DevFeed: [Happy tenth birthday Trino!]() Original publisher: [Read original article]() Author: Brian Olsen, Martin Traverso, Dain Sundstrom, David Phillips, Eric Hwang Published: 2022-08-08T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [big-data](), [SQL](), [data](), [GitHub](), [Slack](), [Development](), [Apache Iceberg]() Tags: [analytics](), [big-data](), [blog-post](), [community](), [data](), [distributed](), [github](), [pull-requests](), [releases](), [scale](), [slack](), [sql](), [youtube]() ### AI overview An anniversary article celebrating Trino's first ten years describes its origins as a community-driven fork of Presto, its role as a distributed big data SQL query engine, and its growth into a system that queries diverse data ecosystems through ANSI SQL. It also highlights learning resources, community milestones, and ways to participate. ### Source excerpt It's inspiring and mindblowing to reflect on the ten year journey that has produced the community around Trino. Trino is the community-driven fork from Presto, the distributed big data SQL query engine created at Facebook in 2012. We are a community of engineers, scientists, analysts, and visionaries that work in a fast paced world where the expectations on the time to insights from our analytics and the scale of the data are ever-increasing. Sometimes words only do so much justice to encompass a journey like this one, so we created a video to let you experience it yourself! Enjoy! ## Why leaving Facebook/Meta was the best thing we could do for the Trino Community DevFeed: [Why leaving Facebook/Meta was the best thing we could do for the Trino Community]() Original publisher: [Read original article]() Author: Martin Traverso, Dain Sundstrom, and David Phillips Published: 2022-08-02T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data analytics](), [big-data](), [Open Source](), [SQL](), [Meta]() Tags: [analytics](), [big-data](), [company](), [leaving](), [linkedin](), [meta](), [netflix](), [open-source](), [sql]() ### AI overview The article explains why leaving Facebook/Meta enabled the creators of Presto to build a broader open-source community around a distributed SQL query engine for big data analytics. It describes how Presto grew beyond Facebook, attracted participation from companies including Airbnb, Dropbox, Netflix, and LinkedIn, and served a wide range of analytics needs. ### Source excerpt It might surprise some that our departure from Facebook was one of the simplest decisions we've ever made. Many posts that discuss leaving a FAANG company focus on leaving some grand sum of money or prestige of working at the company. For us, we were leaving the company where we had launched a project that we knew would quickly outgrow the walls of Facebook, and solve a much larger set of problems in the analytics domain. At the time we didn't quite anticipate that Presto, a distributed SQL query engine for big data analytics, would be adopted around the globe by thousands of companies and an overwhelming number of industries. We appreciate Facebook for serving as the launchpad that inspired others to adopt Presto. Despite the harmonious beginnings, once the needs of the community and Facebook no longer aligned, we had to leave, but we'll get to that part shortly. ## Cinco de Trino recap: Learn how to build an efficient data lake DevFeed: [Cinco de Trino recap: Learn how to build an efficient data lake]() Original publisher: [Read original article]() Author: Brian Olsen, Brian Zhan Published: 2022-05-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [data-engineering](), [big-data](), [Query (disambiguation)](), [datasets](), [Apache Iceberg](), [Stack Overflow](), [X (Twitter)]() Tags: [analytics](), [architecture](), [big-data](), [build](), [clusters](), [community](), [conference](), [cost-savings](), [data](), [data-engineering](), [data-lake](), [etl](), [learn](), [pipelines](), [recap](), [recovery](), [scale](), [tools]() ### AI overview A recap of the Cinco de Trino conference covering Trino's role in data lakehouse architectures, interactive and federated querying, adaptive query planning, and fault-tolerant execution. It highlights Project Tardigrade for autoscaling, spot-instance use, cost savings, and failure recovery, plus a Starburst Galaxy lab for ingesting, cleaning, and analyzing Twitter and Stack Overflow data. ### Source excerpt When Trino (formerly PrestoSQL) arrived on the scene almost 10 years ago, it immediately became known as the much faster alternative to the data warehouse of big data, Apache Hive. The use cases that you, as the community, have built had far exceeded anything we had imagined in complexity. Together we've made Trino not only the fastest way to interactively query large data sets, but also a convenient way to run federated queries across data sources to make moving all the data optional. At Cinco de Trino, we came full circle back to the next iteration of analytics architecture with the data lake. This conference offers advice from industry thought leaders about how to use best lakehouse tools with Trino to manage that data complexity. Hear from industry thought leaders like Martin Traverso (Trino), Dain Sundstrom (Trino), James Campbell (Great Expectations), Jeremy Cohen (DBT Labs), Ryan Blue (Iceberg), Denny Lee (Delta Lake), Vinoth Chandar (Hudi). You can watch the talks on-demand on the Cinco de Trino playlist. In this post, I'd like to cover the key items from each talk you won't want to miss. ## Why and how we use primitive maps DevFeed: [Why and how we use primitive maps]() Original publisher: [Read original article]() Author: Mikhail Chernyakov Senior Software Engineer Published: 2022-03-09T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data](), [Java](), [Optimization](), [Data structures]() Tags: [backend](), [big-data](), [data](), [engineering](), [java](), [memory](), [open-source](), [optimization](), [performance](), [structure]() ### AI overview This article explains how trivago uses concurrent wrappers around primitive maps in Java-based in-memory storage services. It describes bit-packing records into primitive values to reduce memory overhead when storing billions of records. ### Source excerpt At trivago we operate on petabytes of data. In live-traffic applications that are related to the bidding business cases we use our in-house in-memory key-value storage-service written in Java to... [Next page]()