# apache-spark Published articles for apache-spark. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Shehab Amin on Spark Compatibility, Rust, and LakeSail DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-02T12:22:23Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust](), [data-engineering](), [apache-arrow](), [agentic-coding]() Tags: [apache-arrow](), [apache-spark](), [data-engineering](), [rust]() ### AI overview A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines. ### Source excerpt Data Engineering Central Podcast. ## Lakesail: A Rust-based alternative to Spark without the JVM DevFeed: [Lakesail: A Rust-based alternative to Spark without the JVM]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-18T21:11:44Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust]() Tags: [apache-spark](), [rust](), [spark]() ### AI overview This commentary article introduces Lakesail as a Rust-based implementation or alternative to Spark that avoids the JVM overhead. It places Lakesail within the broader data-engineering ecosystem, where established tools such as Spark and Kafka continue to coexist with newer projects including Polars, DuckDB, and Daft. ### Source excerpt ya know ... black magic stuff ## Operating Kubernetes at scale: a few stories from running Amazon EKS DevFeed: [Operating Kubernetes at scale: a few stories from running Amazon EKS]() Original publisher: [Read original article]() Author: Neelendra Bhandari Published: 2026-06-30T13:00:00Z Content type: article Language: en Sources: [Kubernetes Overview, News and Trends | The New Stack]() Topics: [Amazon EKS](), [Kubernetes](), [Availability](), [systems](), [Apache Spark](), [AI Platform]() Tags: [agentic-ai](), [ai-infrastructure](), [amazon-eks](), [apache-spark](), [availability](), [aws-marketplace](), [cloud-services](), [data](), [gpu](), [kubernetes](), [post-contributed](), [real-time](), [sponsor-aws-marketplace](), [sponsored-post-contributed](), [systems]() ### AI overview This article describes how operating Amazon EKS at very large scale shaped approaches to Kubernetes control-plane resilience. It argues that availability problems often arise when components amplify faults, and explains how changing workload patterns--including foundation-model training, real-time inference, Apache Spark analytics, and agentic AI--create extremely rapid state transitions that challenge traditional Kubernetes operating assumptions. ### Source excerpt Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack. ## Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs DevFeed: [Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs]() Original publisher: [Read original article]() Author: Sagibhuvana Published: 2026-06-30T11:01:01Z Content type: article Language: en Sources: [Expedia]() Topics: [Apache Spark](), [Large Language Model](), [debugging](), [Optimization](), [MCP]() Tags: [apache-spark](), [big-data](), [data-engineering](), [debugging](), [innovation](), [llm](), [llms](), [mcp](), [mcp-server](), [optimization](), [spark]() ### AI overview This article describes an automated workflow that uses large language models to analyze Spark SQL physical plans, identify performance anti-patterns, and suggest fixes for long-running jobs. It uses an open-source Spark MCP server to provide SQL execution metadata, plans, stage and task metrics, shuffle, spill, and I/O statistics. ### Source excerpt Expedia Group Technology -- InnovationUsing large language models to reveal bottlenecks in Spark SQL execution plansPhoto by Luis del Río If you've ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation. When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don't clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure -- turning optimization into a time-consuming, trial-and-error exercise. So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here's why we built it, how it works, and what we learned after running it on real workloads. Why we even needed this Across teams we kept bumping into the same pain points: Slow or failing jobs that needed deep Spark expertise to debug SQL that didn't behave the way people assumed Skew, spills, or partition issues hiding beneath the surface Tuning efforts that were reactive and inconsistent At some point, the obvious question came up: Can an LLM look at a Spark SQL plan and tell you what's likely to go wrong -- where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk? Turns out: yes How we approached it1. Bring in real context with the Spark MCP server We used an open-source Spark MCP server that exposes: SQL exec ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## Kubeflow SDK v0.4.0: Model Registry, SparkConnect, and Enhanced Developer Experience DevFeed: [Kubeflow SDK v0.4.0: Model Registry, SparkConnect, and Enhanced Developer Experience]() Original publisher: [Read original article]() Author: Kubeflow SDK Team Published: 2026-03-19T05:00:00Z Content type: release Language: en Sources: [Kubeflow]() Topics: [SDKs](), [MLOps](), [Apache Spark](), [data-processing](), [Hyperparameter optimization](), [Machine learning](), [Kubernetes](), [Python]() Tags: [apache-spark](), [data-processing](), [hyperparameter-optimization](), [kubernetes](), [mlops](), [python](), [release](), [sdk]() ### AI overview Kubeflow SDK v0.4.0 introduces a Model Registry Client, SparkClient with SparkConnect support, namespaced TrainingRuntimes, dataset and model initializers, and new documentation. The release targets a unified Python interface for AI workloads on Kubernetes across data processing, model management, and ML pipelines. ### Source excerpt Explore the full documentation at sdk.kubeflow.org ## How Data Powers Agent Productivity DevFeed: [How Data Powers Agent Productivity]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2025-10-06T04:22:31Z Content type: article Language: en Sources: [Helpshift]() Topics: [data](), [dashboards]() Tags: [analytics](), [apache-spark](), [big-data](), [customers](), [data](), [data-analysis](), [metrics](), [support]() ### AI overview This article describes how a customer support team developed custom metrics to measure agent productivity more accurately. The metrics aim to distinguish productive work from idle or merely available time and support staffing, scheduling, and performance decisions. ### Source excerpt As a data engineer, I used to see metrics as just numbers on a dashboard -- until I realized they're the lens through which customers view and run their operations. In customer support, for example, agent productivity metrics aren't just figures, they're actionable insights that drive efficiency, shape staffing decisions, and directly impact customer satisfaction. These aren't just charts -- they help customers understand the value we provide, how well things are working, and what decisions to make next. Realizing this changed how I think about building analytics. ➡💡The Question That Shifted Our Perspective In customer support, how well the team works really matters. It affects how much the company spends, how happy the customers are, and how the team feels about their work. Support managers often ask: Are we staffed correctly for the volume we're handling? Are agents spending their time productively? How can we optimize scheduling and performance? When we began our Agent Workforce Management project, we already had a few standard metrics in place like online time, login time, and available time. These told us when agents were present -- but not what they were actually doing. Customers weren't asking "Are our agents online?" They were asking "How productive are our agents?" And truthfully, we didn't have a good answer. There was no visibility into how much time was being spent on real work versus idle time. No way to differentiate between being "available" and being "productive". This made it hard for teams to identify gaps, support high performers, or spot patterns that needed attention. This project was all about answering that question in the right way. 🔍📊 Custom Metrics We Built We designed a set of new metrics that give a clearer picture of how agents spend their time. These metrics give us a deeper understanding of how time is actually being spent, helping us move beyond assumptions and focus on what really drives productivity. Engagement Metrics These show how ## From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform DevFeed: [From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform]() Original publisher: [Read original article]() Author: Rajit Saha Published: 2025-08-04T18:03:19Z Content type: opinion Language: en Sources: [Udemy Engineering]() Topics: [DataOps](), [MLOps](), [Generative AI](), [Apache Spark](), [Amazon Bedrock](), [Amazon S3](), [Amazon Redshift](), [Amazon SageMaker](), [apache-flink]() Tags: [ai](), [aiops](), [amazon-s3](), [analytics](), [apache-flink](), [apache-spark](), [bedrock](), [dataops](), [large-language-models](), [llmops](), [mlops]() ### AI overview The article describes how DataOps, MLOps, and AI/LLM Ops commonly rely on separate systems and teams for data processing, model deployment, inference, evaluation, orchestration, governance, and monitoring. It then introduces Databricks' Data Intelligence Platform as a unified environment intended to bring these domains together. ### Source excerpt Introduction In modern data-driven businesses, the pace of innovation in analytics and artificial intelligence has outstripped the capacity of many teams. Three distinct disciplines emerged to handle this expansion: Data platform (DataOps) teams built data lakes on cloud storage such as Amazon S3, processed them with Apache Spark and Hive on EMR, ingested streaming data with Spark Structured Streaming or Apache Flink, and loaded tabular copies into MPP warehouses like Redshift for interactive SQL and BI. Cataloguing and governance were offloaded to external tools such as DataHub, and fine-grained access controls required third-party services like Privacera. This architecture worked, but it required separate workflows for batch and streaming, extra systems for lineage and governance, and a mosaic of operational teams. MLOps teams provided an additional layer. Data scientists used notebook environments (for example, Amazon SageMaker) to preprocess data, train, and evaluate models. Deploying models meant writing integration code to move features into a serving layer, to register models in disparate registries and to build custom APIs for inference. Feature stores and model registries were bought from additional vendors. Updates and monitoring were often manual processes. AI/LLM Ops teams are a new addition because generative AI requires specialized components: LLM gateways (e.g., Amazon Bedrock) to proxy access to foundation models; evaluation tooling to compare large language models; orchestration frameworks for agents; vector databases for retrieval augmented generation; and of course another layer of security, access management and cost control. These tools seldom integrate seamlessly with existing data and ML pipelines. This fragmented state makes it difficult to react quickly when product requirements change. Each new capability requires another system, another integration, and another team. Meanwhile, budgets tighten and go-to-market timelines shrink. The questio ## FIPS-ing the Un-FIPS-able: Apache Spark DevFeed: [FIPS-ing the Un-FIPS-able: Apache Spark]() Original publisher: [Read original article]() Published: 2025-04-17T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [Apache Spark](), [chainguard](), [container images](), [Cryptography](), [data-processing](), [distributed-systems](), [Data Infrastructure](), [Cybersecurity]() Tags: [apache](), [apache-spark](), [chainguard](), [chainguard-containers](), [cloud-native](), [compliance](), [container-images](), [cryptography](), [cybersecurity](), [data-infrastructure](), [fedramp](), [fips](), [fraud-detection](), [open-source](), [real-time-data-streaming](), [recommendation-systems](), [spark](), [spark-operator]() ### AI overview Chainguard announces FIPS-validated container images for Apache Spark and Spark Operator, built entirely from source. The article explains the demand for FIPS-compatible Spark in regulated environments and describes the effort to overcome incompatibilities between Spark and FIPS-approved cryptographic libraries. ### Source excerpt Chainguard now offers FIPS-validated container images for Apache Spark and Spark Operator. See how we did it. ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines DevFeed: [Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines]() Original publisher: [Read original article]() Author: Ilay Chen Published: 2024-02-21T16:42:14Z Content type: tutorial Language: en Sources: [PayPal Technology]() Topics: [Apache Spark](), [GPU](), [RAPIDS](), [Cloud](), [Data pipelines](), [migration](), [upgrade](), [Open Source]() Tags: [apache-spark](), [big-data](), [cloud](), [cloud-computing](), [data-pipelines](), [data-processing](), [gpu](), [machine-learning](), [migration](), [rapids](), [upgrade]() ### AI overview A PayPal engineering blog explains how upgrading from Apache Spark 2 to Spark 3 and migrating workloads to GPU clusters with NVIDIA Spark RAPIDS can accelerate selected big-data processing tasks and potentially reduce cloud costs by up to 70%. It covers the migration, parameter tuning, challenges, and reported benefits. ### Source excerpt By Ilay Chen and Tomer Akirav At PayPal, hundreds of thousands of Apache Spark jobs run on an hourly basis, processing petabytes of data and requiring a high volume of resources. To handle the growth of machine learning solutions, PayPal requires scalable environments, cost awareness and constant innovation. This blog explains how Apache Spark 3 and GPUs can help enterprises potentially reduce Apache Spark's jobs cloud costs by up to 70% for big data processing and AI applications. Our journey will begin with a brief introduction of Spark RAPIDS -- Apache Spark's accelerator that leverages GPUs to accelerate processing via the RAPIDS libraries. We will then review PayPal's CPU-based Spark 2 application, our upgrade to Spark 3 and its new capabilities, explore the migration of our Apache Spark application to a GPU cluster, and how we tuned Spark RAPIDS parameters. We will then discuss some challenges we encountered and the benefits of the updates. Libra scales in the cloud, generated by AIBackground GPUs are everywhere, and their parallelism characteristics are perfect for processing AI and graphics applications, among other things. For those unfamiliar: what makes GPUs different from CPUs, computation-wise, is that CPUs have a limited amount of very strong cores, whereas GPUs have thousands, or even tens of thousands or more, relatively weak cores that work together very well. PayPal has been leveraging GPUs to train models for some time now, and so we decided to evaluate if the parallelism of the GPU can be helpful with processing big data applications based on Apache Spark. In our research, we encountered NVIDIA's Spark RAPIDS open-source project. It has many purposes, however we focused on Spark RAPIDS's cost reduction potential, because enterprises like PayPal spend lots of money on running Spark jobs in the cloud. Using Spark with GPUs isn't common in the industry yet, but according to our findings as described in this blog, the potential benefits could be enorm ## Integrating Confluent Schema Registry with Apache Spark applications DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications]() Original publisher: [Read original article]() Author: Dominik Liebler Published: 2022-01-24T08:04:19Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kafka](), [Apache Spark](), [Streaming](), [data lake](), [parquet](), [Kotlin](), [Kubernetes](), [ceph](), [JSON Schema]() Tags: [apache-spark](), [backpressure](), [ceph](), [confluent](), [data-engineering](), [data-lake](), [data-pipeline](), [json](), [kafka](), [kotlin](), [kubernetes](), [parquet](), [payload](), [pipelines](), [schema](), [schemaregistry](), [serialization](), [spark](), [streaming]() ### AI overview This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached. ### Source excerpt At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas ## Плагин Big Data Tools теперь поддерживает IntelliJ IDEA Ultimate, PyCharm Professional, DataGrip 2021.3 EAP и DataSpell DevFeed: [Плагин Big Data Tools теперь поддерживает IntelliJ IDEA Ultimate, PyCharm Professional, DataGrip 2021.3 EAP и DataSpell]() Original publisher: [Read original article]() Author: olegchir (JetBrains) Published: 2021-09-28T06:17:21Z Content type: release Language: ru Sources: [JetBrains RU]() Topics: [Apache Spark](), [ide](), [IntelliJ IDEA](), [PyCharm](), [Data Science](), [Amazon S3](), [Kafka](), [Python]() Tags: [amazon](), [apache-kafka](), [apache-spark](), [apache-zeppelin](), [aws](), [aws-s3](), [configuration](), [hadoop](), [ide](), [intellij-idea](), [kafka](), [pycharm](), [python](), [s3](), [spark](), [zeppelin]() ### AI overview JetBrains released a new Big Data Tools plugin build compatible with IntelliJ IDEA Ultimate and PyCharm Professional 2021.3, with planned support for DataGrip 2021.3 EAP and support for running in DataSpell. The update adds features including Spark Submit run configurations, Kafka monitoring, AWS S3 named profiles, Zeppelin notebook search, and Python interpreter selection. ### Source excerpt Недавно мы выпустили новую сборку плагина Big Data Tools, совместимую со свежими (2021.3) версиями IntelliJ IDEA Ultimate и PyCharm Professional. Когда в октябре выйдет DataGrip 2021.3, эта сборка тоже будет с ним работать. Более того, теперь мы умеем запускаться в DataSpell -- новой IDE для Data Science. Если вы используете старые версии Big Data Tools, сейчас самое время обновиться и попробовать новую версию плагина вместе со свежей версией IDE! В этом году мы много чего улучшили и добавили совершенно новые фичи (например, запуск Spark Submit в виде Run Configuration). Вот небольшой список изменений за этот год. Этот список -- лишь небольшая капля в море того, что изменилось с прошлого года. Читать далее ## Real-Time Machine Learning at Headspace DevFeed: [Real-Time Machine Learning at Headspace]() Original publisher: [Read original article]() Author: Headspace Published: 2021-08-16T16:53:19Z Content type: article Language: en Sources: [Stories by Headspace on Medium]() Topics: [Machine Learning & Artificial Intelligence](), [real-time](), [Inference](), [recommendations](), [Apache Spark](), [databricks](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Amazon SageMaker AI](), [Streaming](), [Orchestration](), [Model Development]() Tags: [android](), [apache-spark](), [aws](), [data](), [databricks](), [headspace-engineering](), [inference](), [machine-learning](), [orchestration](), [real-time](), [recommendations](), [sqs](), [streaming](), [web]() ### AI overview This article describes Headspace's infrastructure for real-time machine learning inference and personalized content recommendations. It explains how the team uses modular Publishing, Receiver, Orchestration, and Serving layers with Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and SageMaker. ### Source excerpt Author: Yu Chen, Senior Software Engineer / Co-author: Koyuki Nakamori, Senior Engineering Manager Introduction Data is often most valuable when it can be immediately leveraged to make decisions in the moment, but traditionally consumer data is ingested, transformed, persisted, and sits dormant for lengthy periods of time before being used by machine learning and analytics teams. Headspace's core products are iOS, Android, and web-based apps that focus on improving the health and happiness of its users through mindfulness, meditation, sleep, exercise, and focus content. Machine learning models are core to our user experiences by offering recommendations that engage our users with new relevant, personalized content that builds consistent habits in their lifelong journey. Finding a way to leverage user data to generate real-time insights and decisions means that consumer-facing products like the Headspace app can dramatically shorten the end-to-end user feedback loop: actions that users perform a few seconds, or minutes ago can be incorporated in generating a more relevant, personalized, and context-specific content recommendation for the user. This means our ML models could incorporate dynamic features that update throughout the course of a user's day, or even an individual session: Current session bounce rates for sleep content Semantic embeddings for recent user search terms (if a user recently searched for "preparing for big exam", the ML model can assign more weight to Focus-themed meditations) Users' biometric data (i.e., if step counts and heart rate are increasing over the last 10 minutes, we can recommend Move or Exercise content) The Headspace Machine Learning team architected a solution by decomposing the infrastructure systems into modular Publishing, Receiver, Orchestration, and Serving layers, leveraging Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and Sagemaker to deliver real-time inference capabilities for our machine learning mod ## How to Reliably Scale Your Data Platform for High Volumes DevFeed: [How to Reliably Scale Your Data Platform for High Volumes]() Original publisher: [Read original article]() Author: Arbab Ahmed Published: 2020-12-08T17:30:29Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data](), [Platform Engineering](), [Kafka](), [MySQL](), [parquet](), [Apache Spark](), [systems]() Tags: [analytics](), [apache](), [apache-parquet](), [apache-spark](), [data](), [data-platform-engineering](), [infrastructure](), [insights](), [kafka](), [parquet](), [platform](), [scale](), [spark](), [streaming](), [systems]() ### AI overview Shopify's Data Platform Engineering team describes how it prepared the data platform to handle the high-volume Black Friday and Cyber Monday event. The platform experienced an average throughput increase of 150 percent and processes data through ingestion, batch or stream processing, and delivery to merchants, partners, and internal teams. The article covers the use of Apache Parquet, Apache Spark, dbt, MySQL, Kafka, and tiered services to prioritize reliability and infrastructure investment. ### Source excerpt In this post, we'll outline the approach we took to reliably scale our data platform in preparation for Black Friday and Cyber Monday. ## A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow) DevFeed: [A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow)]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2020-07-06T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [data-engineering](), [Go Language](), [Python](), [SDKs](), [AI, ML & Data Engineering](), [datasets](), [Google Cloud Platform (GCP)](), [etl](), [apache-flink](), [Apache Spark]() Tags: [apache-flink](), [apache-spark](), [beam](), [benchmarking](), [big-data](), [cloud-dataflow](), [data-engineering](), [dataflow](), [distributed](), [etl](), [gcp](), [go](), [golang](), [google-cloud](), [open-source](), [performance](), [programming](), [python](), [spark]() ### AI overview A comparison of Go and Python for data engineering using Apache Beam and Google Dataflow. It examines their SDK differences, drawbacks, benchmark performance, and the feasibility of switching between them. ### Source excerpt In Part 2 of our comparison of Python and go from a Data Engineering perspective, we'll finally take a look at Apache Beam and Google Dataflow and how the go SDK and the Python SDK differ, what drawbacks we're dealing with, how fast it is by running extensive benchmarks, and how feasible it is to make the switch ## ApacheCon Europe 2019 -- Thoughts and Insights by Airflow Committers DevFeed: [ApacheCon Europe 2019 -- Thoughts and Insights by Airflow Committers]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2019-11-22T00:00:00Z Content type: opinion Language: en Sources: [Apache Airflow Blog]() Topics: [Open Source](), [airflow](), [Apache Spark](), [flink]() Tags: [airflow](), [apache](), [apache-spark](), [community](), [flink](), [foundation](), [open-source](), [organization](), [projects]() ### AI overview The article shares insights from Apache project committers after ApacheCon Europe 2019, focusing on the Apache Software Foundation's community-centered model for organizing and sustaining open-source projects such as Airflow, Spark, Flink, and Beam. ### Source excerpt Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Way--a way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the "community over code" motto, we can enjoy such awesome projects like Apache Spark, Flink, Beam, or Airflow (and many more). After this year's ApacheCon, Polidea's engineers talked with Committers of Apache projects, such as--Aizhamal Nurmamat kyzy, Felix Uellendall, and Fokko Driesprong--about insights to what makes the ASF such an amazing organization. You can read the insights after the ApacheCon 2019. ## Creating Readable Spark Jobs DevFeed: [Creating Readable Spark Jobs]() Original publisher: [Read original article]() Published: 2018-08-06T00:00:00Z Content type: article Language: en Sources: [SoundCloud Backstage Blog]() Topics: [Apache Spark](), [Functional programming](), [Scala](), [Programming]() Tags: [apache-flink](), [apache-spark](), [batch](), [engineering](), [functional-programming](), [kafka](), [scala](), [spark](), [streaming]() ### AI overview This article explains how to make Apache Spark jobs more readable by breaking complex, deeply nested operations into smaller composable units. It presents functional programming techniques in Scala and illustrates them with a simplified SoundCloud content-routing example based on artist and label metadata rules. ### Source excerpt Nowadays, it's rather common to encounter Apache Spark being utilized in a lot of companies that need to process huge amounts of data, and things aren't any different here at SoundCloud -- as one can imagine, we have lots of data to process all the time. ## How Target Performance Tunes Machine Learning Applications DevFeed: [How Target Performance Tunes Machine Learning Applications]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-09-29T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Machine learning](), [distributed-systems](), [Apache Spark](), [Hadoop](), [Scalability](), [Filesystems](), [datasets](), [Algorithms](), [Transactions](), [Open Source]() Tags: [algorithms](), [apache-spark](), [data](), [data-science-and-engineering](), [distributed](), [engineering](), [hadoop](), [machine-learning](), [open-source](), [scalability](), [science]() ### AI overview The article discusses improving the performance and relevance of machine-learning applications by evaluating matrix multiplication at scale. It describes distributed computing requirements, distributed file storage, scalability, fault tolerance, and the use of Apache Hadoop, Apache Spark, and Apache Mahout. ### Source excerpt At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c ## Getting the current filename with Spark and HDFS DevFeed: [Getting the current filename with Spark and HDFS]() Original publisher: [Read original article]() Published: 2014-09-28T11:50:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Apache Spark](), [Hadoop]() Tags: [apache-spark](), [foreach](), [hadoop](), [hdfs](), [map](), [reduce](), [scala](), [spark](), [subclass]() ### AI overview A tutorial shows how to obtain the filename being processed in a Spark map/reduce job. It uses Spark 1.1.0's HadoopRDD.mapPartitionsWithInputSplit with FileSplit, and notes that the approach requires an InputFormat subclass of FileInputFormat. ### Source excerpt It's occasionally useful when writing map/reduce jobs to get a hold of the current filename that's being processed. There's a few ways to do this, depending on the version of Spark that you're using. Spark 1.1.0 introduced a new method on HadoopRDD that makes this super easy: import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.mapred.{FileSplit, TextInputFormat} import org.apache.spark.rdd.HadoopRDD // Create the text file val text = sc.hadoopFile("file:///tmp/files", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], sc.defaultMinPartitions) // text: org.apache.spark.rdd.RDD[(org.apache.hadoop.io.LongWritable, org.apache.hadoop.io.Text)] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 // Cast to a HadoopRDD val hadoopRdd = text.asInstanceOf[HadoopRDD[LongWritable, Text]] // hadoopRdd: org.apache.spark.rdd.HadoopRDD[org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 val fileAndLine = hadoopRdd.mapPartitionsWithInputSplit { (inputSplit, iterator) ⇒ val file = inputSplit.asInstanceOf[FileSplit] iterator.map { tpl ⇒ (file.getPath, tpl._2) } } fileAndLine.foreach(println) // prints lines like (file:/tmp/files/part-00000,line1...), (file:/tmp/files/part-00000,line2...), ... There are a few things to note: - You can't simply use sc.textFile because under the hood, the textFile method actually maps the resulting HadoopRDD, returning a MappedRDD which is not what we want - This will only work if your InputFormat is a subclass of FileInputFormat - mapPartitionsWithInputSplit returns a new iterator, in this case we simply map over it (which is done lazily!) to include the split's filename