# pipelines Published articles for pipelines. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## sem-ai 0.4.0: smaller responses, safer access, and pre-flight checks DevFeed: [sem-ai 0.4.0: smaller responses, safer access, and pre-flight checks]() Original publisher: [Read original article]() Author: Pete Miloravac Published: 2026-09-16T12:01:04Z Content type: release Language: en Sources: [Semaphore Engineering]() Topics: [AI-assisted coding](), [Claude Code](), [Command-line interface](), [Authorization](), [API](), [configuration]() Tags: [ai-coding-agents](), [api](), [audio](), [claude-code](), [command-line](), [configuration](), [permissions](), [pipelines](), [product-news](), [release](), [video]() ### AI overview Semaphore's sem-ai 0.4.0 release adds more compact workflow and pipeline responses, context switching across organizations and credentials, and support for managing pre-flight checks. The update is aimed at reducing context usage and enabling more targeted permissions for AI coding agents. ### Source excerpt AI coding agents work best when they receive the right information without unnecessary noise. They also need clearly defined permissions and reliable safeguards for the changes they make. sem-ai 0.4.0 improves all three areas. The release introduces more compact pipeline responses, flexible context switching, and support for Semaphore pre-flight checks. Watch Nick demonstrate the new [...] The post sem-ai 0.4.0: smaller responses, safer access, and pre-flight checks appeared first on Semaphore. ## Transform and route security logs to Microsoft Sentinel tables using Observability Pipelines DevFeed: [Transform and route security logs to Microsoft Sentinel tables using Observability Pipelines]() Original publisher: [Read original article]() Author: Zara Boddula; Danielle Park Published: 2026-09-16T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [observability pipelines](), [SIEM, Security](), [Security](), [log management](), [Microsoft]() Tags: [azure](), [cisco-meraki](), [devsecops](), [fortigate](), [log-management](), [logs](), [observability-pipelines](), [pipelines](), [security](), [siem](), [telemetry](), [threat-detection]() ### AI overview Datadog's Observability Pipelines Packs transform firewall, VPN, and network detection logs into Microsoft Sentinel table schemas before ingestion. The post describes Packs for Palo Alto Networks, Fortinet, Cisco ASA, Cisco Meraki, and ExtraHop, including filtering and noise reduction to help control Sentinel ingest volume while retaining visibility. ### Source excerpt Learn how Observability Pipelines Packs map security logs to Microsoft Sentinel schemas and help control downstream ingest volume. ## Announcing instance preference lists for Amazon SageMaker AI training jobs DevFeed: [Announcing instance preference lists for Amazon SageMaker AI training jobs]() Original publisher: [Read original article]() Author: Kanwaljit Khurmi Published: 2026-09-15T16:01:47Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Amazon SageMaker AI](), [GPU](), [Model Development](), [data-processing]() Tags: [amazon-sagemaker-ai](), [announcements](), [gpu](), [intermediate-200](), [model-development](), [pipelines](), [scheduled](), [training]() ### AI overview Amazon SageMaker AI introduces instance preference lists for training and processing jobs. Users can specify up to five instance types in priority order, and SageMaker AI launches the job on the first option with available capacity, reducing manual retries and capacity monitoring. ### Source excerpt Amazon SageMaker AI now offers instance preference lists for training and processing jobs. Specify an ordered list of up to five instance types, and SageMaker AI automatically launches on the first type with available capacity, eliminating manual retry loops and capacity-watching scripts. ## AI-Ready Private Cloud with Cisco and VMware DevFeed: [AI-Ready Private Cloud with Cisco and VMware]() Original publisher: [Read original article]() Author: sabina anja Published: 2026-09-08T15:33:39Z Content type: article Language: en Sources: [VMware Blogs]() Topics: [Artificial Intelligence](), [Network](), [Cloud](), [networking](), [cloud-infrastructure](), [distributed-training](), [datacenter](), [GPU](), [Latency](), [Inference](), [model-deployment](), [inference-endpoints]() Tags: [ai](), [cisco](), [cloud](), [cloud-infrastructure](), [cloud-platform](), [data-center](), [distributed-training](), [fabric](), [gpu](), [home-page](), [inference](), [inference-endpoints](), [latency](), [networking](), [object-storage](), [pipelines](), [private-cloud](), [telemetry](), [vcf-9-1](), [vcf-networking](), [vmware](), [vmware-cloud-foundation]() ### AI overview This article explains why an AI-ready private cloud requires more than adding GPUs. It focuses on how Broadcom and Cisco are integrating VMware Cloud Foundation with Cisco Nexus One Fabric to address AI workload networking, including bandwidth-intensive east-west traffic, bursty north-south traffic, latency, congestion management, and telemetry across virtual and physical infrastructure. ### Source excerpt An AI-ready private cloud is not simply a private cloud with GPUs added to it. What determines whether a private cloud platform can actually serve AI workloads effectively is everything built around them: how the fabric carries traffic, how the tenancy model lets teams consume capacity, and how policy and telemetry stay coherent across the ... Continued The post AI-Ready Private Cloud with Cisco and VMware appeared first on VMware Blogs. ## How we built a benchmarking framework to horizontally accelerate transaction model research DevFeed: [How we built a benchmarking framework to horizontally accelerate transaction model research]() Original publisher: [Read original article]() Author: Nubank Editorial Published: 2026-09-03T13:53:30Z Content type: article Language: en Sources: [Nubank]() Topics: [benchmarking](), [Framework](), [Transformer](), [data]() Tags: [architectures](), [automated](), [benchmarking](), [data](), [data-science-machine-learning](), [experimentation](), [framework](), [model](), [performance](), [pipelines](), [training]() ### AI overview Nubank built an automated benchmarking framework for horizontally evaluating transformer-based transaction representation models across multiple downstream tasks and trials. The framework made experimentation reproducible and statistically rigorous, helping the team identify improvements that generalize across applications. It increased the team's capacity to run experiments by roughly five times per month while reducing operational overhead. ### Source excerpt The framework that transformed weeks of manual experimentation into automated pipelines for horizontal transaction model research The post How we built a benchmarking framework to horizontally accelerate transaction model research appeared first on Building Nubank. ## Build a Multi-Agent GTM Intelligence System DevFeed: [Build a Multi-Agent GTM Intelligence System]() Original publisher: [Read original article]() Author: Avi Chawla Published: 2026-08-25T20:26:56Z Content type: tutorial Language: en Sources: [Daily Dose of Data Science]() Topics: [Code](), [API]() Tags: [agent](), [api](), [building](), [code](), [multi-agent](), [pipelines](), [research]() ### AI overview A hands-on tutorial for building a multi-agent go-to-market intelligence pipeline with Seltz. It explains how combining structured people and company-news records can help identify timely re-engagement opportunities. ### Source excerpt ...explained with code. ## Where Kafka Fits in Modern Systems DevFeed: [Where Kafka Fits in Modern Systems]() Original publisher: [Read original article]() Author: Nikki Siapno Published: 2026-08-25T12:02:09Z Content type: tutorial Language: en Sources: [Level Up Coding System Design Newsletter]() Topics: [Kafka](), [Event-Streaming](), [systems](), [distributed-systems](), [data]() Tags: [architecture](), [consumer](), [distributed-systems](), [durability](), [event-streaming](), [kafka](), [parallelism](), [partition](), [pipelines](), [real-time](), [systems]() ### AI overview This article explains when Apache Kafka is appropriate for distributed systems. It presents Kafka as a durable event log whose replayability, consumer fan-out, and partition-based parallelism support real-time data pipelines and shared event-driven architectures, while warning that Kafka can add unnecessary operational complexity for simpler asynchronous workflows. ### Source excerpt When should you actually use Kafka? And when is it just unnecessary complexity? ## Harness RT Agents Detect Resilience Risks and Generate Tests for CD Pipelines and Kubernetes Workloads DevFeed: [Harness RT Agents Detect Resilience Risks and Generate Tests for CD Pipelines and Kubernetes Workloads]() Original publisher: [Read original article]() Author: Uma Mukkara Published: 2026-08-24T00:00:00Z Content type: release Language: en Sources: [Harness Blog]() Topics: [Resilience](), [Testing](), [Continuous Delivery (CD)](), [Kubernetes](), [dashboards]() Tags: [agentic](), [agents](), [blog](), [chaos](), [configuration](), [continuous-delivery](), [dashboards](), [deployment](), [experiments](), [infrastructure](), [insights](), [kubernetes](), [load](), [pipelines](), [product](), [resilience](), [services](), [teams](), [testing](), [tests]() ### AI overview Harness announces an update to Resilience Testing called RT Agents. The agents analyze CD pipelines and Kubernetes workloads for resilience risks, recommend the testing needed to confirm those risks, and can generate and run chaos experiments or load tests and interpret the results. ### Source excerpt RT Agents detect resilience risk in your CD pipelines and Kubernetes workloads, then generate and run chaos experiments or load tests to confirm it. | Blog ## Fine-Grained Access Control Now Available for All Heroku Customers DevFeed: [Fine-Grained Access Control Now Available for All Heroku Customers]() Original publisher: [Read original article]() Author: Alberto Sigismondi Published: 2026-08-21T17:07:29Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [Access Control](), [Security](), [IAM](), [Zero Trust](), [legacy]() Tags: [access-control](), [deploy](), [heroku](), [iam](), [identity-and-access-management](), [least-privilege](), [legacy](), [news](), [permissions](), [pipelines](), [security](), [security-compliance](), [zero-trust]() ### AI overview Heroku announces that Fine-Grained Access Control is available to all customers. The feature replaces fixed legacy roles with capability-based roles and app-specific permissions for roles such as view, deploy, operate, and manage. ### Source excerpt Fine-Grained Access Controls is now available to all Heroku customers. Heroku's legacy system gave you predefined roles like admin, member, or collaborator, each with a fixed bundle of permissions. It replaces that system with fine-grained roles like view, deploy, operate, and manage, with specific capability sets. Access control is managed at an app-specific layer, giving [...] The post Fine-Grained Access Control Now Available for All Heroku Customers appeared first on Heroku. ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## How MongoDB Atlas and Temporal support reliable production RAG and AI agents DevFeed: [How MongoDB Atlas and Temporal support reliable production RAG and AI agents]() Original publisher: [Read original article]() Author: Suresh Ramappa Published: 2026-08-13T00:00:00Z Content type: opinion Language: en Sources: [Temporal Blog]() Topics: [MongoDB](), [Retrieval Augmented Generation (RAG)](), [AI Infrastructure](), [Artificial Intelligence](), [reliability]() Tags: [agentic](), [agents](), [ai](), [ai-infrastructure](), [api](), [data](), [embeddings](), [long-running](), [mongodb](), [outages](), [pipelines](), [production](), [rag](), [recovery](), [reliability](), [retries](), [retrieval](), [temporal-voices]() ### AI overview The article argues that MongoDB Atlas and Temporal address different reliability needs in production RAG and AI agent systems. Atlas provides operational data, embeddings, vector search, and agent memory in one platform, while Temporal provides durable execution for crash recovery, retries, and long-running ingestion and agent workflows. ### Source excerpt Why MongoDB Atlas and Temporal are better together for AI: one data platform, one durable execution layer, for RAG and agents in prod. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## Q2 2026 Product Update: Harness Pipeline DevFeed: [Q2 2026 Product Update: Harness Pipeline]() Original publisher: [Read original article]() Author: Vishal Vishwaroop Published: 2026-08-06T00:00:00Z Content type: release Language: en Sources: [Harness Blog]() Topics: [ci](), [Git](), [opa](), [configuration](), [Developer experience](), [Monitoring](), [API]() Tags: [2026](), [api](), [ci](), [developer-experience](), [git](), [opa](), [pipeline](), [pipelines](), [update]() ### AI overview Harness's Q2 2026 Pipeline update introduces beta DAG pipelines, allowing stages to declare explicit dependencies. It also adds looping strategies for chained pipelines, earlier matrix exclusions, template and governance improvements, Git-backed entity enforcement, execution visibility, and a pipeline dry-run API. ### Source excerpt DAG pipelines, template overrides, OPA enforcement on Git-backed entities, Git Experience monitoring, and 20 pipeline improvements -- Q2 2026 in review. | Blog ## CI Doesn't Need On-Demand: Moving Our Build Pipelines to Spot Instances DevFeed: [CI Doesn't Need On-Demand: Moving Our Build Pipelines to Spot Instances]() Original publisher: [Read original article]() Author: Yuvraj Singh Singhel Published: 2026-08-05T15:07:35Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [CI/CD](), [GitHub Actions](), [Kubernetes](), [Amazon Web Services]() Tags: [aws](), [ci](), [ci-cd](), [github-actions](), [kubernetes](), [pipelines](), [self-healing]() ### AI overview Razorpay describes a self-healing infrastructure layer for GitHub Actions on Kubernetes that runs most CI workloads on AWS Spot Instances. The system detects spot-node termination, retries jobs, and cleans up orphaned pods; the article states that 80% of CI workloads run on spot instances with a 99.2% job success rate. ### Source excerpt Contributors: Guptaanuj CI/CD pipelines have always had a money-vs-stability problem. Run on-demand AWS instances and your build infrastructure is rock solid, expensive, and predictable. Run on spot instances and your costs drop 70-90%, but AWS can pull the rug with 2 minutes of warning. For most teams, this is a false choice. Either pay full price for reliability, or save money and accept that builds will fail in ways nobody can debug. At Razorpay, we stopped accepting that trade-off. We built a self-healing infrastructure layer for GitHub Actions on Kubernetes that runs 80% of our CI workloads on spot instances while maintaining a 99.2% job success rate. When AWS terminates a spot node mid-build, our system detects it, retries the job, cleans up the orphaned pods, and the developer never knows. This is the story of what we built, why polling wasn't an option, and the war stories that taught us how to do retries without burning everything down. The Problem With Spot Instances On CI Spot instances are AWS capacity that nobody else wants right now. The pricing is brutal compared to on-demand: a c5.2xlarge that costs around $0.34/hour on-demand drops to roughly $0.08/hour on spot. For workloads like CI/CD, where jobs are short-lived and parallelizable, the math is obvious. The catch is in the contract. AWS reserves the right to take spot capacity back at any moment, with a 2-minute warning. That works for some workloads. For others, it's catastrophic. GitHub Actions runners on Kubernetes is the hard case. A typical CI job goes like this: GitHub assigns the job to a runner. The runner is a pod on a Kubernetes cluster running on an AWS Spot instance. The job downloads dependencies, runs tests, builds artifacts. The runner reports back to GitHub. Now insert a spot termination at minute 4 of a 7-minute build. What happens? The runner pod dies mid-job. GitHub never gets a completion signal; the job hangs until timeout, then marks as "failed". A new runner pod gets schedule ## From Batch Snapshots to Near-Real-Time Data DevFeed: [From Batch Snapshots to Near-Real-Time Data]() Original publisher: [Read original article]() Author: Doximity Published: 2026-08-04T16:29:00Z Content type: article Language: en Sources: [Doximity]() Topics: [data](), [Kafka](), [Streaming](), [MySQL](), [Amazon Aurora]() Tags: [amazon-aurora](), [architecture](), [data](), [kafka](), [mysql](), [pipelines](), [streaming]() ### AI overview Doximity describes combining batch database snapshots with Change Data Capture to make data available in minutes while retaining batch-based consistency and recovery guarantees. The design uses Kafka and includes a trusted snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. In a 12-day measurement window, 95% of sampled events reached the queryable intermediate layer within eight minutes of publication to Kafka. ### Source excerpt Change Data Capture (CDC) is often presented as a straightforward pipeline: read a database transaction log, publish each change, and apply those changes to another system. That description is accurate, but it leaves out many of the decisions that determine whether the resulting data can be trusted. At Doximity, we already had a batch pipeline that periodically copied snapshots of application databases into our data warehouse. Those snapshots were reliable, but their freshness was measured in hours. We introduced CDC to make changes available in minutes so downstream transformations and operational analytics would not have to wait for the next batch snapshot. We continued using the batch pipeline for the consistency and recovery guarantees it already provided. Over a 12-day measurement window, 95% of events from a stratified sample of active tables reached the queryable intermediate layer within eight minutes of being published to Kafka. The most interesting parts of the project were not the connections from a source database to Kafka or from Kafka to Snowflake, but four questions we had to answer around them: How could we reuse our existing, transactionally consistent batch snapshots as an on-demand starting point for CDC, without reprocessing every existing row? How could we onboard new tables and absorb schema changes from many source databases across our products without growing operational overhead for each one? How could new changes become queryable without waiting for the warehouse to merge them into place? How could we handle cascading child deletes that MySQL performs but never emits as individual binary-log events? Our answers are the four design decisions in this article: a trusted batch snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. Together, they turned a stream of row changes into a system we could bootstrap, scale, validate, and recover. The sections that follow explain the tradeoffs and guardrails so rea ## Disaster Recovery Testing Best Practices for 2026 DevFeed: [Disaster Recovery Testing Best Practices for 2026]() Original publisher: [Read original article]() Author: Pritesh Kiri Published: 2026-08-04T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [Disaster Recovery](), [Testing](), [Resilience](), [Automation]() Tags: [2026](), [audit](), [automated](), [automation](), [best-practices](), [cross-functional-teams](), [disaster-recovery](), [external](), [metrics](), [pipelines](), [recovery](), [resilience](), [testing]() ### AI overview This article explains how to evolve disaster recovery testing from reactive exercises into a mature, metrics-driven resilience program. It covers risk-aligned testing schedules, automation, continuous improvement, and metrics for demonstrating recovery effectiveness. ### Source excerpt Learn how to move from ad hoc DR tests to a mature, metrics-driven resilience program with risk-aligned schedules, automation, and the KPIs that prove your reco | Blog ## Best Bitbucket Alternatives in 2026 DevFeed: [Best Bitbucket Alternatives in 2026]() Original publisher: [Read original article]() Author: Pete Miloravac Published: 2026-07-29T11:54:28Z Content type: comparison Language: en Sources: [Semaphore Engineering]() Topics: [bitbucket](), [CI/CD](), [GitHub Actions](), [GitLab](), [Self-hosted](), [Cloud](), [Deployment](), [DevSecOps]() Tags: [2026](), [alternatives](), [bitbucket](), [ci-cd](), [circleci](), [cloud](), [compare](), [github](), [github-actions](), [gitlab](), [gitlab-ci](), [linux](), [macos](), [pipelines](), [self-hosted]() ### AI overview This comparison guide examines Bitbucket Pipelines and four alternatives--Semaphore, GitHub Actions, GitLab CI/CD, and CircleCI--to help teams choose a CI/CD platform based on source-control ecosystem, visibility, deployment control, hosting model, and operational requirements. ### Source excerpt Bitbucket Pipelines is a logical starting point for teams already using Bitbucket Cloud. It is built into the source-control experience, configured in a bitbucket-pipelines.yml file, and offers both Atlassian-hosted execution and self-hosted runners. But convenience at the repository level is not always the same thing as the best long-term CI/CD operating model. Teams evaluating Bitbucket [...] The post Best Bitbucket Alternatives in 2026 appeared first on Semaphore. ## LoopSmith: Closed-Loop AI Engineering for Self-Correcting Pipelines on Antigravity 2.0 DevFeed: [LoopSmith: Closed-Loop AI Engineering for Self-Correcting Pipelines on Antigravity 2.0]() Original publisher: [Read original article]() Author: Esther Irawati Setiawan Published: 2026-07-29T09:40:30Z Content type: tutorial Language: en Sources: [Google Developer Experts - Medium]() Topics: [AI Engineering](), [google-antigravity](), [Agent Harness](), [SDKs](), [Command-line interface](), [Code]() Tags: [agent-harness](), [agentic-ai](), [ai](), [ai-engineering](), [antigravity](), [cli](), [google-antigravity](), [pipelines](), [sdk](), [verification]() ### AI overview A guide to using Antigravity 2.0 to build closed-loop AI engineering workflows. It presents a state-machine pattern in which an agent writes, runs, and fixes code against a defined objective until the output is verified, while noting that the SDK is pre-v1.0 and its documented interfaces may change. ### Source excerpt LoopSmith: Closed-Loop AI Engineering -- Autonomous /goal Execution for Self-Correcting Pipelines on Antigravity 2.0Stop prompting the agent turn by turn. Hand it an objective, a bar to clear, and let the state machine write, run, and fix its own code until the output is verified.This guide targets Antigravity 2.0 -- the four-surface release (desktop app, agy CLI, google-antigravity SDK, and enterprise cloud) that shares one agent harness. The SDK is pre-v1.0; symbol names and CLI flags below reflect the documented API as of mid-2026. Treat the patterns as stable and re-check exact signatures against the current docs before you ship.Table of contents The problem with the on-demand agent The architectural shift: closed-loop engineering as a state machine Step 1 -- Initialize the project and the state machine (agy) Step 2 -- Define the objective and trigger /goal execution mode (SDK) Step 3 -- Implement the self-correcting loop Step 4 -- Verification and state finalization Conclusion 1. The problem with the on-demand agent Most "AI engineering" today is still conversational. You prompt; the model answers. You notice the answer is wrong; you prompt again. You paste a traceback; it apologizes and tries once more. The intelligence is real -- but you are the control loop. You are the thing that runs the code, reads the error, decides whether the output is good enough, and feeds the next instruction back in. Take the human out of that seat and the whole system stops. That's fine for a chat window. It falls apart the moment you want an agent to produce a deliverable -- a cleaned dataset, a reconciled financial report, a migration that actually compiles. Real analytical work is iterative and self-referential: you write a script, it crashes on a currency string, you fix the parse, it runs but the totals don't reconcile, you fix the aggregation, and only then is the output trustworthy. Every one of those arrows is a decision. An on-demand agent makes you supply all of them. There are ## Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python DevFeed: [Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python]() Original publisher: [Read original article]() Author: Sameer Yadav Published: 2026-07-25T05:00:00Z Content type: tutorial Language: en Sources: [Kubeflow]() Topics: [Apache Spark](), [SDKs](), [Python](), [Kubernetes]() Tags: [batch](), [cleanup](), [gsoc](), [jobs](), [kubernetes](), [logs](), [pipelines](), [python](), [scheduled](), [sdk](), [spark]() ### AI overview This tutorial explains how the Kubeflow SDK's SparkClient supports submitting and managing batch Spark workloads on Kubernetes from Python. It covers script- and function-based jobs, lifecycle operations, log retrieval, cleanup, and the implementation's current boundaries. ### Source excerpt How the SparkClient SDK's new batch job APIs work under the hood -- submit_job(), FileJob/FuncJob, the lifecycle APIs, and log retrieval. ## Building self-healing feature releases with Harness FME metric alerts and Event Relay DevFeed: [Building self-healing feature releases with Harness FME metric alerts and Event Relay]() Original publisher: [Read original article]() Author: Joshua Klein Published: 2026-07-24T00:00:00Z Content type: tutorial Language: en Sources: [Harness Blog]() Topics: [releases](), [feature flags](), [Automation](), [experiments](), [Resilience](), [Monitoring]() Tags: [automation](), [event](), [feature](), [feature-flags](), [metrics](), [monitoring](), [payload](), [pipelines](), [resilience](), [self-healing](), [webhooks]() ### AI overview This tutorial explains how to connect Harness FME metric alerts to Event Relay triggers so a pipeline can automatically mitigate risky feature releases through actions such as killing a feature flag. The pattern separates signal emission, webhook handling, and controlled remediation. ### Source excerpt Learn how to build self-healing feature releases with Harness Feature Management & Experimentation. Connect metric-alert webhooks to Event Relay triggers and au | Blog ## A Step-by-Step Guide to Feature Flag Implementation in CI/CD DevFeed: [A Step-by-Step Guide to Feature Flag Implementation in CI/CD]() Original publisher: [Read original article]() Author: Aaron Newcomb Published: 2026-07-23T00:00:00Z Content type: tutorial Language: en Sources: [Harness Blog]() Topics: [CI/CD](), [feature flags](), [GitOps](), [Automation](), [Microservices](), [Deployment]() Tags: [automation](), [ci-cd](), [deployment](), [feature-flags](), [gitops](), [governance](), [guide](), [microservices](), [pipelines](), [policy-as-code](), [rollback]() ### AI overview A step-by-step tutorial on implementing feature flags in enterprise CI/CD pipelines. It explains how governance, policy as code, GitOps workflows, automation, verification, gradual rollouts, and rollback capabilities can help teams manage releases across many services while maintaining control and compliance. ### Source excerpt Discover how to implement Feature Flags in CI/CD pipelines using governance, automation, and AI-driven delivery. Speed up your releases while keeping them safe. | Blog ## Lambda-powered functions land in OTTL DevFeed: [Lambda-powered functions land in OTTL]() Original publisher: [Read original article]() Author: OpenTelemetry Authors; Docs CC BY Published: 2026-07-22T20:04:49Z Content type: release Language: en Sources: [Blog on OpenTelemetry]() Topics: [OpenTelemetry](), [telemetry](), [sensitive data](), [data]() Tags: [collection-operations](), [experimental](), [functions](), [lambda](), [opentelemetry](), [pipelines](), [release](), [sensitive-data](), [telemetry](), [transformation]() ### AI overview OpenTelemetry Collector Contrib v0.157.0 introduces lambda expressions to OTTL, enabling reusable inline logic in generic higher-order functions. The release adds eight experimental functions for collection transformations, including filtering, mapping, finding, reducing, and conditional operations. ### Source excerpt As telemetry pipelines become more sophisticated, so do the transformations they need to perform: sanitizing sensitive data, normalizing inconsistent schemas, and enforcing attribute contracts. While OTTL provides a rich set of transformation functions, expressing collection operations has required dedicated functions with hardcoded behavior for each new use case. OpenTelemetry Collector Contrib v0.157.0 changes that by introducing lambda expressions to OTTL. Lambdas let users pass inline logic directly to generic higher-order functions, making complex collection transformations both reusable and concise. The release includes eight new functions that leverage this capability: Filter, MapEach, MapKeys, Any, All, Find, Reduce, and When. ## Agentic Data Engineering Is Here -- But Can It Close the Loop? DevFeed: [Agentic Data Engineering Is Here -- But Can It Close the Loop?]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-22T14:05:27Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [Orchestration](), [data observability](), [Artificial Intelligence](), [AI-assisted coding](), [DuckDB](), [Apache Iceberg]() Tags: [agentic](), [ai-agents](), [data-engineering](), [duckdb](), [orchestration](), [pipelines]() ### AI overview A podcast conversation with Hugo Lu about agentic data engineering and the infrastructure needed for data platforms to execute work, observe outcomes, validate changes, and improve pipelines safely. It examines why production data systems remain difficult for AI, including schema changes, realistic testing, business semantics, and secure execution. ### Source excerpt a conversation with Hugo Lu [Next page]()