# data-architecture

A technical framework for designing and governing an organization's data collection, management, movement, storage, and use.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A technical deep dive into how Fivetran centralizes SAP data | Blog | Fivetran

DevFeed: [A technical deep dive into how Fivetran centralizes SAP data | Blog | Fivetran](<https://devfeed.tech/articles/a-technical-deep-dive-into-how-fivetran-centralizes-sap-data-blog-fivetran-140187.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/a-technical-deep-dive-into-how-fivetran-centralizes-sap-data>)

Author: Taylor Brown

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [ABAP](<https://devfeed.tech/topics/abap.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [abap](<https://devfeed.tech/tags/abap.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [archiving](<https://devfeed.tech/tags/archiving.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automation](<https://devfeed.tech/tags/automation.md>), [autonomy](<https://devfeed.tech/tags/autonomy.md>), [blog](<https://devfeed.tech/tags/blog.md>), [business-process](<https://devfeed.tech/tags/business-process.md>), [business-process-automation](<https://devfeed.tech/tags/business-process-automation.md>), [business-value](<https://devfeed.tech/tags/business-value.md>), [data](<https://devfeed.tech/tags/data.md>), [deep-dive](<https://devfeed.tech/tags/deep-dive.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [middleware](<https://devfeed.tech/tags/middleware.md>), [sap](<https://devfeed.tech/tags/sap.md>), [technical](<https://devfeed.tech/tags/technical.md>)

### AI overview

Fivetran's SAP connector extracts changed data through an SAP RFC connection and a downloadable NetWeaver API transport, which requires SAP NetWeaver 7.5 or higher. It uses change pointers and table triggers to limit backend overhead, then validates and compresses data for transfer to a customer-selected warehouse or data lake. The article says the approach uses SAP's application layer rather than ODP or SAP middleware, works with runtime and enterprise HANA environments including RISE Private, and supports ECC through RFC; ECC customers with an enterprise license may also connect directly to Oracle or SQL Server databases. It also describes SAP OData support, several secure connection methods, and hybrid deployment. Direct database access for S/4HANA requires an enterprise license, while RISE customers have limited infrastructure control and connection options. The article presents Fivetran as a way to centralize SAP and other enterprise data for analytics and AI, while noting that customers should assess SAP's technical and contractual constraints.

### Source excerpt

Break free from SAP data silos. Learn how Fivetran securely centralizes SAP data to power analytics, automation, and AI -- without middleware.

## The importance of unstructured data for AI

DevFeed: [The importance of unstructured data for AI](<https://devfeed.tech/articles/the-importance-of-unstructured-data-for-ai-140261.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-importance-of-unstructured-data-for-ai>)

Author: Charles Wang

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-platforms](<https://devfeed.tech/topics/data-platforms.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [automated](<https://devfeed.tech/tags/automated.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [unstructured-data](<https://devfeed.tech/tags/unstructured-data.md>)

### AI overview

Generative AI can extract meaning from unstructured text, images, code, audio and video, while many enterprise questions also require structured data for reporting and quantitative analysis. The article says unstructured data makes up an estimated 80-90% of organizational data, but is harder to standardize, govern and check for compliance than structured data. It presents automated integration as a way to centralize both types in a warehouse or data lake: Fivetran's file connectors ingest PDFs, images and documents alongside structured sources, and its Connector SDK supports building connectors for unsupported sources. The article also notes that LLMs can struggle with purely computational questions, so they may need integration with systems specialized in mathematical reasoning. These are the article's claims about Fivetran and AI use; it does not provide an independent product comparison.

### Source excerpt

Much of your organization's most important data is unstructured. Automated data integration allows you to access it, and AI to analyze it.

## Unleashing the power of Fivetran and Azure Synapse for accelerated data movement | Blog | Fivetran

DevFeed: [Unleashing the power of Fivetran and Azure Synapse for accelerated data movement | Blog | Fivetran](<https://devfeed.tech/articles/unleashing-the-power-of-fivetran-and-azure-synapse-for-accelerated-data-movement-blog-fivetran-140266.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/unleashing-the-power-of-fivetran-and-azure-synapse-for-accelerated-data-movement>)

Author: Kelly Kohlleffel

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Azure](<https://devfeed.tech/topics/azure.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [azure](<https://devfeed.tech/tags/azure.md>), [azure-synapse](<https://devfeed.tech/tags/azure-synapse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>)

### AI overview

Fivetran can move database data into Azure Synapse through managed connectors, handling historical loads and incremental change capture while automatically replicating schema changes and recovering from failures. In the Oracle RDS example, setup requires source credentials, a connection method, an update method, and selected schemas and tables; the initial historical sync reportedly finished in under a minute. Standard connectors offer Log Miner or Teleport sync and can run at intervals as short as five minutes; High Volume Agent (HVA) requires deploying an agent at the source and offers one-minute intervals. HVR also uses a source agent and supports continuous replication, including an initial refresh followed by ongoing change capture. The article presents Synapse Data Studio, SQLPro Studio, and Preset.io as ways to query or visualize the resulting data. It frames Fivetran and Azure Data Factory as tools to evaluate against delivery speed, latency, scale, maintenance, security, usability, and total costs, rather than recommending one tool for every case.

### Source excerpt

Learn how the power of Fivetran and Azure Synapse work to accelerate data movement

## QuarkDB: A Hybrid Database System for Quantum Data Management

DevFeed: [QuarkDB: A Hybrid Database System for Quantum Data Management](<https://devfeed.tech/articles/quarkdb-a-hybrid-database-system-for-quantum-data-management-138302.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20807>)

Author: Wenrui Zhang, Yu Wang, Xiao Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [sql-database](<https://devfeed.tech/topics/sql-database.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>)

Tags: [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [database](<https://devfeed.tech/tags/database.md>), [execution](<https://devfeed.tech/tags/execution.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [query](<https://devfeed.tech/tags/query.md>), [query-engine](<https://devfeed.tech/tags/query-engine.md>), [storage](<https://devfeed.tech/tags/storage.md>), [version](<https://devfeed.tech/tags/version.md>)

### AI overview

QuarkDB unifies relational and quantum data in one object model and defines a Quantum Structured Query Language. Its hybrid query engine selects cost-based plans across classical and quantum operators; its execution engine uses a three-level QuarkDB-IR to support quantum execution and reduce processing costs; and its storage engine supports version management, state reconstruction, and data reuse. The abstract reports query-latency speedups of 2.14x to 25.86x versus SQLite, PostgreSQL, and Qute across the authors' real-world workloads. It provides no workload-specific results or evaluation details in the supplied excerpt.

### Source excerpt

arXiv:2611.20807v1 Announce Type: new Abstract: Emerging quantum applications like quantum communication generate large volumes of high-dimensional quantum data, requiring efficient data analysis and storage. However, classical database systems often struggle to meet requirements such as quantum processing or version management. This paper presents QuarkDB designed for efficient quantum data management. QuarkDB establishes a unified object model for relational and quantum data and defines the syntax of Quantum Structured Query Language. To support efficient quantum data management, QuarkDB integrates three key components: (1) a Hybrid Query Engine with cost-based hybrid plan selection over classical and quantum operators, (2) an Execution Engine with a three-level instructions intermediate representation, QuarkDB-IR, to support quantum execution and reduce quantum processing costs, and (3) a Storage Engine supporting quantum data version management, state reconstruction and data reuse. Compared with classical databases such as SQLite and PostgreSQL or quantum databases like Qute, our QuarkDB achieves query-latency speedups ranging from 2.14x to 25.86x speedup under extensive real-world workloads.

## PX-CSI 26.3 Adds Cross-Array Snapshots for Kubernetes Storage

DevFeed: [PX-CSI 26.3 Adds Cross-Array Snapshots for Kubernetes Storage](<https://devfeed.tech/articles/px-csi-26-3-adds-cross-array-snapshots-for-kubernetes-storage-121799.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/solutions/px-csi-26-3-kubernetes-storage-2/>)

Author: Peter Eicher

Published: 2026-11-06T16:00:00Z

Content type: release

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [self-service](<https://devfeed.tech/topics/self-service.md>), [openebs localpv zfs](<https://devfeed.tech/topics/openebs-localpv-zfs.md>), [Disk image](<https://devfeed.tech/topics/disk-image.md>), [ZFS](<https://devfeed.tech/topics/zfs-file-system-features.md>)

Tags: [array](<https://devfeed.tech/tags/array.md>), [cross](<https://devfeed.tech/tags/cross.md>), [csi](<https://devfeed.tech/tags/csi.md>), [everpure](<https://devfeed.tech/tags/everpure.md>), [flasharray](<https://devfeed.tech/tags/flasharray.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-storage](<https://devfeed.tech/tags/kubernetes-storage.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [solutions](<https://devfeed.tech/tags/solutions.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

PX-CSI 26.3 lets Kubernetes clusters use mixed SAN protocols across nodes and supports ARM64. For recovery, it can asynchronously send FlashArray snapshots to a second array in the same cluster (both arrays require Purity 6.6.1 or later) and restore FlashBlade snapshots to new volumes without overwriting the source. ActiveCluster stretched pods now support FlashArray realms over iSCSI and Fibre Channel, combining high availability with tenant separation. New controls include enforced FlashArray File Services quotas, per-StorageClass NFS server pinning (Purity 6.8.7 or later), TLS certificate verification, provisioning pauses per array, and immediate volume eradication. PX-CSI is bundled with FlashArray and FlashBlade and manages volume lifecycle through Kubernetes; version 26.3 requires Portworx Operator 26.4.0 or later.

### Source excerpt

PX-CSI 26.3 Adds Cross-Array Snapshots for Kubernetes Storage by Everpure Blog Explore how PX-CSI 26.3 expands Kubernetes storage management for FlashArray and FlashBlade, with cross-array FlashArray snapshots, FlashBlade snapshot restores, and more. The post PX-CSI 26.3 Adds Cross-Array Snapshots for Kubernetes Storage appeared first on Everpure Blog.

## How to assess your organization's AI readiness

DevFeed: [How to assess your organization's AI readiness](<https://devfeed.tech/articles/how-to-assess-your-organization-s-ai-readiness-121083.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/how-to-assess-your-ai-readiness>)

Author: Daniel Poppy

Published: 2026-11-05T21:49:00Z

Content type: article

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [accountability](<https://devfeed.tech/tags/accountability.md>), [act](<https://devfeed.tech/tags/act.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [advanced](<https://devfeed.tech/tags/advanced.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-adoption](<https://devfeed.tech/tags/ai-adoption.md>), [ai-readiness](<https://devfeed.tech/tags/ai-readiness.md>), [data](<https://devfeed.tech/tags/data.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [insights](<https://devfeed.tech/tags/insights.md>), [organization](<https://devfeed.tech/tags/organization.md>), [takes](<https://devfeed.tech/tags/takes.md>)

### AI overview

AI adoption alone does not show whether an organization can scale AI reliably; readiness depends on the data and organizational capabilities supporting it. The Fivetran + dbt Labs model assesses six dimensions: data quality, governance, ownership, cost and operational efficiency, context, and data architecture and interoperability. The excerpt recommends assessing each separately to expose uneven maturity, then bringing together stakeholders across data, technology, governance, and AI to compare perspectives and prioritize capabilities based on the organization's next use cases. Readiness needs reassessment as use cases, scale, and autonomy change. The text reports that 42% of organizations cited data quality and lineage issues as a barrier to agentic AI goals, while 86% of data leaders considered platform extensibility and interoperability important or critical. These findings are attributed to Fivetran's 2026 survey of 400 data professionals.

### Source excerpt

AI adoption moves fast. Readiness takes longer. Assess 6 dimensions to see where your data foundation limits AI.

## AI Workloads Are Breaking Old Storage Assumptions

DevFeed: [AI Workloads Are Breaking Old Storage Assumptions](<https://devfeed.tech/articles/ai-workloads-are-breaking-old-storage-assumptions-121201.md>)

Original publisher: [Read original article](<https://www.backblaze.com/blog/ai-workloads-are-breaking-old-storage-assumptions/>)

Author: Maddie Presland

Published: 2026-11-05T16:40:36Z

Content type: article

Language: en

Sources: [Backblaze](<https://devfeed.tech/sources/backblaze-blog-cloud-storage-cloud-backup.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [systems](<https://devfeed.tech/topics/systems.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [2013](<https://devfeed.tech/tags/2013.md>), [2022](<https://devfeed.tech/tags/2022.md>), [2025](<https://devfeed.tech/tags/2025.md>), [2026](<https://devfeed.tech/tags/2026.md>), [aging](<https://devfeed.tech/tags/aging.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [ai-data-centers](<https://devfeed.tech/tags/ai-data-centers.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [b2cloud](<https://devfeed.tech/tags/b2cloud.md>), [backblaze](<https://devfeed.tech/tags/backblaze.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [drive](<https://devfeed.tech/tags/drive.md>), [featured](<https://devfeed.tech/tags/featured.md>), [featured-cloud-storage](<https://devfeed.tech/tags/featured-cloud-storage.md>), [network](<https://devfeed.tech/tags/network.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [stats](<https://devfeed.tech/tags/stats.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Storage evaluation for AI workloads needs more than price per terabyte: bursty, parallel data movement and weak performance can leave costly GPUs idle. Backblaze presents three data sources: Network Stats tracks traffic by network type and reports unusually high neocloud traffic, while noting the dataset is young and limited to Backblaze's own network; Performance Stats benchmarks single- and multithreaded uploads and downloads across file sizes, publishing methodology and rate-limited results, but says two quarters are too little for firm conclusions; Drive Stats reports long-term hard-drive reliability, not SSDs, with a 2025 annual failure rate of 1.36% across 557 million drive-days. The post recommends using these findings to map requirements to workload needs and running tests against one's own stack. It also describes how larger drives increase storage density while adding complexity to erasure-coded protection.

### Source excerpt

AI workloads are rewriting the rules for cloud storage. Explore what Backblaze's Network Stats, Performance Stats, and Drive Stats reveal about data movement, storage performance, and drive reliability, and use the findings to evaluate storage for your own AI workloads. The post AI Workloads Are Breaking Old Storage Assumptions appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

## Mobility-Aware Mobile Edge Computing for Energy-Balanced Data Collection in IoT-Enabled Underwater Sensor Networks

DevFeed: [Mobility-Aware Mobile Edge Computing for Energy-Balanced Data Collection in IoT-Enabled Underwater Sensor Networks](<https://devfeed.tech/articles/mobility-aware-mobile-edge-computing-for-energy-balanced-data-collection-in-iot-enabled-underwater-sensor-networks-119178.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06186>)

Author: Nguyen Thi My Binh, Do Dinh Hoan, Tong Tien Manh, Vu Minh Yen, Dang Trong Hop, Ho Viet Duc Luong

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Internet of things](<https://devfeed.tech/topics/iot.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [clean energy data centers](<https://devfeed.tech/topics/clean-energy-data-centers.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [iot](<https://devfeed.tech/tags/iot.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sensor](<https://devfeed.tech/tags/sensor.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [trajectory-planning](<https://devfeed.tech/tags/trajectory-planning.md>)

### AI overview

MECEBDC targets energy-balanced data collection in IoT-enabled underwater sensor networks by jointly optimizing energy-aware clustering, cluster-head selection, and autonomous underwater vehicle (AUV) data collection. Its metaheuristic routing algorithm accounts for underwater mobility, including gravitational pressure and buoyancy, and uses MEC-enabled AUV computing, storage, and mobility to plan trajectories and adapt data gathering in 3D. The abstract reports that simulations and theoretical analysis found lower node energy consumption, more efficient data collection, and longer network lifetime than existing schemes; it provides no specific methods, measurements, or comparator details.

### Source excerpt

arXiv:2611.06186v1 Announce Type: cross Abstract: Underwater wireless sensor networks (UWSNs) face critical challenges in energy imbalance, connectivity, and data collection due to harsh environments. To address these issues, this paper focuses on the problem is how to achieve energy-balanced data collection in IoT-enabled UWSNs with a mobility-aware mobile edge computing (MEC), named MECEBDC. Unlike existing approaches, the proposed scheme jointly optimizes energy-aware clustering, cluster head selection, and AUV-assisted data collection. Furthermore, we design a novel metaheuristic-based AUV routing algorithm that incorporates realistic underwater mobility dynamics, explicitly considering gravitational pressure and buoyancy forces acting on the AUV. By leveraging the computing, storage, and mobility capabilities of MEC-enabled AUVs, the proposed approach enables efficient trajectory planning and adaptive data gathering in a stable 3D underwater environment. Extensive simulations and theoretical analysis demonstrate that the proposed method significantly reduces node energy consumption, improves data collection efficiency, and prolongs network lifetime compared with existing schemes, making it well suited for long-term underwater IoT applications.

## Immutable Backup Architecture Explained: How WORM and Air-Gapped Storage Actually Stop Ransomware

DevFeed: [Immutable Backup Architecture Explained: How WORM and Air-Gapped Storage Actually Stop Ransomware](<https://devfeed.tech/articles/immutable-backup-architecture-explained-how-worm-and-air-gapped-storage-actually-stop-ransomware-106123.md>)

Original publisher: [Read original article](<https://hackernoon.com/immutable-backup-architecture-explained-how-worm-and-air-gapped-storage-actually-stop-ransomware?source=rss>)

Author: Pallavi Rani

Published: 2026-10-29T07:24:30Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>)

Tags: [admin](<https://devfeed.tech/tags/admin.md>), [air-gapped](<https://devfeed.tech/tags/air-gapped.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-object-lock](<https://devfeed.tech/tags/amazon-s3-object-lock.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automated](<https://devfeed.tech/tags/automated.md>), [aws-s3](<https://devfeed.tech/tags/aws-s3.md>), [azure](<https://devfeed.tech/tags/azure.md>), [backup](<https://devfeed.tech/tags/backup.md>), [backup-and-recovery-testing](<https://devfeed.tech/tags/backup-and-recovery-testing.md>), [backups](<https://devfeed.tech/tags/backups.md>), [blob](<https://devfeed.tech/tags/blob.md>), [blob-storage](<https://devfeed.tech/tags/blob-storage.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [bypass](<https://devfeed.tech/tags/bypass.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [identity](<https://devfeed.tech/tags/identity.md>), [identity-and-access-management](<https://devfeed.tech/tags/identity-and-access-management.md>), [immutable-backup](<https://devfeed.tech/tags/immutable-backup.md>), [immutable-backups](<https://devfeed.tech/tags/immutable-backups.md>), [immutable-storage](<https://devfeed.tech/tags/immutable-storage.md>), [ransomware](<https://devfeed.tech/tags/ransomware.md>), [security-architecture](<https://devfeed.tech/tags/security-architecture.md>), [storage](<https://devfeed.tech/tags/storage.md>), [worm](<https://devfeed.tech/tags/worm.md>)

### AI overview

A stolen production credential can compromise backups unless storage, identity and recovery controls create separate boundaries. Storage-level WORM retention blocks deletion or overwriting until the retention period expires; application-level immutability can be disabled by an attacker with sufficient application privileges. S3 Object Lock governance mode permits bypass by specially authorized users, while compliance mode blocks deletion and overwriting during retention, even by the account root user. Azure Blob Storage likewise distinguishes removable unlocked retention from locked retention. Immutability prevents alteration; air-gapping limits reachability, but offline copies still depend on controlled reconnection. The article recommends separating production, backup and storage identities, restricting break-glass access, and requiring a second independent approval for destructive actions. It also calls for isolated restore tests that verify applications run, since healthy backup jobs do not ensure recovery. These controls add operational burden, retention costs and recovery delays, especially for offline copies.

### Source excerpt

Immutable storage blocks deletion, but a stolen admin identity can still do damage. Follow one credential through WORM, air gaps and quorum approval.

## The Best Metadata Management Tools for 2026

DevFeed: [The Best Metadata Management Tools for 2026](<https://devfeed.tech/articles/the-best-metadata-management-tools-for-2026-105913.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/purely-educational/best-metadata-management-tools/>)

Author: Everpure

Published: 2026-10-28T18:34:17Z

Content type: article

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [knowledge-management](<https://devfeed.tech/topics/knowledge-management.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [best](<https://devfeed.tech/tags/best.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [data](<https://devfeed.tech/tags/data.md>), [educational](<https://devfeed.tech/tags/educational.md>), [everpure](<https://devfeed.tech/tags/everpure.md>), [management](<https://devfeed.tech/tags/management.md>), [management-tools](<https://devfeed.tech/tags/management-tools.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [tools](<https://devfeed.tech/tags/tools.md>), [unstructured-data](<https://devfeed.tech/tags/unstructured-data.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Metadata management tools harvest and govern technical, business, and operational metadata; most enterprise products combine that broader work with a catalog's search and discovery. The article groups options into enterprise platforms, cloud or platform-native catalogs, and open source platforms. Selection depends on how many systems hold important data, unstructured-data coverage, governance maturity, and available engineering capacity. Compare active metadata, lineage depth, sensitivity labeling, agent access through APIs or MCP, interoperability, and total cost; test coverage against real file shares and object stores. Native catalogs can suit a dominant platform, enterprise suites span diverse estates, and open source avoids license fees while shifting deployment, upgrades, and connector work to the team. A recurring limitation is missing sensitivity labels and business context in unstructured data, so the article describes pairing catalogs with discovery and classification tools that feed labels into catalogs and access policies. It presents Everpure as a complementary provider, not a catalog replacement.

### Source excerpt

The Best Metadata Management Tools for 2026 by Everpure Blog Explore the best metadata management tools for 2026 by category, with tips on lineage, active metadata, AI agent access, and coverage for unstructured data. The post The Best Metadata Management Tools for 2026 appeared first on Everpure Blog.

## BigQuery to Databricks: A Strategic Framework

DevFeed: [BigQuery to Databricks: A Strategic Framework](<https://devfeed.tech/articles/bigquery-to-databricks-a-strategic-framework-102249.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/bigquery-databricks-strategic-framework>)

Author: Takero Ibuki

Published: 2026-10-27T11:07:53Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [academy](<https://devfeed.tech/tags/academy.md>), [account](<https://devfeed.tech/tags/account.md>), [adopting](<https://devfeed.tech/tags/adopting.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assist](<https://devfeed.tech/tags/ai-assist.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-workloads](<https://devfeed.tech/tags/analytics-workloads.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migration](<https://devfeed.tech/tags/migration.md>), [organizations](<https://devfeed.tech/tags/organizations.md>), [platform](<https://devfeed.tech/tags/platform.md>), [range](<https://devfeed.tech/tags/range.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

The guide recommends a phased BigQuery migration in workload-sized waves, starting with an estate assessment and ending with validation and decommissioning. It describes BI-first teams using Lakehouse Federation to let Databricks read BigQuery during dashboard migration; ETL-first teams can move transformations to Databricks while BigQuery serves curated open-format tables, with external-table limitations such as manual schema refresh. Bulk history can be exported as Parquet to Google Cloud Storage, while frequently updated data can use the Storage API connector or federation. For logic, it proposes rule-based SQL transpilation for routine work, LLM assistance for dialect quirks, and engineers for complex cases. Validation spans row counts, schemas and types, and aggregate and row-level comparisons; the text cautions that SQL function differences can affect hashes. It also recommends migrating permissions before data and setting parity criteria to guide cutover. The supplied excerpt frames governance, interoperability, and workforce training as supporting benefits.

### Source excerpt

Migration as a strategic evolutionMany organizations support a range of data and analytics workloads on BigQuery today...

## Cross-Modal Data Co-Locality for Heterogeneous Storage Systems

DevFeed: [Cross-Modal Data Co-Locality for Heterogeneous Storage Systems](<https://devfeed.tech/articles/cross-modal-data-co-locality-for-heterogeneous-storage-systems-101620.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.34595>)

Author: Abdullah Al-Mamun

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [ann](<https://devfeed.tech/topics/ann.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [affinity](<https://devfeed.tech/tags/affinity.md>), [ann](<https://devfeed.tech/tags/ann.md>), [aws](<https://devfeed.tech/tags/aws.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [dram](<https://devfeed.tech/tags/dram.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [graph](<https://devfeed.tech/tags/graph.md>), [indexes](<https://devfeed.tech/tags/indexes.md>), [latency](<https://devfeed.tech/tags/latency.md>), [locality](<https://devfeed.tech/tags/locality.md>), [modal](<https://devfeed.tech/tags/modal.md>), [storage](<https://devfeed.tech/tags/storage.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

Coloc reduces cross-tier data movement and query latency for multimodal vector workloads by treating cross-modal access locality as a storage-layout signal. Its access graph combines semantic affinity, co-access, and temporal behavior; a movement-aware optimizer places frequently co-accessed vectors within capacity and topology constraints, while a migration manager balances locality gains against migration cost and a confidence-gated prefetcher masks remaining slow-tier latency. It operates above existing ANN indexes without changing embeddings, search, or retrieval semantics. On a multi-tier AWS testbed, during a scheduled workload shift, its manager matched the locality of periodic global reorganization while moving 5.9 times less data and finishing 3.9 times faster.

### Source excerpt

arXiv:2610.34595v1 Announce Type: new Abstract: Distributed vector stores increasingly serve multimodal workloads whose representations are jointly accessed but independently partitioned and placed. As a result, cross-modal physical locality remains largely unoptimized. Multimodal queries can incur remote accesses and cross-tier data movement even when each modality's ANN search remains efficient. We present Coloc, a storage-management layer that makes cross-modal access locality a first-class data-layout signal. A Cross-Modal Access Graph combines semantic affinity, co-access, and temporal behavior. A movement-aware optimizer places frequently co-accessed vectors under capacity and topology constraints. A migration manager adapts placement as workloads shift by weighing locality gains against migration cost. A confidence-gated prefetcher hides remaining slow-tier latency. Coloc operates above existing ANN indexes without modifying embeddings, search, or retrieval semantics. We deploy Coloc on a multi-tier AWS testbed spanning GPU, NUMA-DRAM, NVMe, and S3. Coloc substantially reduces data movement and query latency while leaving ANN retrieval unchanged. Under a scheduled workload shift, its migration manager matches the locality of periodic global reorganization while moving 5.9x less data and completing 3.9x faster.

## What Okta learned governing context for AI agents at scale

DevFeed: [What Okta learned governing context for AI agents at scale](<https://devfeed.tech/articles/what-okta-learned-governing-context-for-ai-agents-at-scale-101841.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/okta-ai-agent-context-governance>)

Author: Daniel Poppy

Published: 2026-10-26T22:14:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [context](<https://devfeed.tech/topics/context.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [okta](<https://devfeed.tech/topics/okta.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [context](<https://devfeed.tech/tags/context.md>), [context-for-ai-agents](<https://devfeed.tech/tags/context-for-ai-agents.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [learn](<https://devfeed.tech/tags/learn.md>), [okta](<https://devfeed.tech/tags/okta.md>), [scale](<https://devfeed.tech/tags/scale.md>), [source](<https://devfeed.tech/tags/source.md>), [source-of-truth](<https://devfeed.tech/tags/source-of-truth.md>)

### AI overview

Okta found that governing what AI agents can see and trust was harder than building the agents. Its early Snowflake semantic views were disconnected from dbt's business logic, lacked an external catalog and auditability, and bundled instructions without distinguishing stable knowledge from information to look up. The team moved governance into a three-layer dbt framework: model and column metadata for safe uses, exclusions and automatic filters; Markdown docs for company facts and domain routing; and PR-reviewed semantic views for facts, dimensions and metrics. A custom skill converts existing Snowflake semantic-view DDL into dbt specs. The layers sync to Agents Schema: Claude reads metadata over MCP and docs as Markdown, while Cortex uses a search service and its own skill configuration. Both therefore use the same context through different interfaces, and a merged dbt change reaches consuming agents without separate config edits. Planned work includes regression tests, broader domain coverage and feedback from agent answers into context docs; these are future plans, not reported capabilities.

### Source excerpt

Okta rebuilt how it governs context for AI agents, using dbt as the single source of truth.

## Scaling Data Movement For Physical AI

DevFeed: [Scaling Data Movement For Physical AI](<https://devfeed.tech/articles/scaling-data-movement-for-physical-ai-91875.md>)

Original publisher: [Read original article](<https://semiengineering.com/scaling-data-movement-for-physical-ai/>)

Author: Ed Sperling

Published: 2026-10-21T07:15:31Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [Physical AI](<https://devfeed.tech/topics/physical-ai.md>), [Chip design](<https://devfeed.tech/topics/chip-design.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Cache Coherence](<https://devfeed.tech/topics/cache-coherence.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [arteris](<https://devfeed.tech/tags/arteris.md>), [chip](<https://devfeed.tech/tags/chip.md>), [chiplets](<https://devfeed.tech/tags/chiplets.md>), [coherency](<https://devfeed.tech/tags/coherency.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [physical-ai](<https://devfeed.tech/tags/physical-ai.md>), [semiconductor-engineering](<https://devfeed.tech/tags/semiconductor-engineering.md>), [system-level-design](<https://devfeed.tech/tags/system-level-design.md>), [videos](<https://devfeed.tech/tags/videos.md>)

### AI overview

Physical AI systems must process real-time data across mixed compute elements while making decisions and preserving battery life, so engineers need to decide where data is processed and minimize its movement. Ashley Stevens of Arteris says coherent interconnects are better suited to CPUs, while non-coherent interconnects are better for AI acceleration. The excerpt identifies these design choices as key challenges but provides no further detail on implementation or other challenges.

### Source excerpt

One size does not fit all. The post Scaling Data Movement For Physical AI appeared first on Semiconductor Engineering.

## AI didn't create your IT data problem, it just exposed it

DevFeed: [AI didn't create your IT data problem, it just exposed it](<https://devfeed.tech/articles/ai-didn-t-create-your-it-data-problem-it-just-exposed-it-90844.md>)

Original publisher: [Read original article](<https://www.computerworld.com/article/4237895/ai-didnt-create-your-it-data-problem-it-just-exposed-it.html>)

Author: Anna Boulware

Published: 2026-10-20T17:00:28Z

Content type: article

Language: en

Sources: [Computerworld](<https://devfeed.tech/sources/computerworld.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [asset](<https://devfeed.tech/tags/asset.md>), [but](<https://devfeed.tech/tags/but.md>), [cmdb](<https://devfeed.tech/tags/cmdb.md>), [complete](<https://devfeed.tech/tags/complete.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [configuration-management](<https://devfeed.tech/tags/configuration-management.md>), [console](<https://devfeed.tech/tags/console.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [cybersecurity-research](<https://devfeed.tech/tags/cybersecurity-research.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [desk](<https://devfeed.tech/tags/desk.md>), [devices](<https://devfeed.tech/tags/devices.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exposed](<https://devfeed.tech/tags/exposed.md>), [it](<https://devfeed.tech/tags/it.md>), [just](<https://devfeed.tech/tags/just.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [storytype-brandpost](<https://devfeed.tech/tags/storytype-brandpost.md>)

### AI overview

AI pilots in IT operations can expose fragmented, inconsistent data that already exists across tools and teams. The excerpt illustrates this with conflicting device counts across an endpoint console, scanner, and CMDB; AI outputs can then reflect those discrepancies, while inconsistent ticket histories can teach routing models to repeat them. It also reports research figures linking organizational alignment and AI outcomes, including higher shares of IT professionals saying AI makes work faster and better where it is treated as business-critical. The proposed response is to map disagreements between systems, assign data ownership, agree on shared definitions, automate reconciliation, track data quality continuously, and begin with one use case and its data sources. The aim is trusted, current context, not perfect data.

### Source excerpt

When an AI pilot in IT operations underdelivers, the first instinct is to question the technology. Was the model wrong? Did the tool overpromise? Did the team pick the wrong use case? Those are fair questions. But in many organizations, the answer sits somewhere less glamorous: in a configuration management database that hasn't been reconciled in two years, in asset data in spreadsheets, or in a service desk where three teams use the same category field in three different ways. Transform fragmented IT data into the trusted data foundation AI needs -> AI doesn't create these problems. It surfaces them quickly and publicly. The silo problem nobody owns Most IT environments grew one tool at a time. The endpoint team adopted a management platform. Security brought in a scanner. Service management built workflows in the ITSM tool. Procurement tracked purchases in its own system. Each decision made sense on its own. Few were ever designed to share data with one another. The result is a landscape where every team holds a partial view of the same environment. The endpoint console shows 9,400 devices. The scanner sees 8,700. The CMDB lists 10,200, including hundreds retired long ago. Nobody is exactly wrong, but nobody is complete. People have learned to work around these gaps. An experienced analyst knows which system to trust for which question and fills in the rest from memory. That institutional knowledge is valuable, but it's invisible to AI. And that's where the trouble starts. The gaps are organizational, too Fragmentation isn't only technical. Ivanti's 2026 State of Cybersecurity research found that 48% of security professionals say IT teams don't respond urgently to security concerns, and 40% say IT lacks an understanding of the organization's risk tolerance. Those aren't tooling gaps. They're alignment gaps, and they shape how data is collected, labeled, and maintained across every system AI will eventually draw from. When AI meets these gaps, the outputs reflect th

## Governing Sensitive Data Exports in Snowflake: Data Movement Policies and a Custom Guardrail Layer

DevFeed: [Governing Sensitive Data Exports in Snowflake: Data Movement Policies and a Custom Guardrail Layer](<https://devfeed.tech/articles/governing-sensitive-data-exports-in-snowflake-data-movement-policies-and-a-custom-guardrail-layer-88759.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/governing-sensitive-data-exports-in-snowflake-data-movement-policies-and-a-custom-guardrail-layer-2582dea6fc62?source=rss----34b6daafc07---4>)

Author: Rachel Blum

Published: 2026-10-16T19:01:02Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>), [data loss prevention](<https://devfeed.tech/topics/data-loss-prevention.md>), [Access Control](<https://devfeed.tech/topics/access-control.md>), [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [audit](<https://devfeed.tech/topics/audit.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-trail](<https://devfeed.tech/tags/audit-trail.md>), [block](<https://devfeed.tech/tags/block.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [cells](<https://devfeed.tech/tags/cells.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [custom](<https://devfeed.tech/tags/custom.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [exports](<https://devfeed.tech/tags/exports.md>), [guardrail](<https://devfeed.tech/tags/guardrail.md>), [health-data](<https://devfeed.tech/tags/health-data.md>), [layer](<https://devfeed.tech/tags/layer.md>), [policies](<https://devfeed.tech/tags/policies.md>), [sensitive-data](<https://devfeed.tech/tags/sensitive-data.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

The prototype combines Snowflake Data Movement Policies (DMP) with a configuration-driven SQL export service to govern sensitive health-data exports. DMP rules attached through propagating tags can block or cap movement by channel, alert on activity, allow role-based exceptions, and restrict agents accessing protected data. DMP evaluates row budgets, so it cannot detect risky combinations of fields or suppress groups based on result size. The custom SAFE_EXPORT procedure adds those checks: it validates requested dimensions against a whitelist, blocks configured combinations, aggregates data, suppresses cells below each partner's threshold, and permits staged delivery only for approved partners. An aggregation policy also constrains ad-hoc queries. Both layers feed a unified audit view, but native violation records may take hours to appear, UI download-button enforcement creates no violation record, and rare enforcement events may go unrecorded. The prototype uses synthetic data; DMP requires Snowflake Enterprise Edition or higher. The article also cautions that row caps apply per statement and agents use the user's default role.

### Source excerpt

How to block risky exports, suppress small cells, and keep AI agents away from regulated data, using native Snowflake controls plus a small configuration-driven extension. Export Governance Console -- Policy InventoryIntroduction Organizations that share health-related data with external partners face a familiar tension. The data is valuable precisely because it is granular, yet that same granularity creates re-identification risk once the data leaves the platform. When the data is distributed across dozens of downstream partners, and those partners increasingly use AI agents to explore it, governance can no longer depend on manual review or on trusting each consumer to behave correctly. This article describes a prototype built for a common scenario: a data provider that shares health-related data with a network of external partners. The provider's governance requirements were typical of the industry: Block exports that combine fields in ways that could re-identify individuals. Suppress small counts, with thresholds that may differ by partner (for example, 5, 10, or 20). Apply consistent governance across every partner, from a central governance team. Maintain a complete audit trail of what was blocked, alerted or approved. Allow specific, approved roles to make exceptions. Prevent AI agents from reading sensitive data. The solution combines two layers. The first is Snowflake Data Movement Policies (DMP), a native feature that limits, alerts on or blocks data movement based on rules attached to data through tags. The second is a small custom export service, written in Snowflake SQL, that handles the requirements DMP does not address on its own: field-combination rules and dynamic small-cell suppression. Both layers write to a single audit trail, and a Cortex Agent sits on top of that audit trail to answer governance questions in natural language. All data in the prototype is synthetic. Note: Data Movement Policies require Snowflake Enterprise Edition or higher. Consu

## Governing your lakehouse with Fivetran Managed Data Lake Service | Blog | Fivetran

DevFeed: [Governing your lakehouse with Fivetran Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/governing-your-lakehouse-with-fivetran-managed-data-lake-service-blog-fivetran-136781.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/governing-your-lakehouse-with-fivetran-managed-data-lake-service>)

Author: Andrew Madson

Published: 2026-10-13T16:41:32Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Security, Identity, & Compliance](<https://devfeed.tech/topics/security-identity-compliance.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [athena](<https://devfeed.tech/tags/athena.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [automated](<https://devfeed.tech/tags/automated.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-privatelink](<https://devfeed.tech/tags/aws-privatelink.md>), [azure](<https://devfeed.tech/tags/azure.md>), [biglake](<https://devfeed.tech/tags/biglake.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [governance](<https://devfeed.tech/tags/governance.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>)

### AI overview

The excerpt presents Fivetran Managed Data Lake governance as a single-source-of-truth model: Fivetran alone writes Iceberg metadata, while query engines read it, and Fivetran republishes the correct version if AWS Glue drifts. Engines can use the Iceberg REST Catalog with OAuth2; Athena and Redshift require Glue integration enabled at destination setup, a permanent choice with regional and naming constraints. Access controls include short-lived, table-scoped S3 credentials, cross-account IAM, and optional KMS, Lake Formation, and Business Critical PrivateLink configuration. Automated retention, metadata version cleanup, biweekly orphan-file removal, compaction, deduplication, and schema evolution address lifecycle work. The excerpt warns against manual catalog changes and raw Parquet queries, and notes cloud-specific catalog requirements, including BigQuery Admin access for GCS BigLake.

### Source excerpt

Lakehouse governance requires observability, access control, data lifecycle management, and more. Fivetran MDLS makes it easy.

## Shift Left Complicates Fab Data Management

DevFeed: [Shift Left Complicates Fab Data Management](<https://devfeed.tech/articles/shift-left-complicates-fab-data-management-84109.md>)

Original publisher: [Read original article](<https://semiengineering.com/shift-left-complicates-fab-data-management/>)

Author: Anne Meixner and Laura Peters

Published: 2026-10-08T07:02:25Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [advanced-process-control](<https://devfeed.tech/tags/advanced-process-control.md>), [ai-ml-inference](<https://devfeed.tech/tags/ai-ml-inference.md>), [apc](<https://devfeed.tech/tags/apc.md>), [data](<https://devfeed.tech/tags/data.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [data-veracity](<https://devfeed.tech/tags/data-veracity.md>), [data-volume](<https://devfeed.tech/tags/data-volume.md>), [digital-twins](<https://devfeed.tech/tags/digital-twins.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [equipment-sensors](<https://devfeed.tech/tags/equipment-sensors.md>), [intel-foundry](<https://devfeed.tech/tags/intel-foundry.md>), [lam-research](<https://devfeed.tech/tags/lam-research.md>), [onto-innovation](<https://devfeed.tech/tags/onto-innovation.md>), [pdf-solutions](<https://devfeed.tech/tags/pdf-solutions.md>), [semiconductor](<https://devfeed.tech/tags/semiconductor.md>), [shift-left](<https://devfeed.tech/tags/shift-left.md>), [siemens-eda](<https://devfeed.tech/tags/siemens-eda.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [synopsys](<https://devfeed.tech/tags/synopsys.md>), [test-measurement-and-analytics](<https://devfeed.tech/tags/test-measurement-and-analytics.md>), [top-stories](<https://devfeed.tech/tags/top-stories.md>), [virtual-twins](<https://devfeed.tech/tags/virtual-twins.md>), [yield](<https://devfeed.tech/tags/yield.md>)

### AI overview

Semiconductor fabs are using high-resolution equipment sensor data and AI/ML inference to detect yield problems earlier and improve in-situ process control. The resulting growth in data volume and speed creates infrastructure and data-management challenges. Fabs must preserve context and traceability across process modules so that siloed local optimization does not undermine cross-fab analytics and model quality.

### Source excerpt

Streaming sensor data creates new challenges for advanced process control. The post Shift Left Complicates Fab Data Management appeared first on Semiconductor Engineering.

## NetApp and Nvidia rethink storage for AI factories

DevFeed: [NetApp and Nvidia rethink storage for AI factories](<https://devfeed.tech/articles/netapp-and-nvidia-rethink-storage-for-ai-factories-63646.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/01/netapp-targets-ai-factory-storage-architecture-netappinsight/>)

Author: Chad Wilson

Published: 2026-10-01T18:46:42Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [sharding](<https://devfeed.tech/topics/sharding.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-factories](<https://devfeed.tech/tags/ai-factories.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-ready-data](<https://devfeed.tech/tags/ai-ready-data.md>), [ai-ready-enterprise](<https://devfeed.tech/tags/ai-ready-enterprise.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arindam-banerjee](<https://devfeed.tech/tags/arindam-banerjee.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [autonomous-control-plane](<https://devfeed.tech/tags/autonomous-control-plane.md>), [cloud-integration](<https://devfeed.tech/tags/cloud-integration.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [cyber-resilience](<https://devfeed.tech/tags/cyber-resilience.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [data-mobility](<https://devfeed.tech/tags/data-mobility.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-protection](<https://devfeed.tech/tags/data-protection.md>), [data-resilience](<https://devfeed.tech/tags/data-resilience.md>), [enterprise-storage](<https://devfeed.tech/tags/enterprise-storage.md>), [hybrid-cloud](<https://devfeed.tech/tags/hybrid-cloud.md>), [infrastructure-modernization](<https://devfeed.tech/tags/infrastructure-modernization.md>), [intelligent-automation](<https://devfeed.tech/tags/intelligent-automation.md>), [jason-hardy](<https://devfeed.tech/tags/jason-hardy.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [multicloud](<https://devfeed.tech/tags/multicloud.md>), [netapp](<https://devfeed.tech/tags/netapp.md>), [netapp-insight-2026](<https://devfeed.tech/tags/netapp-insight-2026.md>), [netappinsight](<https://devfeed.tech/tags/netappinsight.md>), [news](<https://devfeed.tech/tags/news.md>), [novus](<https://devfeed.tech/tags/novus.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [storage](<https://devfeed.tech/tags/storage.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

NetApp and Nvidia are developing storage architecture for AI factories that separates data and metadata functions so they can scale independently. The article describes how this is intended to reduce competition between transactional metadata requests and large data transfers, keep GPUs supplied, and handle concurrent access from AI agents. It also discusses API- and SDK-based access for AI teams that are not storage specialists.

### Source excerpt

Storage architecture is being rewritten for artificial intelligence factories. Traditional enterprise storage was designed around workloads that scaled in relatively predictable ways. AI changes that equation by combining heavy data movement with transactional metadata activity, often on shared infrastructure. NetApp Inc. is addressing that pressure through its work with Nvidia Corp., a partnership that dates [...] The post NetApp and Nvidia rethink storage for AI factories appeared first on SiliconANGLE.

## Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables

DevFeed: [Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables](<https://devfeed.tech/articles/your-data-your-storage-your-rules-a-2026-guide-to-storing-unity-catalog-managed-tables-61920.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/your-data-your-storage-your-rules-2026-guide-storing-unity-catalog-managed-tables>)

Author: Elizabeth Bowman; Isabel Dong

Published: 2026-09-29T16:40:16Z

Content type: tutorial

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [s3](<https://devfeed.tech/tags/s3.md>), [storage](<https://devfeed.tech/tags/storage.md>), [unity-catalog](<https://devfeed.tech/tags/unity-catalog.md>)

### AI overview

The guide explains how Databricks Unity Catalog managed tables store data in a cloud account chosen by the organization, while Unity Catalog automates table management. It describes setting storage locations at the metastore, catalog, or schema level, and how external tools can access governed data through open APIs and the Iceberg REST Catalog.

### Source excerpt

Unity catalog managed tables allow you to control the placement of your data when...

## MetaKarta v12 Unifies Enterprise Metadata Management for AI, BI, and Databases

DevFeed: [MetaKarta v12 Unifies Enterprise Metadata Management for AI, BI, and Databases](<https://devfeed.tech/articles/metakarta-v12-unifies-enterprise-metadata-management-for-ai-bi-and-databases-59216.md>)

Original publisher: [Read original article](<https://sdtimes.com/data/metakarta-v12-unifies-enterprise-metadata-management-for-ai-bi-and-databases/>)

Author: SD Times Newswire

Published: 2026-09-24T13:51:30Z

Content type: release

Language: en

Sources: [SD Times](<https://devfeed.tech/sources/sd-times.md>)

Topics: [data-governance](<https://devfeed.tech/topics/data-governance.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lineage](<https://devfeed.tech/tags/data-lineage.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [integration](<https://devfeed.tech/tags/integration.md>), [latest-news](<https://devfeed.tech/tags/latest-news.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [metadata-plus](<https://devfeed.tech/tags/metadata-plus.md>), [metadatsops](<https://devfeed.tech/tags/metadatsops.md>), [newswire](<https://devfeed.tech/tags/newswire.md>)

### AI overview

MITI announced general availability of MetaKarta v12, whose Semantic Hub compiles governed business definitions into databases, BI tools, and AI agents. The release emphasizes shared metadata, lineage, governance, and portable semantic artifacts across enterprise platforms.

### Source excerpt

MOUNTAIN VIEW, Calif. -- Meta Integration Technology, Inc. (MITI) today announced general availability of MetaKarta v12, including Semantic Hub that compiles governed business definitions into the databases, BI tools, and AI agents that depend on them. A definition authored once in MetaKarta lands as a native artifact inside Snowflake, Databricks, Power BI, and Tableau, and... ... continue reading The post MetaKarta v12 Unifies Enterprise Metadata Management for AI, BI, and Databases appeared first on SD Times.

## Managing 3D-IC Design And IP

DevFeed: [Managing 3D-IC Design And IP](<https://devfeed.tech/articles/managing-3d-ic-design-and-ip-59319.md>)

Original publisher: [Read original article](<https://semiengineering.com/managing-3d-ic-design-and-ip/>)

Author: Keith Felton

Published: 2026-09-24T07:05:05Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [3D](<https://devfeed.tech/topics/3d.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [version](<https://devfeed.tech/topics/version.md>), [Software](<https://devfeed.tech/topics/software.md>), [Specifications](<https://devfeed.tech/topics/specifications.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [3d-ics](<https://devfeed.tech/tags/3d-ics.md>), [a-system-perspective](<https://devfeed.tech/tags/a-system-perspective.md>), [blogs-sld](<https://devfeed.tech/tags/blogs-sld.md>), [chiplets](<https://devfeed.tech/tags/chiplets.md>), [components](<https://devfeed.tech/tags/components.md>), [data](<https://devfeed.tech/tags/data.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [defects](<https://devfeed.tech/tags/defects.md>), [design](<https://devfeed.tech/tags/design.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [heterogeneous-integration](<https://devfeed.tech/tags/heterogeneous-integration.md>), [integration](<https://devfeed.tech/tags/integration.md>), [ip](<https://devfeed.tech/tags/ip.md>), [revision-control](<https://devfeed.tech/tags/revision-control.md>), [semiconductor](<https://devfeed.tech/tags/semiconductor.md>), [siemens-eda](<https://devfeed.tech/tags/siemens-eda.md>), [software](<https://devfeed.tech/tags/software.md>), [system-level-design](<https://devfeed.tech/tags/system-level-design.md>), [traceability](<https://devfeed.tech/tags/traceability.md>)

### AI overview

This article explains how design data management, traceability, IP management, and revision control support multi-chiplet 3D-IC heterogeneous integration. It describes the risks of fragmented design data, including defects, delays, and product failure, and discusses a centralized, access-controlled approach to managing design IP through tape-out.

### Source excerpt

Design data management, traceability, and revision control are critical for multi-chiplet heterogeneous integration. The post Managing 3D-IC Design And IP appeared first on Semiconductor Engineering.

## The Best ETL Tools for Real-Time Data Integration in 2026

DevFeed: [The Best ETL Tools for Real-Time Data Integration in 2026](<https://devfeed.tech/articles/the-best-etl-tools-for-real-time-data-integration-in-2026-58720.md>)

Original publisher: [Read original article](<https://hackernoon.com/the-best-etl-tools-for-real-time-data-integration-in-2026?source=rss>)

Author: Pipelineguy

Published: 2026-09-23T07:42:22Z

Content type: comparison

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Database](<https://devfeed.tech/topics/database.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Cloud Dataflow](<https://devfeed.tech/topics/cloud-dataflow.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [batch-processing](<https://devfeed.tech/tags/batch-processing.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-dataflow](<https://devfeed.tech/tags/cloud-dataflow.md>), [compare](<https://devfeed.tech/tags/compare.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [database](<https://devfeed.tech/tags/database.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event-streaming](<https://devfeed.tech/tags/event-streaming.md>), [flink](<https://devfeed.tech/tags/flink.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

A comparison of ETL tools for real-time data integration, explaining how freshness requirements and architectures such as batch processing, micro-batching, change data capture, and event streaming affect tool selection.

### Source excerpt

Real-time doesn't mean the same thing to every ETL tool. Compare Estuary, Confluent, Fivetran, and more to find the right fit for your data.

## Redis Data Integration Adds Multi-Source Pipelines and Plans Multi-Pipeline Support

DevFeed: [Redis Data Integration Adds Multi-Source Pipelines and Plans Multi-Pipeline Support](<https://devfeed.tech/articles/rdi-support-for-federated-caches-sharded-data-sources-and-multiple-pipelines-58227.md>)

Original publisher: [Read original article](<https://redis.io/blog/multi-source-and-multi-pipeline-rdi/>)

Author: Mirko Ortensi

Published: 2026-09-22T00:00:00Z

Content type: release

Language: en

Sources: [Redis Blog](<https://devfeed.tech/sources/redis-blog-2.md>), [Redis Blog](<https://devfeed.tech/sources/redis-blog.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>)

Tags: [database](<https://devfeed.tech/tags/database.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engineering-ai](<https://devfeed.tech/tags/engineering-ai.md>), [federated](<https://devfeed.tech/tags/federated.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [redis](<https://devfeed.tech/tags/redis.md>), [tech](<https://devfeed.tech/tags/tech.md>)

### AI overview

Redis Data Integration now supports multi-source pipelines that combine data from multiple source databases into a single Redis target. The article also describes planned multi-pipeline support for operating multiple RDI pipelines within one integration architecture.

### Source excerpt

Modern applications rarely rely on one database. Data is often distributed across regions, business units, shards, and different technology stacks. Bringing that data together in real time should not require users to build and operate a separate integ...

[Next page](<https://devfeed.tech/topics/data-architecture.md?cursor=WyIyMDI2LTA5LTIyVDAwOjAwOjAwKzAwOjAwIiwgImI1M2MxOWMyLTMzYzAtNDc5Ny1hMTRhLWFjYmJlMjVkOTdlYSJd>)