# data-governance Published articles for data-governance. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Mistral raises EUR 3B to make sovereign, open-weight AI the technology frontier DevFeed: [Mistral raises EUR 3B to make sovereign, open-weight AI the technology frontier]() Original publisher: [Read original article]() Published: 2026-09-08T12:00:22Z Content type: news Language: en Sources: [Mistral AI Blog]() Topics: [AI Chat](), [data-governance](), [samsung]() Tags: [ai](), [data-governance](), [generative-ai](), [open](), [samsung]() ### AI overview Mistral announced a EUR 3 billion Series D round to expand frontier AI research, compute capacity, infrastructure, and international commercial growth. The article presents its open-weight, full-stack approach as a way for organizations to retain control over data, models, compute, and production systems. ### Source excerpt Mistral today announced that it has raised EUR 3 billion in a Series D funding round at a post-money valuation of more than EUR 21 billion. ## ISO 42001 Evidence: What Auditors Ask For DevFeed: [ISO 42001 Evidence: What Auditors Ask For]() Original publisher: [Read original article]() Author: info@goteleport.com (Preet Dhatt) Published: 2026-09-03T00:00:00Z Content type: article Language: en Sources: [Teleport]() Topics: [audit](), [AI Development](), [Deployment](), [CI/CD](), [data-governance](), [Monitoring](), [Requirements]() Tags: [ai](), [ai-development](), [article](), [audit](), [ci-cd](), [data-governance](), [deployment](), [eu](), [iso](), [monitoring](), [requirements]() ### AI overview This article explains what auditors look for in ISO 42001 evidence and how organizations can connect access, approvals, infrastructure, CI/CD activity, releases, and production changes into an evidence trail. It also discusses monitoring, human oversight, third-party access, and data governance within an AI Management System. ### Source excerpt Prepare ISO 42001 audit evidence that proves your controls work in practice. ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## Navigating SAMA, ADGM & DFSA Requirements with Teleport DevFeed: [Navigating SAMA, ADGM & DFSA Requirements with Teleport]() Original publisher: [Read original article]() Author: info@goteleport.com (Mukund Cadambi) Published: 2026-08-12T00:00:00Z Content type: article Language: en Sources: [Teleport]() Topics: [Requirements](), [Cybersecurity](), [Cloud](), [audit trail](), [data-governance](), [Architecture & Design]() Tags: [audit-trail](), [cloud](), [compliance](), [cybersecurity](), [data-governance](), [requirements](), [security]() ### AI overview This article explains how SAMA, ADGM, and DFSA requirements affect cloud service providers operating in Saudi Arabia and the UAE. It describes challenges involving data sovereignty, cybersecurity, identity and access controls, privileged and third-party access, and audit evidence, and presents a unified identity layer as an approach to operationalizing compliance. ### Source excerpt Learn how Teleport helps CSPs meet SAMA, ADGM, and DFSA compliance. ## Four pillars of agentic AI success DevFeed: [Four pillars of agentic AI success]() Original publisher: [Read original article]() Author: Simon Sear Published: 2026-07-29T15:47:00Z Content type: article Language: en Sources: [Scott Logic]() Topics: [Artificial Intelligence](), [Software Engineering](), [data](), [legacy](), [systems](), [context]() Tags: [accessibility](), [agentic-ai](), [ai](), [ai-accelerated-development](), [ai-agents](), [architecture](), [artificial-intelligence](), [context](), [data](), [data-architecture](), [data-governance](), [data-strategy](), [delivery](), [engineering](), [governance](), [legacy](), [legacy-modernisation](), [software-engineering](), [systems]() ### AI overview This article argues that organizations need strong foundations before scaling agentic AI. It identifies four pillars: modernizing legacy systems, building reliable data foundations and architecture, managing governance and risk, and democratizing access to data. ### Source excerpt Everyone is talking about agentic AI. That's not surprising. The promise is huge: AI agents that can plan, reason, use tools, work across systems and get real work done. In software engineering, that could mean faster delivery and better quality. In operations, it could mean complex processes moving with less manual effort, fewer handovers and better decisions. ## Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more. DevFeed: [Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more.]() Original publisher: [Read original article]() Author: Premika Srinivasan Published: 2026-07-29T15:00:10Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [apache-flink](), [Kafka](), [Monitoring](), [data-governance](), [Model Context Protocol](), [Claude Code](), [codex]() Tags: [apache-flink](), [apache-kafka](), [claude-code](), [codex](), [confluent](), [confluent-platform](), [data-governance](), [mcp](), [mcp-server](), [monitoring]() ### AI overview Confluent Platform 8.3.0, built on Apache Kafka 4.3.0, adds simplified Apache Flink SQL operations, expanded monitoring through Unified Stream Manager, easier KRaft migration, and structured governance for data in motion. The release also introduces an MCP server for Confluent Platform for Apache Flink, allowing AI agents such as Claude Code and Codex to inspect and manage Flink resources through existing security controls. ### Source excerpt Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more ## AWS Weekly Roundup: Local Zone in Athens, Claude Opus 5 on AWS, Lambda durable execution for .NET, and more (July 27, 2026) DevFeed: [AWS Weekly Roundup: Local Zone in Athens, Claude Opus 5 on AWS, Lambda durable execution for .NET, and more (July 27, 2026)]() Original publisher: [Read original article]() Author: Daniel Abib Published: 2026-07-27T14:54:41Z Content type: news Language: en Sources: [AWS News Blog]() Topics: [Amazon Web Services](), [AWS Local Zones](), [Latency](), [Amazon Bedrock](), [Amazon EC2](), [Amazon S3](), [Claude](), [cloud-infrastructure](), [Amazon Elastic Container Service](), [anthropic](), [data-governance]() Tags: [amazon-bedrock](), [amazon-bedrock-agentcore](), [amazon-connect](), [amazon-ec2](), [amazon-s3](), [amazon-sagemaker-ai](), [announcements](), [anthropic](), [aws](), [aws-lambda](), [aws-local-zones](), [claude](), [data-governance](), [latency](), [news](), [week-in-review]() ### AI overview An AWS weekly roundup covering a new Local Zone in Athens, Greece, and selected AWS launches and updates, including Claude Opus 5 on Amazon Bedrock. The article highlights local data processing, data residency, low latency, and infrastructure for regional workloads. ### Source excerpt Last week I had the privilege of spending three days in São Paulo with technical builders from across Latin America, brought together for a regional tech event full of deep-dive sessions, hands-on workshops, and conversations with customers and partners. What struck me most wasn't any single session, it was the energy of a technical community [...] ## The data platform 1Password needed didn't exist. So we built it. DevFeed: [The data platform 1Password needed didn't exist. So we built it.]() Original publisher: [Read original article]() Author: info@1password.com (Wayne Duso; Mandy Gu; and Amie Bright) Published: 2026-07-22T00:00:00Z Content type: article Language: en Sources: [Blog on 1Password Blog]() Topics: [data](), [Data Infrastructure](), [data-governance](), [real-time](), [Unified Access](), [Artificial Intelligence]() Tags: [agentic](), [ai](), [building-1password](), [data](), [data-governance](), [data-infrastructure](), [developers](), [real-time](), [sql](), [systems](), [unified-access](), [workflows]() ### AI overview 1Password describes building an internal data platform to make business data accessible, trustworthy, and available in real time across product, finance, engineering, analytics, SQL, APIs, dashboards, and AI workflows. The effort addresses bottlenecks caused by a centralized data lake, bespoke pipelines, and tightly coupled storage, governance, and compute. ### Source excerpt Consider a few tasks that take place across every business, every day: A product team ships a feature and wants to know if customers are using it successfully. A finance team needs customer and account information for planning. An analyst needs definitions to create a report. An AI assistant needs operational context to answer a business question. Those sound like different workflows, but they all rely on the same underlying data. And each one of these actors, across each of these teams, needs that data to be both accessible and trustworthy. At 1Password, trust is at the center of everything we build. Millions of people and businesses rely on us to protect the credentials, secrets, and access workflows that power modern work. The same principle applies to our own internal data. As our products, systems, and use of AI evolved, data became a shared dependency across the business. It powers everything from Unified Access andsecure agentic access patterns for customers, to the workflows used by product, finance, and engineering teams. As those systems grew, so did the number of people and applications that depended on our internal data. But our data infrastructure did not respond well to this. We had built a centralized data lake supported by a growing collection of bespoke data pipelines and one-off solutions. Each new use case required another integration or transformation. Over time, the data platform became a bottleneck: teams turned to CSVs to move faster, and data engineers spent more time maintaining pipelines than enabling new capabilities. What we learned was that manually moving data was no longer enough. Data needs to be available in real time, accessible wherever it's needed, and trusted through the forms our customers need, whether that is through SQL, APIs, dashboards or AI workflows. Privacy and governance need to be built into data the moment it's created so every downstream use remains secure and unambiguous by design. This is the foundation we're build ## Announcing Upvest as Confluent's 2026 EMEA Data Streaming Startup of the Year DevFeed: [Announcing Upvest as Confluent's 2026 EMEA Data Streaming Startup of the Year]() Original publisher: [Read original article]() Author: Tim Graczewski Published: 2026-07-16T00:07:01Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Streaming](), [real-time](), [Resilience](), [event driven](), [Scalability](), [Cloud](), [data-governance](), [Microservices](), [Architecture & Design](), [Disaster Recovery]() Tags: [2026](), [cloud](), [cloud-native](), [compliance](), [confluent](), [confluent-cloud](), [data](), [data-governance](), [disaster-recovery](), [emea](), [event-driven](), [fintech](), [infrastructure](), [microservices](), [real-time](), [resilience](), [revolut](), [scalability](), [startup](), [streaming]() ### AI overview Confluent recognizes Berlin-based Upvest as its inaugural EMEA Data Streaming Startup of the Year. The article describes Upvest's real-time, event-driven fintech infrastructure, which supports embedded investments, client onboarding, data governance, disaster recovery, scalability, and regulatory resilience for major financial institutions. ### Source excerpt Confluent for Startups provides an easy on-ramp to Confluent Cloud for early stage startups with great data streaming use cases. ## Prepare for the EU AI Act with Harness AI Security DevFeed: [Prepare for the EU AI Act with Harness AI Security]() Original publisher: [Read original article]() Author: Vikas Gautam Published: 2026-07-02T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [Securing AI](), [Security](), [Responsibility & Safety](), [Model Context Protocol]() Tags: [ai](), [ai-security](), [auditability](), [compliance](), [data-governance](), [mcp](), [security](), [shadow-ai](), [vulnerabilities]() ### AI overview Harness describes how its AI Security platform supports EU AI Act compliance through AI asset discovery, risk classification, data-flow visibility, runtime protection, auditability, and continuous monitoring. ### Source excerpt Learn how Harness AI Security helps organizations meet EU AI Act requirements with AI asset discovery, risk classification, runtime protection, auditability, an | Blog ## Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions DevFeed: [Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions]() Original publisher: [Read original article]() Author: Iraklikhorguani Published: 2026-06-10T18:42:08Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [data]() Tags: [ai-and-mcp](), [data](), [data-governance](), [data-quality](), [knowledge-base](), [lyft](), [metric-standardization](), [metrics](), [python](), [semantic-layer](), [sql]() ### AI overview Lyft describes an internal Metric Semantic Layer that centralizes metric definitions, metadata, and SQL to maintain consistent terminology, governance, and downstream use. ### Source excerpt Written by Rohit Channe and Simran Mirchandani at Lyft. Motivation At Lyft, data isn't just a resource -- it's woven into everything we do. Metrics drive key forecasts, steer operational decisions, and put our boldest hypotheses to the test. But as Lyft scaled, products launched and evolved, and team members came and went, we found ourselves at risk of different teams using different definitions for a given metric. What did "Metric ABC" actually mean? The answer often depended on the context and application of the team you asked. The consequences were predictable. Without centralized version control or a shared standard, outdated metric definitions crept into decision-making. Our solution was to build an internal Metric Semantic Layer (MSL): a centralized repository that serves as a single, authoritative home for every metric's definition -- providing both a clear, plain-English description and the definitive SQL code. No more hunting across codebases or tribal knowledge -- just one place to store and access a standardized, agreed-upon definition. With MSL, we have a single source of truth -- consistent terminology and assumptions across every team, so everyone is genuinely speaking the same language. We achieve this through three key principles: Simplified onboarding and change management -- update a metric definition once, and the change automatically and frictionlessly flows through every downstream application that depends on it Intentional governance -- clarified ownership, defined scope, clear accountability for data quality, and a structure resilient enough to survive org changes, team rotations, and attrition Transparency and accessibility -- definitions are easy for both technical and non-technical users (and downstream applications) to find and integrate into day-to-day workflows Solution Taking the above principles into account, we implemented the Metrics Semantic Layer as a Python package: 1 -- Simplified onboarding and change management through flexible metric ## Introducing the OpenAI Economic Research Exchange DevFeed: [Introducing the OpenAI Economic Research Exchange]() Original publisher: [Read original article]() Published: 2026-06-08T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [OpenAI](), [data-governance](), [jobs](), [data]() Tags: [ai](), [company](), [data](), [data-governance](), [economics](), [external](), [jobs](), [openai](), [research]() ### AI overview OpenAI is launching the Economic Research Exchange, a platform for structured collaborations with external researchers studying AI's economic effects on workers, firms, institutions, and the broader economy. Selected projects may use privacy-protected OpenAI tools and datasets under defined milestones, data governance, and review processes. ### Source excerpt OpenAI launches the Economic Research Exchange to study AI's impact on jobs, productivity, and the economy. Applications are now open for selected research projects. ## EU AI Act Compliance: Requirements, Risks, and What to Document DevFeed: [EU AI Act Compliance: Requirements, Risks, and What to Document]() Original publisher: [Read original article]() Author: info@goteleport.com (Kayne McGladrey, CISSP) Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [Artificial Intelligence](), [Requirements](), [Documentation](), [audit](), [data-governance](), [Logging](), [Monitoring](), [Deployment](), [data]() Tags: [ai](), [compliance](), [cybersecurity](), [data](), [data-governance](), [dataset](), [deployment](), [development](), [documentation](), [eu](), [iso-27001](), [lifecycle](), [logging](), [monitoring](), [nist](), [real-time](), [reproducibility](), [soc-2](), [versioning]() ### AI overview A practical guide to EU AI Act compliance covering requirements, risks, technical documentation, logging, data governance, lifecycle evidence, traceability, monitoring, and human oversight. It outlines key compliance milestones from August 2025 through August 2027. ### Source excerpt Cut through EU AI Act complexity with practical guidance on requirements, risks, and documentation. ## CockroachDB v26.1 Adds Security and Compliance Features DevFeed: [CockroachDB v26.1 Adds Security and Compliance Features]() Original publisher: [Read original article]() Author: David Bressler Published: 2026-02-03T00:00:00Z Content type: release Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Security](), [data-governance](), [Zero Trust](), [Cloud](), [Databases](), [JSON Web Tokens](), [OpenID connect (OIDC)](), [Self-hosted](), [Azure]() Tags: [azure](), [cloud](), [cockroachdb](), [compliance](), [data-governance](), [database](), [encryption](), [jwt](), [oidc](), [security](), [self-hosted](), [zero-trust]() ### AI overview CockroachDB v26.1 introduces security and compliance improvements for CockroachDB Cloud and self-hosted deployments. The release includes data governance and zero-trust access controls for AI agents, expanded HIPAA and PCI/DSS compliance participation on Azure, and identity integrations using JWT and OpenID Connect. ### Source excerpt Enterprise security is a key factor in reducing cost-of-ownership and getting new solutions into production efficiently. With CockroachDB v26.1, available as of February 3rd in CockroachDB Cloud and February 18th for CockroachDB self-hosted, we are introducing security improvements that help organizations integrate CockroachDB more seamlessly with their existing security infrastructure. ## How to optimize real-time data ingestion in Snowflake and Iceberg DevFeed: [How to optimize real-time data ingestion in Snowflake and Iceberg]() Original publisher: [Read original article]() Author: Sesethu Mhlana Published: 2026-01-21T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [real-time](), [Apache Iceberg](), [Streaming](), [Kafka](), [data-governance](), [systems]() Tags: [cost](), [data](), [data-governance](), [high-availability](), [infrastructure](), [integration](), [kafka](), [monitoring](), [performance](), [pipelines](), [real-time](), [schema](), [storage](), [streaming]() ### AI overview This guide explains how architectural inefficiencies in real-time data ingestion can increase Snowflake credits, storage costs, and query delays. It focuses on streaming pipelines involving Apache Iceberg and describes the infrastructure overhead of Kafka deployments, including brokers, coordination services, connectors, schema registries, and monitoring. ### Source excerpt Learn about the hidden costs of real-time data ingestion and practical strategies to optimize your streaming infrastructure. ## Streaming IoT and event data into Snowflake and ClickHouse DevFeed: [Streaming IoT and event data into Snowflake and ClickHouse]() Original publisher: [Read original article]() Author: Mdu Sibisi Published: 2025-12-09T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Internet of things](), [Streaming](), [Redpanda-Connect](), [clickhouse](), [data observability](), [Compression](), [data-governance](), [DataOps](), [Database]() Tags: [clickhouse](), [cloud-storage](), [compliance](), [data](), [data-governance](), [event](), [iot](), [pipelines](), [redpanda-connect](), [streaming]() ### AI overview A guide to streaming IoT and event data through Redpanda and Redpanda Connect into Snowflake and ClickHouse. It compares ClickHouse for real-time analysis with Snowflake for scalable cloud storage, historical reporting, and querying, while discussing governance, compression, data freshness, and robust pipeline design. ### Source excerpt Learn how to stream IoT and event data into Snowflake and ClickHouse using Redpanda ## AI in South Korea--OpenAI's Economic Blueprint DevFeed: [AI in South Korea--OpenAI's Economic Blueprint]() Original publisher: [Read original article]() Published: 2025-10-23T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [AI Chat]() Tags: [ai](), [ai-adoption](), [ai-data-centers](), [company](), [data-governance](), [foundation-models](), [frontier-ai](), [gpu](), [openai](), [partnerships](), [policy](), [samsung](), [stargate]() ### AI overview OpenAI's Economic Blueprint for South Korea proposes scaling trusted AI adoption through sovereign capabilities and strategic collaboration with frontier AI developers. ### Source excerpt OpenAI's Korea Economic Blueprint outlines how South Korea can scale trusted AI through sovereign capabilities and strategic partnerships to drive growth. ## Hijacking Amazon EventBridge for launching Cross-Account attacks DevFeed: [Hijacking Amazon EventBridge for launching Cross-Account attacks]() Original publisher: [Read original article]() Author: Ramesh Ramani Published: 2025-06-25T07:00:00Z Content type: article Language: en Sources: [Square Corner Blog RSS Feed]() Topics: [Amazon EventBridge](), [Security](), [Amazon Web Services](), [Architecture & Design](), [Serverless](), [AWS IAM](), [Exploit]() Tags: [amazon-eventbridge](), [aws](), [aws-eventbridge](), [communication](), [data-governance](), [data-loss-prevention](), [engineering](), [exploit](), [iam](), [security](), [serverless]() ### AI overview This article examines how misconfigured cross-account Amazon EventBridge flows can create inbound and outbound security risks. It describes possible event injection, triggering of vulnerable processing logic, and data exfiltration, and provides guidance for securing these configurations. ### Source excerpt Securing the invisible paths: How cross-account event flows can become security blind spots ## Announcing General Availability of ClickHouse BYOC (Bring Your Own Cloud) on AWS DevFeed: [Announcing General Availability of ClickHouse BYOC (Bring Your Own Cloud) on AWS]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-02-20T00:00:00Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Amazon VPC](), [Amazon Web Services](), [Cloud](), [Deployment](), [autoscaling](), [data-governance]() Tags: [availability](), [aws](), [byoc-deployment](), [clickhouse](), [cloud](), [cloud-native](), [compliance](), [data-governance](), [scale](), [security](), [vpc]() ### AI overview The article announces the general availability of ClickHouse BYOC on AWS, a fully managed ClickHouse Cloud service deployed in a customer's own AWS account and VPC. Customer data, compute, storage, cache, and backups remain within the customer's VPC, supporting security, compliance, governance, workload isolation, scaling, and lower infrastructure costs. ### Source excerpt Today, we announce the GA of ClickHouse BYOC on AWS. A fully managed ClickHouse Cloud service deployed in your own AWS account. Designed for large-scale deployments, with personalized support and onboarding. SOC 2 and ISO 27001 aligned. ## Data Inventory DevFeed: [Data Inventory]() Original publisher: [Read original article]() Author: Ritesh Modi Published: 2025-02-19T14:24:14Z Content type: tutorial Language: en Sources: [Wise Engineering - Medium]() Topics: [data-governance](), [data-architecture](), [Security & Privacy](), [Databases](), [Provisioning](), [Terraform]() Tags: [backend](), [data](), [data-governance](), [databases](), [frontend](), [governance](), [least-privilege](), [network-segmentation](), [platform](), [provisioning](), [security](), [security-privacy](), [terraform]() ### AI overview Wise's Data Governance team describes Data Inventory, a catalog of data assets that records their locations and security or privacy controls. The article explains the system's constraints, design outcomes, and process for discovering operational databases and their metadata. ### Source excerpt The Data Governance team, part of Data Platform, develops and operates platform products through which autonomous teams can govern their data. The team's vision is to democratise data in a secure and compliant manner. Data Inventory was the team's first product. What is Data Inventory? A data inventory is an extensive catalog of the Wise's data assets. It helps us understand where the data is located and what kind of security / privacy controls are in place. This is also a requirement for ISO 27001:2022. What are the constraints? Any system has to work well with our controls and way-of-working. This led to following constraints: It shall be built in a way that satisfies our security controls such as network segmentation, least privilege access, secure secret management, etc. It shall be easy to integrate with other in-house platforms to provide cohesive experience. It shall be built to connect with heterogeneous and fragmented infrastructure. Many data systems are built through off-the-shelf or managed service offering but others are built & operated in-house. The system shall be integrated by default with technology provided by Data Platform. On another hand, it shall be extensible enough to integrate with data systems not maintained by Data Platform. How did we build our inventory? At high level, we split this into 4 main outcomes: 1. Register assets at correct granularity. 2. Identify the correct owner and make them accountable for the life cycle. 3. Extract the schema of the data asset. 4. Classify the asset with correct sensitivity. This is how design looks at high level: To achieve the above design and outcome, we follow the following steps: Step 0: Discover data systems network metadata Scanners need to be aware of the network metadata before doing scanning. For example: - What is the host or connection string? - What is the technology of the data system? - What is the name of the data system? Let us take an example of how we solve it in our operational datab ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## Lakehouse summer camp at Trino Fest 2023 DevFeed: [Lakehouse summer camp at Trino Fest 2023]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2023-04-05T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [data-architecture](), [Query (disambiguation)](), [data-governance](), [SQL](), [Data Space](), [Apache Iceberg]() Tags: [analytics](), [collaboration](), [community](), [conference](), [data-governance](), [data-lake](), [education](), [event](), [latest-features](), [sql](), [tools-and-integrations](), [trends]() ### AI overview Trino Fest 2023 is a virtual two-day event focused on Trino as a data lakehouse query engine and on improving data lakehouse management. The event emphasizes education, community collaboration, and practical insights into Trino, its ecosystem, and data and analytics platforms. ### Source excerpt Get ready to kick off your summer with Commander Bun Bun at Trino Fest 2023! This year's event is going virtual and will take place over two days, the 14th and 15th of June. The focus of the event will be on Trino as a data lakehouse query engine, with discussions on how new features and the ecosystem around Trino can support better data lakehouse management. Trino Fest 2023 is the new annual summer event dedicated to all things Trino. Building on the success of last year's Cinco de Trino, we're excited to bring the community together once again to explore the latest trends and innovations in Trino and data lakehouse management. With a focus on education, community collaboration, and inspiration, Trino Fest 2023 will be a valuable experience for anyone interested in improving their data and analytics platform. We hope to see you there as attendee, speaker, or sponsor! Read below to find out how to sign up. ## Federating them all on Starburst Galaxy DevFeed: [Federating them all on Starburst Galaxy]() Original publisher: [Read original article]() Author: Monica Miller Published: 2022-12-14T00:00:00Z Content type: tutorial Language: en Sources: [Trino Blog]() Topics: [data](), [Amazon S3](), [CSV]() Tags: [architecture](), [building](), [data](), [data-governance](), [s3]() ### AI overview A recap of a Trino Summit session that builds a Starburst Galaxy data lakehouse. It ingests Pokémon encounter data from S3, combines it with Pokédex data in MongoDB, optimizes the data into ORC files, applies role-based access control, and creates visualizations. ### Source excerpt As the Trino Summit 2022 recap post series continues on, I have been reading all the wonderful posts by our awesome speakers, facilitated by the Trino developer relations team. Because I have a perpetual fear of missing out, I convinced them that I should get in on the fun. For this latest installment in the series, I will be recapping my very own Trino Summit talk. Basically, I'm ripping off Bo Burnham's comedy bit where he reacts to his own reaction video, blog style. In this session, I demonstrate building a data lakehouse architecture with Starburst Galaxy, the fastest and easiest way to get up running with Trino. Before I dive into the recap, I want to thank the Trino community for showing up. I am grateful that I was able to meet and learn from so many members of the community in person. ## My first weeks at Acer Europe DevFeed: [My first weeks at Acer Europe]() Original publisher: [Read original article]() Published: 2022-10-16T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Azure](), [dashboards](), [Databases](), [etl](), [DevOps](), [Purview](), [active directory](), [data-governance]() Tags: [active-directory](), [azure](), [dashboards](), [data-governance](), [databases](), [developer](), [devops](), [etl](), [purview]() ### AI overview A Business Intelligence Developer reflects on their first weeks at Acer Europe, describing the transition from a small company to a large IT organization. The post covers the scale of its data, reporting, ETL, Azure services, and cross-team work, along with the information overload and learning challenges of the first week. ### Source excerpt This is a post for my future self when I will be asking myself: "How were my first days at Acer like?". Note: I'm working at Acer EMEA HQ,... [Next page]()