# Data validation Published articles for Data validation. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Starlette vs FastAPI: what FastAPI actually adds DevFeed: [Starlette vs FastAPI: what FastAPI actually adds]() Original publisher: [Read original article]() Author: Farhan Hasin Chowdhury Published: 2026-07-20T07:00:00Z Content type: comparison Language: en Sources: [Honeybadger]() Topics: [FastAPI](), [Pydantic](), [ASGI](), [OpenAPI Specification](), [Dependency injection](), [Python](), [WebSocket](), [Cross-origin resource sharing (CORS)]() Tags: [cors](), [data-validation](), [dependency-injection](), [fastapi](), [frameworks](), [openapi](), [python](), [python-articles](), [starlette](), [websocket]() ### AI overview This comparison explains how FastAPI builds on Starlette and Pydantic. Starlette provides the ASGI-based HTTP layer, while Pydantic handles typed data validation; FastAPI adds type-driven parameter parsing, dependency injection, and automatic OpenAPI documentation. It also discusses when using raw Starlette may be preferable. ### Source excerpt FastAPI is built on Starlette, but most developers never look at what's underneath. Learn what FastAPI actually adds on top of Starlette and Pydantic, what comes straight from Starlette, and when dropping down to raw Starlette makes more sense than pulling in the full stack. ## Structured outputs with Pydantic AI DevFeed: [Structured outputs with Pydantic AI]() Original publisher: [Read original article]() Author: Ed Berry Published: 2026-03-24T10:34:08Z Content type: tutorial Language: en Sources: [FreeAgent]() Topics: [Large Language Model](), [JSON Schema](), [Prompt Engineering](), [Pydantic](), [Python](), [Structured-data]() Tags: [ai](), [data](), [data-ml](), [data-science](), [data-validation](), [genai](), [json-schema](), [llms](), [prompt-engineering](), [python](), [schema](), [structured-output]() ### AI overview This article explains structured outputs for LLMs using Pydantic AI. It introduces the relationship between Python type hints, static analysis, Pydantic, and runtime data validation, while noting that Pydantic AI is model-agnostic. ### Source excerpt One of the challenges of working with LLMs is getting them to respond with a consistent format, such as a given JSON schema. Anyone who has tried to solve this issue with prompt engineering knows how frustrating it can be. You add a 'MUST' here and an 'always return JSON' there, but still the output [...] ## Moving from Django DRF to Ninja API / Pydantic DevFeed: [Moving from Django DRF to Ninja API / Pydantic]() Original publisher: [Read original article]() Author: Michael Sidharta Published: 2025-11-10T05:35:43Z Content type: comparison Language: en Sources: [Kogan.com]() Topics: [Django](), [Pydantic](), [API](), [Python](), [Development](), [OpenAPI Specification](), [Swagger]() Tags: [api-documentation](), [data-validation](), [django](), [fastapi](), [python](), [swagger]() ### AI overview This article examines moving from Django REST Framework API patterns to Django Ninja API and Pydantic. It describes motivations including reducing boilerplate, improving performance for some use cases, using modern Python type hints and data validation, and enhancing developer experience. It also outlines Django Ninja's type-based validation, automatic OpenAPI documentation, performance focus, and simplified endpoint definitions. ### Source excerpt As our project grows, we're always looking for ways to streamline development, improve performance, and enhance the developer experience. Recently, we've been exploring a shift from our traditional Django REST Framework (DRF) API patterns to a combination of Django Ninja API and Pydantic. This blog post will delve into our motivations for this change, the benefits we've observed, and some considerations for others contemplating a similar transition. Why Consider a Change from Django DRF? Django REST Framework has been a robust and widely adopted solution for building APIs with Django. It provides a comprehensive set of tools, including serializers, viewsets, and excellent browser-based API interfaces. However, as our needs evolved, we identified areas where a different approach could offer advantages: Boilerplate Code: While DRF offers powerful abstractions, creating serializers, views, and viewsets can sometimes lead to a significant amount of boilerplate code, especially for simpler APIs. Performance: For certain use cases, the overhead of DRF's serializer validation and rendering can impact performance, particularly in high-throughput scenarios. Modern Python Features: We were keen to leverage modern Python features like type hints and data validation more extensively, which are core to Pydantic. Developer Experience: A more concise and explicit way to define API endpoints and data structures could improve developer productivity and reduce potential errors. Introducing Django Ninja API and PydanticDjango Ninja API Django Ninja is a web framework for building APIs with Django and Python 3.6+ type hints. It's heavily inspired by FastAPI and offers a number of compelling features: Type Hinting for API Endpoints: You define your request and response models using Pydantic, and Ninja automatically validates and serializes the data based on these type hints. Automatic OpenAPI (Swagger) Documentation: Just like FastAPI, Ninja generates interactive API documentation out o ## JSON - The Fine Print: Part 3 - Zero vs Missing Values DevFeed: [JSON - The Fine Print: Part 3 - Zero vs Missing Values]() Original publisher: [Read original article]() Published: 2025-05-19T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [JSON](), [Go Language](), [Code]() Tags: [api-layers](), [code](), [common-serialization-errors](), [data-serialization](), [data-structure-separation](), [data-transmission](), [data-validation](), [encoding-json](), [go](), [go-json-types](), [go-language-serialization](), [go-language-validation](), [go-type-mapping](), [json](), [json-best-practices](), [json-go-mapping](), [json-marshaling](), [json-unmarshaling](), [json-validation](), [passing-serialized-data](), [rest-apis](), [schema-less-data-format](), [serialization](), [serialization-mistakes](), [serialization-protocols](), [unmarshalling]() ### AI overview This tutorial examines how to distinguish zero values from missing or NULL fields when consuming JSON in Go. It explains basic unmarshalling into structs, notes that absent fields do not cause errors, and introduces DisallowUnknownFields for validating exact JSON input. ### Source excerpt Introduction In part 1 we took a higher level view on serialization in general and JSON in specific. In part 2 we looked at emitting JSON. In part 3, we'll look at an issue you might encounter when consuming JSON, Zero vs NULL field values. To clarify the definition of NULL, this means the absence of value. So here is the question: Given a field in a Go struct set to its zero value, how do you know that zero value was set by the user or it's zero because it was never provided? ## JSON - The Fine Print: Part 2 - Emitting JSON DevFeed: [JSON - The Fine Print: Part 2 - Emitting JSON]() Original publisher: [Read original article]() Published: 2025-02-04T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [JSON](), [API](), [Code](), [Structured-data]() Tags: [api](), [api-layers](), [code](), [common-serialization-errors](), [data](), [data-serialization](), [data-structure-separation](), [data-transmission](), [data-validation](), [encoding](), [encoding-json](), [function](), [go-json-types](), [go-language-serialization](), [go-language-validation](), [go-type-mapping](), [http](), [io](), [io-writer](), [json](), [json-best-practices](), [json-go-mapping](), [json-marshaling](), [json-unmarshaling](), [json-validation](), [map](), [passing-serialized-data](), [properties](), [rest-apis](), [schema](), [schema-less-data-format](), [serialization](), [serialization-mistakes](), [serialization-protocols](), [standard](), [structure](), [writing]() ### AI overview This tutorial explains how to emit JSON in Go using the encoding/json package. It compares json.Marshal, which returns bytes, with json.Encoder, which writes to an io.Writer, and discusses error handling in HTTP handlers. It also demonstrates marshaling maps and structs and using field tags to control JSON property names. ### Source excerpt Introduction In part 1 we took a high-level view on serialization and JSON. In this part, we'll roll our sleeves and start working with JSON, focused on emitting JSON. You might think this is a basic topic, but there is much more to it than just calling json.Marshal. json.Marshal vs json.Encoder The encoding/json package has two main APIs: Marshal and NewEncoder. The Marshal function returns a []byte while the NewEncoder function will write to an io.Writer. The question is: When should you use one API over the other? ## JSON - The Fine Print: Part 1 DevFeed: [JSON - The Fine Print: Part 1]() Original publisher: [Read original article]() Published: 2024-10-23T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [JSON](), [API]() Tags: [api-layers](), [apis](), [common-serialization-errors](), [data-serialization](), [data-structure-separation](), [data-transmission](), [data-validation](), [database](), [elasticsearch](), [encoding-json](), [go-json-types](), [go-language-serialization](), [go-language-validation](), [go-type-mapping](), [json](), [json-best-practices](), [json-go-mapping](), [json-marshaling](), [json-unmarshaling](), [json-validation](), [passing-serialized-data](), [rest-apis](), [schema-less-data-format](), [sensitive-data](), [serialization](), [serialization-mistakes](), [serialization-protocols]() ### AI overview This article explains JSON and serialization fundamentals, then examines common mistakes. It recommends serializing data at program boundaries and using separate data structures for API, business, and storage layers to avoid inefficiency, coupling, and accidental exposure of sensitive data. ### Source excerpt Introduction Everybody knows JSON, it's a simple serialization format and the default format for REST APIs. Like many other topics, there are fine points you should know in order to work with JSON more effectively and avoid common mistakes. In this article we'll explore some big picture aspects and some low level details of using JSON. Serialization Before diving into JSON, I'd like to take a look at serialization in general and discuss common mistakes I've seen my customers make. ## Data Integrity with JSON Serialization - Ep.1 DevFeed: [Data Integrity with JSON Serialization - Ep.1]() Original publisher: [Read original article]() Published: 2024-10-03T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [JSON](), [Software Engineering](), [software-architecture]() Tags: [common-serialization-mistakes](), [custom-json-validation](), [data-integrity-in-software-engineering](), [data-serialization-in-api-design](), [data-validation](), [efficient-data-serialization](), [how-to-validate-json-data](), [introduction-to-json-for-engineers](), [json](), [json-best-practices](), [json-best-practices-for-api](), [json-data-validation-techniques](), [json-for-engineers](), [json-in-software-development](), [json-optimization-for-storage](), [json-serialization-tutorial](), [miki-tebeka-json-guide](), [serialization](), [serialization-in-software-architecture](), [software-architecture](), [software-layer-separation-and-data-handling]() ### AI overview Episode 1 of JSON for Engineers explains serialization fundamentals and how JSON converts data structures to byte sequences. It recommends serializing at system boundaries such as APIs and storage, keeping native types internally, separating models across layers, and validating data beyond JSON syntax. ### Source excerpt Introduction: Welcome to Episode 1 of JSON for Engineers! In this first episode, Miki Tebeka dives into the fundamentals of serialization, with a special focus on JSON, one of the most widely-used data formats in software engineering. Miki draws from his extensive development experience to explain how JSON fits into the bigger picture of data interchange and serialization, laying the groundwork for engineers to use it effectively in real-world applications. ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## Separate data schema from data representation DevFeed: [Separate data schema from data representation]() Original publisher: [Read original article]() Author: Yehonathan Sharvit Published: 2022-06-22T02:35:24Z Content type: article Language: en Sources: [Klipse]() Topics: [JSON Schema](), [Data validation](), [Programming]() Tags: [code](), [d](), [data-validation](), [dop](), [json-schema](), [programming]() ### AI overview An exploration of the fourth principle of Data-Oriented Programming: keeping data schemas separate from data representations. The article explains how schemas describe expected data shapes and how validation libraries such as Ajv can check data against JSON Schema and report validation errors. ### Source excerpt With data separated from code and represented with generic and immutable data structures, now comes the question of how do we express the shape of the data? In DOP, the expected shape is expressed as a data schema that is kept separated from the data itself. The main benefit of Principle #4 is that it allows developers to decide which pieces of data should have a schema and which pieces of data should not. ## Pinion -- The Load Framework Part-2 DevFeed: [Pinion -- The Load Framework Part-2]() Original publisher: [Read original article]() Author: Saurabh Jain Published: 2021-10-29T16:50:24Z Content type: article Language: en Sources: [Groupon Engineering -- Medium]() Topics: [data lake](), [Framework](), [Data Quality](), [Transactions](), [Streaming](), [Open Source]() Tags: [acid](), [audit](), [aws](), [cloud](), [data-validation](), [delta-lake](), [deltalake](), [hdfs](), [logging](), [optimization](), [reporting](), [s3](), [schema](), [science](), [spark](), [streaming](), [transactions]() ### AI overview This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel. ### Source excerpt Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a