# data transformation Published articles for data transformation. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Using Polars in unison with Databricks Unity Catalog DevFeed: [Using Polars in unison with Databricks Unity Catalog]() Original publisher: [Read original article]() Author: Maria Vechtomova Published: 2025-05-24T13:57:40Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [data transformation](), [Apache Spark](), [Benchmark]() Tags: [benchmark](), [data-transformation](), [databricks](), [performance](), [spark]() ### AI overview This tutorial explains how to read data from Databricks Unity Catalog directly into Polars dataframes using PyArrow and Deltalake, avoiding an intermediate PySpark dataframe. It also presents a TPC-H benchmark comparing Polars with PySpark or Spark SQL for some data-transformation workloads, with reported potential performance improvements and cost savings for certain data sizes. ### Source excerpt The advent of Polars is not surprising given the performance it delivers. ## Concise Link Descriptions in netlab Topologies DevFeed: [Concise Link Descriptions in netlab Topologies]() Original publisher: [Read original article]() Published: 2025-01-20T07:22:00Z Content type: tutorial Language: en Sources: [ipSpace.net blog]() Topics: [Network](), [YAML](), [data-processing]() Tags: [data-transformation](), [interfaces](), [netlab](), [network](), [ospf](), [router](), [yaml]() ### AI overview This tutorial explains how netlab represents links between lab devices and the shorthand formats available for defining them in YAML. It covers hyphen-separated node strings, node lists, dictionary-based link definitions, link attributes, and interface parameters, while showing that all formats are transformed into a common interfaces-based data model. ### Source excerpt One of the goals we're always trying to achieve when developing netlab features is to make the lab topologies as concise as possible1. Among other things, netlab supports numerous ways of describing links between lab devices, allowing you to be as succinct as possible. A bit of a background first: In the end, netlab collects all links in the links list before starting the data transformation process. Every entry in the links list is a dictionary. That dictionary can contain link attributes and must contain a list of interfaces connected to the link. Every interface must have a node (specifying the lab device it belongs to) and could contain additional interface attributes. Read more ... ## A behind-the-scenes look at building interactive analysis capabilities in Benchling DevFeed: [A behind-the-scenes look at building interactive analysis capabilities in Benchling]() Original publisher: [Read original article]() Author: Wonja Fairbrother Published: 2024-06-11T13:01:25Z Content type: article Language: en Sources: [Benchling]() Topics: [data-processing](), [web applications](), [Amazon EKS](), [Amazon S3](), [API](), [Back end](), [Front end](), [JSON]() Tags: [apache-arrow](), [apache-parquet](), [api](), [architecture](), [backend](), [data](), [data-analysis](), [data-transformation](), [data-visualization](), [duckdb](), [frontend](), [json](), [real-time](), [s3](), [storage](), [systems]() ### AI overview This article explains the architecture and design decisions behind Benchling's Interactive Analysis capabilities. The system lets scientists select data from multiple sources, transform and analyze it in real time, and visualize results within Benchling. Its architecture uses the Benchling web application, a stateless service running on EKS, temporary S3 storage, and a JSON-based transformation API. ### Source excerpt Authors: Wonja Fairbrother and Eli Levine Science is iterative. To design the next experiment, scientists need to analyze the results of previous ones. Interactive Analysis in Benchling allows scientists to perform real-time data transformation, visualization, and analysis without having to transfer it into other systems. In this post we will describe the architecture behind interactive analysis capabilities in Benchling and give a peek into the decision journey we took along the way¹. Interactive Analysis allows scientists to: 1. Select data from many sources: Benchling entity and results data Instrument data Notebook tables Data upload via both API and UI 2. Transform, visualize, and analyze data in real time, without leaving Benchling: Data transformations: filtering, aggregations, window functions, etc. Visualizations: line chart, bar chart, scatter plot, etc. Scientific analysis methods: IC50 and various curve fitting functions Overall architecture The architecture backing Interactive Analysis consists of: The Benchling web application An auto-scaling stateless internal service running on EKS that performs the transformations Temporary S3 storage locations for input and output data, shared between the web app and the service The frontend of the application is responsible for taking in input datasets and transformation configurations from users. The backend of the web application collects all the input data from the appropriate sources, serializes and uploads the data to S3, and sends a synchronous transformation request to the service. The service's API consists of one main endpoint that takes in a JSON payload of transformation parameters. The service can accept a single transformation, or a list of many transformations to perform. In this endpoint, the service downloads and deserializes the input data, performs the transformation with an analysis engine, and serializes and uploads the resulting data to S3. Each request spins up its own self-contained in-memor ## Testing Network Automation Data Transformation DevFeed: [Testing Network Automation Data Transformation]() Original publisher: [Read original article]() Published: 2024-05-20T06:08:00Z Content type: tutorial Language: en Sources: [ipSpace.net blog]() Topics: [Network](), [Testing](), [Automation](), [data-processing](), [GitHub Actions](), [YAML](), [Python](), [Refactoring]() Tags: [automation](), [bug](), [data-transformation](), [github](), [github-actions](), [netlab](), [network-automation](), [python](), [refactoring](), [testing](), [yaml]() ### AI overview The article explains how netlab tests the transformation of high-level network automation data into device data models. It describes pre-commit and pre-push checks, GitHub Actions, YAML and Python validation, transformation tests, error-handling tests, and regression tests created from fixed bugs. ### Source excerpt Every complex enough network automation solution has to introduce a high-level (user-manageable) data model that is eventually transformed into a low-level (device) data model. High-level overview of the process The transformation code (business logic) is one of the most complex pieces of a network automation solution, and there's only one way to ensure it works properly: you test the heck out of it ;) Let me show you how we solved that challenge in netlab. Read more ... ## Swift Result Builders: Creating Custom DSLs for Binary Formatted Data DevFeed: [Swift Result Builders: Creating Custom DSLs for Binary Formatted Data]() Original publisher: [Read original article]() Author: Paul Kraft Published: 2023-08-03T14:22:38Z Content type: tutorial Language: en Sources: [QuickBird Studios Blog]() Topics: [Swift](), [Declarative programming](), [data](), [SwiftUI](), [Functional programming](), [Programming](), [data-processing](), [Bluetooth](), [Internet](), [P2P]() Tags: [bluetooth](), [communication](), [data-transformation](), [declarative](), [deep-dive](), [functional-programming](), [internet](), [ios](), [post](), [programming](), [swift](), [swiftui]() ### AI overview This tutorial explains Swift result builders and declarative programming, then applies custom result builders to encoding binary-formatted data. It also introduces DataKit, a Swift library for declaratively specifying binary message formats for encoding and decoding. ### Source excerpt SwiftUI has revolutionized how we build UI, introducing a more intuitive, declarative approach. Instead of prescribing a series of steps to reach an end goal, we describe the outcome and let the program determine the path The post Swift Result Builders: Creating Custom DSLs for Binary Formatted Data appeared first on QuickBird Studios. ## Interfaces 101 : Parsing Command Flags Ep. 9 DevFeed: [Interfaces 101 : Parsing Command Flags Ep. 9]() Original publisher: [Read original article]() Published: 2023-03-27T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [Go Language](), [interfaces](), [Command-line interface](), [Code](), [client](), [HTTP](), [API]() Tags: [cli](), [command-line](), [data-transformation](), [flag](), [go](), [golang](), [how-to](), [http](), [implementation](), [interfaces](), [learn](), [unmarshalling](), [validation]() ### AI overview A video tutorial explains how to decode a command-line flag representing an address into a user-defined Go struct. It covers the interface methods involved and validates the supplied flag value. ### Source excerpt Introduction In episode 8, Miki developed a Go HTTP client that had a method to check the health of a theoretical API. The method would construct the request URL and return an error based on the response code received from the server. Miki then created a type that would mock Go's http.RoundTripper interface and replace his client's transport with said type to test it. To improve the efficacy, Miki's stub will simulate an erroneous response to see if his client's Health method is working as intended. ## Designing Go Interfaces Around Behavior and Concrete Data DevFeed: [Designing Go Interfaces Around Behavior and Concrete Data]() Original publisher: [Read original article]() Published: 2018-03-23T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [interfaces](), [Programming](), [data](), [data-processing]() Tags: [abstraction](), [ardan-labs](), [blog](), [data-transformation](), [go](), [go-programming](), [golang](), [interfaces](), [performance](), [programming](), [technical]() ### AI overview This article presents a data-oriented approach to designing interfaces in Go. It argues that developers should focus on interface behavior and its relationship with concrete data rather than implementation details or broad abstraction layers. ### Source excerpt Introduction I've been seeing a lot of question about interfaces lately on Slack. Most of the time the answers are technical and focus on implementation details. Implementation is important to help with debugging, but implementation doesn't help with design. When it comes to designing code with interfaces, behavior has to be the main focus. In this post, I hope to provide a different way to think about interfaces and how to design code with them. I want you to stop focusing on the implementation details and focus on the relationship interfaces have with concrete data.