# reproducibility Published articles for reproducibility. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Собрать прошлое: как архивировать весь трафик сборки SONiC DevFeed: [Собрать прошлое: как архивировать весь трафик сборки SONiC]() Original publisher: [Read original article]() Author: denballakh (Яндекс, Yandex Cloud & Yandex Infrastructure) Published: 2026-07-20T07:03:18Z Content type: tutorial Language: ru Sources: [Яндекс - Как мы делаем Яндекс / Статьи]() Topics: [Docker](), [Debian](), [HTTP](), [docker registry](), [GitHub](), [PyPI](), [Python]() Tags: [build](), [debian](), [docker](), [docker-registry](), [github](), [http](), [infrastructure](), [mitm](), [proxy](), [pypi](), [reproducibility](), [sonic]() ### AI overview This article explains how Yandex Infrastructure developed a caching MITM proxy for SONiC and YaSONiC builds. The proxy captures HTTP/HTTPS traffic and serves previously saved responses to improve build reproducibility when external dependencies change or disappear. ### Source excerpt Интернет -- это зависимость без зафиксированной версии. Сегодня он возвращает одно, завтра -- другое, а послезавтра -- ничего. Поэтому мы решили сохранить для сборки тот интернет, который она однажды увидела. Для этого пришлось перехватывать HTTPS, выпускать собственные сертификаты, переподписывать индексы Debian и выяснять, почему Docker отправляет "случайные" заголовки. В этой статье я расскажу, как разрабатывал кеширующую MITM-проксю, которая перехватывает весь HTTP/HTTPS-трафик и отдаёт ранее сохранённые ответы. Разберём также неочевидные технические проблемы, с которыми пришлось столкнуться. Читать далее ## Training Orchestrator: Unifying Model Training at Yelp DevFeed: [Training Orchestrator: Unifying Model Training at Yelp]() Original publisher: [Read original article]() Author: Ying Wang and Nathan Sponberg, Software Engineer Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Machine Learning & Artificial Intelligence](), [Orchestration](), [Apache Spark](), [Training AI Models](), [Model Development](), [configuration](), [configuration-management](), [Monitoring]() Tags: [configuration](), [machine-learning](), [maintenance](), [mlflow](), [model-training](), [orchestration](), [reproducibility](), [spark](), [tooling]() ### AI overview Yelp's Core Machine Learning Team developed Training Orchestrator to standardize how machine learning teams define and run Spark-based model training. The configuration-driven system addresses duplicated code, inconsistent configurations, limited local testing, scattered validation and monitoring, and poor reproducibility across environments. ### Source excerpt At Yelp, we train many machine learning models on different schedules. Applied machine learning teams all have their own set of Spark-based training batches, scripts, and configurations. Over time, these diverged, leading to duplicated code, subtle inconsistencies, and a growing maintenance burden. Yelp's Core Machine Learning Team has developed excellent tooling across our ML ecosystem over the years: feature stores for reproducible data, a unified training library for neural networks and gradient-boosted trees, seamless Spark integration, and MLflow services for model tracking and deployment. But there was still one key piece missing right in the middle: a standardized way to... ## The hidden variables in your agent eval DevFeed: [The hidden variables in your agent eval]() Original publisher: [Read original article]() Author: Waldek Mastykarz Published: 2026-07-08T12:11:17Z Content type: article Language: en Sources: [Developer Blogs]() Topics: [AI-assisted coding](), [context](), [Operating system](), [Shell]() Tags: [agent](), [agent-experience](), [ai](), [ai-coding-agents](), [ax](), [benchmarks](), [context](), [reproducibility]() ### AI overview This seventh article in Microsoft's Agent Experience series examines hidden environmental variables that can make the same AI coding-agent evaluation produce different results. It discusses context such as operating system, shell, diagnostics, build errors, file paths, and workspace details. ### Source excerpt This is the seventh article in a series about Agent Experience (AX): the practice of making AI coding agents work correctly with your technology. The series covers what you can and can't control in the agent stack, how to measure whether your extensions are helping or hurting, and how to iterate toward better outcomes. You [...] The post The hidden variables in your agent eval appeared first on Microsoft for Developers. ## Why Agentic AI Challenges Traditional Enterprise Change Management DevFeed: [Why Agentic AI Challenges Traditional Enterprise Change Management]() Original publisher: [Read original article]() Author: flashdba Published: 2026-06-02T12:56:04Z Content type: opinion Language: en Sources: [flashdba]() Topics: [Artificial Intelligence](), [reproducibility](), [Testing](), [Deployment]() Tags: [agentic](), [agentic-ai](), [ai](), [cloud](), [databases](), [databases-and-agentic-ai](), [deployment](), [inferencing](), [performance](), [release](), [reproducibility](), [testing]() ### AI overview The article argues that enterprise change management depends on software behaving reproducibly. Agentic AI can produce different outputs from the same inputs, weakening the assumptions behind testing, UAT sign-off, approval gates, and rollback procedures. ### Source excerpt Enterprise change management was built around software that behaves reproducibly - and that assumption is load-bearing in ways that only become visible when agentic AI removes it. ## Using Dolt's Versioned SQL Database for Feature-Store Point-in-Time Reads DevFeed: [Using Dolt's Versioned SQL Database for Feature-Store Point-in-Time Reads]() Original publisher: [Read original article]() Published: 2026-05-15T00:00:00Z Content type: opinion Language: en Sources: [Alex Korbonits]() Topics: [feature-store](), [Git](), [reproducibility](), [Databases](), [SQL]() Tags: [dedupe](), [feature-store](), [git](), [reproducibility](), [rfc](), [row-number](), [sql](), [sql-database]() ### AI overview This article describes a prototype Feast offline-store plugin backed by Dolt, a version-controlled SQL database. It argues that Dolt's revision-based reads can provide point-in-time feature retrieval and reproducible training snapshots without the usual append-only log deduplication CTEs. Tests on toy datasets produced identical results with shorter queries, though the article presents this as an early spike rather than a production benchmark. ### Source excerpt A weekend spike asked whether Dolt's AS OF reads could replace the ROW_NUMBER dedupe at the heart of every feature store's point-in-time join. Four weeks later, the RFC is quiet and the plugin's get_historical_features works end-to-end against a live Dolt server. Notes on building anyway. ## Guix improves one-line deployment with more secure channel file downloads DevFeed: [Guix improves one-line deployment with more secure channel file downloads]() Original publisher: [Read original article]() Author: Ludovic Courtès Published: 2026-05-12T15:00:00Z Content type: article Language: en Sources: [GNU Guix -- Blog]() Topics: [Security](), [Deployment](), [Software](), [Bash]() Tags: [bash](), [deployment](), [reproducibility](), [research](), [security](), [software]() ### AI overview This article examines Guix one-line deployment commands and the security risks of downloading and executing arbitrary channel files. It introduces a feature for downloading channel files while addressing trust and security concerns. ### Source excerpt When offered the option to run other people's code, a prime consideration is often ease of deployment. While much progress has been made in support of rapid deployment, the security implications of those quick deployments is often overlooked. In this post, we look at a new feature of guix time-machine and guix pull in support of one-line deployment commands : the ability to download channel files, but without compromising on security. Sharing code The normal workflow to share software and make it easily deployable with Guix goes like this: someone puts their packager hat on and... ## EU AI Act Compliance: Requirements, Risks, and What to Document DevFeed: [EU AI Act Compliance: Requirements, Risks, and What to Document]() Original publisher: [Read original article]() Author: info@goteleport.com (Kayne McGladrey, CISSP) Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [Artificial Intelligence](), [Requirements](), [Documentation](), [audit](), [data-governance](), [Logging](), [Monitoring](), [Deployment](), [data]() Tags: [ai](), [compliance](), [cybersecurity](), [data](), [data-governance](), [dataset](), [deployment](), [development](), [documentation](), [eu](), [iso-27001](), [lifecycle](), [logging](), [monitoring](), [nist](), [real-time](), [reproducibility](), [soc-2](), [versioning]() ### AI overview A practical guide to EU AI Act compliance covering requirements, risks, technical documentation, logging, data governance, lifecycle evidence, traceability, monitoring, and human oversight. It outlines key compliance milestones from August 2025 through August 2027. ### Source excerpt Cut through EU AI Act complexity with practical guidance on requirements, risks, and documentation. ## Don't Let Flakes Pin You Down: Unlocking Better Inputs DevFeed: [Don't Let Flakes Pin You Down: Unlocking Better Inputs]() Original publisher: [Read original article]() Author: toastal Published: 2026-03-26T00:00:00Z Content type: tutorial Language: en Sources: [Nixcademy Blog]() Topics: [Nix](), [version-control](), [Git](), [Tooling]() Tags: [best-practices](), [features](), [fetch](), [reproducibility](), [version-control]() ### AI overview This article examines the limitations of Nix flakes' input-pinning model and introduces nixtamal as a way to work with better inputs. It explains how pinning supports reproducibility, contrasts flakes with earlier tooling such as niv and npins, and discusses limitations including restricted version-control support. ### Source excerpt Flakes gave us input pinning, but you might be missing out on better inputs. Let's see how nixtamal unlocks them. ## How I learned to stop worrying and love the latest tag DevFeed: [How I learned to stop worrying and love the latest tag]() Original publisher: [Read original article]() Published: 2026-02-11T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [container images](), [Docker](), [Dockerfile](), [chainguard containers](), [Docker Hardened Images](), [Kubernetes](), [YAML]() Tags: [chainguard](), [chainguard-automations](), [chainguard-containers](), [chainguard-factory](), [container-image-digests](), [cryptographic](), [digestabot](), [digests](), [docker](), [docker-hardened-images](), [hardened-images](), [latest-tag](), [reproducibility]() ### AI overview The article explains how to use the latest tag as part of a secure container image update strategy. It recommends pinning images to cryptographic digests to ensure reproducibility, enable reliable rollbacks, and prevent unexpected version changes, while retaining tags for readability and tooling. It also introduces automated workflows for finding and updating digests. ### Source excerpt The latest tag isn't unsafe by default -- pin images to digests for reproducible, secure updates while staying current with automated workflows. ## Meet Guix at FOSDEM DevFeed: [Meet Guix at FOSDEM]() Original publisher: [Read original article]() Author: Ludovic Courtès Published: 2026-01-22T13:00:00Z Content type: article Language: en Sources: [GNU Guix -- Blog]() Topics: [FOSDEM](), [Package Management](), [reproducibility](), [systems]() Tags: [ci-cd](), [community](), [fosdem](), [guix-days](), [package-management](), [reproducibility](), [systems](), [talks]() ### AI overview The Guix community will participate in FOSDEM with talks on package management and reproducibility, bioinformatics workflows, Guile development, build systems, functional reactive programming, container images, and GNU/Hurd. About sixty Guix contributors will then gather for Guix Days on February 2-3. ### Source excerpt It's that time of the year again: next week is FOSDEM time! As in previous years , many Guix people will be in Brussels. Right after FOSDEM, about sixty of us will gather on February 2-3 for the Guix Days! First things first: Guix presence at FOSDEM. On Saturday, January 31st : In Name resolution in package management systems -- A reproducibility perspective , Gábor Boskovits will look will look at how several package managers refer to packages and how this affects reproducibility. Simon Tournier will... ## Malware-Resistant Python without the Guesswork DevFeed: [Malware-Resistant Python without the Guesswork]() Original publisher: [Read original article]() Published: 2025-08-01T00:00:00Z Content type: opinion Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard libraries for python](), [chainguard libraries](), [Security](), [chainguard](), [Open Source](), [Malware]() Tags: [chainguard-libraries](), [chainguard-libraries-for-python](), [malicious-packages](), [malware](), [num2words](), [provenance](), [pypi](), [python](), [reproducibility](), [sbom](), [secure-packages](), [security](), [signing](), [slsa](), [supply-chain]() ### AI overview The article presents Chainguard Libraries for Python as a way to reduce malware and software supply chain risks in Python package consumption. It describes rebuilding packages from upstream source in an isolated, reproducible SLSA Level 2 environment, and publishing signed SBOMs and provenance information. ### Source excerpt The recent compromise of the num2words package never made it into Chainguard Libraries for Python. Get the breakdown from the team on our packages you can trust. ## Introduction to MLOps DevFeed: [Introduction to MLOps]() Original publisher: [Read original article]() Author: Maria Vechtomova Published: 2025-07-28T17:19:47Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [MLOps](), [databricks](), [Monitoring](), [Deployment](), [Testing]() Tags: [audit-trail](), [course](), [databricks](), [deployment](), [introduction](), [mlops](), [monitoring](), [production](), [reproducibility](), [testing]() ### AI overview Lecture 1 of a hands-on MLOps with Databricks course explains what production means for machine-learning workflows. It uses a demand-forecasting example to show why scheduled notebooks may lack testing, monitoring, error handling, version control, deployment processes, rollback, and audit trails, and introduces MLOps principles for improving reliability and control. ### Source excerpt Lecture 1 of MLOps with Databricks course ## Why Chainguard's Full-Stack Approach to Secure Software Supply Chain Is Built to Scale DevFeed: [Why Chainguard's Full-Stack Approach to Secure Software Supply Chain Is Built to Scale]() Original publisher: [Read original article]() Published: 2025-07-09T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [supply-chain-security](), [chainguard os](), [chainguard](), [Open Source]() Tags: [automated](), [chainguard](), [chainguard-factory](), [chainguard-os](), [cve](), [dependencies](), [open-source-artifacts](), [open-source-software-security](), [provenance](), [reproducibility](), [secure-software](), [secure-software-supply-chain](), [software-supply-chain](), [supply-chain-security]() ### AI overview The article presents Chainguard's integrated approach to software supply chain security, combining Chainguard OS with the Chainguard Factory. It describes reproducible source builds, incremental updates, traceable contents, verifiable metadata, and automated maintenance of open source artifacts. ### Source excerpt Learn how Chainguard OS and the Chainguard Factory delivers the only scalable path to secure, reliable software artifacts. ## Bamba: Inference-Efficient Hybrid Mamba2 Model DevFeed: [Bamba: Inference-Efficient Hybrid Mamba2 Model]() Original publisher: [Read original article]() Author: LINSONG CHU; Divya Kumari; Tri Dao; Albert Gu; Raghu Ganti; Dakshi Agrawal; Mudhakar Srivatsa; Davis Wertheimer; Yu Chin Fabian Lim; Antoni Viros Published: 2024-12-18T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Mamba](), [Inference](), [Large Language Model](), [vllm](), [hugging face](), [ibm](), [datasets](), [Open Source]() Tags: [community](), [data](), [hugging-face](), [ibm](), [inference](), [llm](), [mamba](), [model](), [nlp](), [nvidia](), [open-source](), [reproducibility](), [research](), [vllm]() ### AI overview The article introduces Bamba-9B, an inference-efficient hybrid Mamba2 model trained by IBM, Princeton, CMU, and UIUC on open data. It reports higher throughput and lower latency than standard transformers in vLLM, and releases training resources, checkpoints, and reproducibility materials for community experimentation. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Aequitas Flow step-by-step: a Fair ML optimization framework DevFeed: [Aequitas Flow step-by-step: a Fair ML optimization framework]() Original publisher: [Read original article]() Author: Sérgio Jesus Published: 2024-08-12T14:35:43Z Content type: tutorial Language: en Sources: [Feedzai]() Topics: [Machine learning](), [Optimization](), [audit](), [Benchmark](), [pandas](), [datasets](), [interfaces]() Tags: [audit](), [benchmark](), [datasets](), [fairness](), [interfaces](), [machine-learning](), [ml](), [open-source](), [optimization](), [pandas](), [reproducibility](), [research](), [responsible-ai]() ### AI overview This tutorial introduces Aequitas Flow, an open-source framework for standardized experiments with fair machine-learning algorithms. It explains how the framework extends Aequitas fairness auditing with reproducible experimentation, support for custom datasets and methods, and workflows for practitioners selecting models and researchers running benchmarks. ### Source excerpt By Sérgio Jesus, Inês Silva, Pedro Saleiro, Hugo Ferreira, Pedro Bizarro In this blog post we will visit Aequitas Flow, an Open-Source framework designed to run complete and standardized experiments of Fair ML algorithms. We encourage you to try Aequitas Flow with the Google Colab Notebooks, which are available in the project's GitHub repository. This blog post is based on the paper by Sérgio Jesus, Pedro Saleiro, Inês Silva, Beatriz M. Jorge, Rita P. Ribeiro, João Gama, Pedro Bizarro, and Rayid Ghani. Table of Contents: 1. What is Aequitas Flow? - 1.1. For Practitioners selecting a model - 1.2. For Researchers running a benchmark 2. Install Aequitas Flow 3. The components of Aequitas Flow - 3.1. Experiment - 3.2. Optimizer - 3.3. Datasets - 3.4. Methods - 3.5. Audit 4. Conclusion What is Aequitas Flow? Aequitas Flow is the codename for the latest version of Aequitas, a well-established package for fairness auditing in the ML community. This version extends the package to include experimentation with Fair ML algorithms. Aequitas started as a software to diagnose and alert for disparities in ML models' decisions depending on sensitive attributes, such as race, gender, or age. To achieve this, the package runs a Bias Audit, in which it calculates several metrics across all data groups, determined by the sensitive attributes, and compares them to identify any gap in performance. While some other packages already implement Fair ML methods, they also introduce an overhead of technical knowledge to configure, evaluate, and deploy these models. Because of this, we extended Aequitas to enable users to conduct experiments using a wide variety of fair ML methods from the literature in an intuitive user experience. Aequitas Flow was built on the principles of extensibility and reproducibility: it allows users to incorporate their own datasets and methods in the framework using familiar interfaces of sci-kit-learn and pandas. It ensures their work can be replicated by performin ## Nix Weekly Recap: 2024-02-25 DevFeed: [Nix Weekly Recap: 2024-02-25]() Original publisher: [Read original article]() Published: 2024-02-25T00:00:00Z Content type: article Language: en Sources: [nixpkgs.news]() Topics: [Nix](), [Development](), [reproducibility](), [Terraform](), [HackSquad]() Tags: [announcement](), [apt](), [community](), [google-summer-of-code](), [mentorship](), [prometheus](), [pull-requests](), [python](), [recap](), [repository](), [reproducibility](), [rust](), [terraform](), [weekly]() ### AI overview A weekly recap of announcements and repository activity in the Nix community, covering configuration visualization, reproducible HTTP/HTTPS responses for sandboxed builds, community funding, Typst support, Google Summer of Code participation, Terraform integration, Nix formatting, XR/AR/VR packages, and various package updates. ### Source excerpt Weekly recap of the announcements and activity in the Nix community and on the NixPkgs package repository. ## Images as Code: The pursuit of declarative image builds DevFeed: [Images as Code: The pursuit of declarative image builds]() Original publisher: [Read original article]() Published: 2024-01-22T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [Code](), [chainguard images](), [Dockerfile](), [bazel](), [distroless](), [Docker](), [Containers](), [Kubernetes](), [Terraform](), [Infrastructure as code](), [Open Source]() Tags: [apk](), [apko](), [bazel](), [chainguard](), [chainguard-images](), [code](), [container-image](), [containers](), [declarative](), [distroless](), [docker](), [infrastructure-as-code](), [kubernetes](), [open-source](), [reproducibility](), [terraform]() ### AI overview The article traces the pursuit of declarative container image builds. It critiques imperative Dockerfile-based builds for making multi-tenant, multi-architecture, and reproducible builds difficult, then discusses Bazel and distroless images as steps toward expressing intended build state. Kubernetes and Terraform are presented as examples of declarative systems, inspiring the idea of "Images as Code." ### Source excerpt Chainguard's CTO Matt Moore describes the process of creating a declarative container image build for Chainguard Images. ## Farewell from the Reproducible Builds Summit 2023! DevFeed: [Farewell from the Reproducible Builds Summit 2023!]() Original publisher: [Read original article]() Published: 2023-11-02T00:00:00Z Content type: news Language: en Sources: [reproducible-builds.org]() Topics: [reproducible builds](), [reproducibility](), [builds](), [Embedded Systems]() Tags: [adb](), [apache](), [buildroot](), [builds](), [coreboot](), [debian](), [dockerignore-usage](), [fedora](), [germany](), [github](), [linux](), [maven](), [openwrt](), [org](), [reproducibility](), [reproducible-builds](), [security](), [summit](), [systemd](), [ubuntu]() ### AI overview A recap of the seventh Reproducible Builds summit in Hamburg, Germany, covering project updates, reproducibility practices, verification services, filesystem images and containers, package reproducibility, SBOMs, and related discussions. ### Source excerpt Farewell from the Reproducible Builds summit, which just took place in Hamburg, Germany: This year, we were thrilled to host the seventh edition of this exciting event. Topics covered this year included: Project updates from openSUSE, Fedora, Debian, ElectroBSD, Reproducible Central and NixOS Mapping the "big picture" Towards a snapshot service Understanding user-facing needs and personas Language-specific package managers Defining our definitions Creating a "Ten Commandments" of reproducibility Embedded systems Next steps in GNU Guix' reproducibility Signature storage and sharing Public verification services Verification use cases Web site audiences Enabling new projects to be "born reproducible" Collecting reproducibility success stories Reproducibility's relationship to SBOMs SBOMs for RPM-based distributions Filtering diffoscope output Reproducibility of filesystem images, filesystems and containers Using verification data A deep-dive on Fedora and Arch Linux package reproducibility Debian rebuild archive service discussion ... as well as countless informal discussions and hacking sessions into the night. Projects represented at the venue included: Debian, openSUSE, QubesOS, GNU Guix, Arch Linux, phosh, Mobian, PureOS, JustBuild, LibreOffice, Warpforge, OpenWrt, F-Droid, NixOS, ElectroBSD, Apache Security, Buildroot, Systemd, Apache Maven, Fedora, Privoxy, CHAINS (KTH Royal Institute of Technology), coreboot, GitHub, Tor Project, Ubuntu, rebuilderd, repro-env, spytrap-adb, arch-repro-status, etc. A huge thanks to our sponsors and partners for making the event possible: Event facilitation Platinum sponsor If you weren't able to make it this year, don't worry; just look out for an announcement in 2024 for the next event. ## Reproducing Chainguard's reproducible image builds DevFeed: [Reproducing Chainguard's reproducible image builds]() Original publisher: [Read original article]() Published: 2023-07-05T00:00:00Z Content type: tutorial Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard images](), [supply-chain-security](), [configuration](), [Tooling]() Tags: [apko](), [attestation](), [build](), [chainguard](), [chainguard-images](), [configuration](), [cosign](), [hardened-images](), [locks](), [reproducibility](), [reproducible-builds](), [secure-image](), [security](), [software-supply-chain](), [supply-chain-security](), [wolfi]() ### AI overview A tutorial explaining how to reproduce a Chainguard Images build using cosign and apko. It describes locking image configurations and notes caveats involving tooling changes and withdrawn packages. ### Source excerpt Learn how to reproduce a Chainguard Images build using cosign and apko. ## Building reproducible Python environments with XARs DevFeed: [Building reproducible Python environments with XARs]() Original publisher: [Read original article]() Author: Pavel Senchanka Published: 2023-04-14T00:00:00Z Content type: article Language: en Sources: [Jane Street]() Topics: [Python](), [Conda](), [Deployment](), [Docker](), [OCaml](), [Open Source]() Tags: [build-system](), [conda](), [declarative](), [deployment](), [docker](), [open-source](), [python](), [reproducibility]() ### AI overview Jane Street describes how it built js-python, a system for creating and deploying centrally built, reproducible Python environments. The system integrates Python environments with OCaml code and uses XAR for deployment after limitations with Conda and other tools became apparent. ### Source excerpt Our traders and researchers love Python for its agility and for its huge open-source ecosystem, especially when it comes to machine learning. But the heavy use of notebooks can make it difficult to support. Notebooks have a very different lifecycle than regular code, and aren't always rigorously version controlled. And while most of our code (much of it written in OCaml) lives in a monorepo, putting all notebooks there is difficult; many notebooks end up being stored all over the place. ## Improving Evaluation Practices in Natural Language Generation DevFeed: [Improving Evaluation Practices in Natural Language Generation]() Original publisher: [Read original article]() Author: Saad Mahamood NLG Expert; Lead Data Scientist Published: 2022-03-31T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Natural language processing](), [recommendations]() Tags: [data-science](), [evaluation](), [language](), [open-source](), [recommendations](), [report](), [reproducibility](), [research]() ### AI overview This article reviews research into evaluation practices for Natural Language Generation. It discusses weaknesses in automated metrics, variation in human evaluation methods, and reproducibility concerns, including findings from trivago's HumEval 2021 work and its proposed Commonsense Evaluation Card. ### Source excerpt Throughout last year I had the opportunity to participate and collaborate on multiple research initiatives in the field of Nat... ## Finding Non-determinism with nixbuild.net DevFeed: [Finding Non-determinism with nixbuild.net]() Original publisher: [Read original article]() Author: support@nixbuild.net Published: 2021-01-13T00:00:00Z Content type: tutorial Language: en Sources: [nixbuild.net blog]() Topics: [Nix](), [reproducibility](), [reproducible builds](), [builds](), [content addressed store]() Tags: [blog](), [blog-post](), [build](), [reproducibility](), [reproducible-builds]() ### AI overview This blog post explains how to use Nix repeated builds to detect non-deterministic outputs by comparing build results bit for bit. It also describes how nixbuild.net records repeated-build results and can help investigate failures from past builds. ### Source excerpt During the last decade, many initiatives focussing on making builds reproducible have gained momentum. reproducible-builds.org is a great resource for anyone interested in how the work progresses in multiple software communities. r13y.com tracks the current reproducibility metrics in NixOS. Nix is particularly suited for working on reproducibility, since it by design isolates builds and comes with tools for finding non-determinism. The Nix community also works on related projects, like Trustix and the content-addressed store. This blog post summarises how nixbuild.net can be useful for finding non-deterministic builds, and announces a new feature related to reproducibility! Repeated Builds The way to find non-reproducible builds is to run the same build multiple times and check for any difference in results, when compared bit-for-bit. Since Nix guarantees that all inputs will be identical between the runs, just finding differing output results is enough to conclude that a build is non-deterministic. Of course, we can never prove that a build is deterministic this way, but if we run the build many times, we gain a certain confidence in it. To run a Nix build multiple times, simply add the -repeat option to your build command. It will run your build the number of extra times you specify. Suppose we have the following Nix expression in deterministic.nix: let inherit (import {}) runCommand; in { stable = runCommand "stable" {} '' touch $out ''; unstable = runCommand "unstable" {} '' echo $RANDOM > $out ''; } We can run repeated builds like this (note that the --builders "" option is there to force a local build, to not use nixbuild.net): $ nix-build deterministic.nix --builders "" -A stable --repeat 1 these derivations will be built: /nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv building '/nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv' (round 1/2)... building '/nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv' (round 2/2)... /nix/store/65 ## Searching for RH Counterexamples -- Deploying with Docker DevFeed: [Searching for RH Counterexamples -- Deploying with Docker]() Original publisher: [Read original article]() Published: 2021-01-04T09:00:00Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Docker](), [Containers](), [reproducibility](), [Database](), [Server]() Tags: [aws](), [containers](), [database](), [docker](), [mathematics](), [programming](), [pytest](), [reproducibility](), [riemann-hypothesis](), [server](), [software]() ### AI overview A tutorial on deploying an application that searches for counterexamples to the Riemann Hypothesis. It explains reproducibility and hermeticity, then uses Docker containers for the database, search application, and planned web server. ### Source excerpt We're ironically searching for counterexamples to the Riemann Hypothesis. Setting up Pytest Adding a Database Search Strategies Unbounded Integers In this article we'll deploy the application on a server, so that it can search for RH counterexamples even when I close my laptop. Servers and containers When deploying applications to servers, reproducibility is crucial. You don't want your application to depend on the details of the computer it's running on. This is a higher-level version of the same principle behind Python virtual environments, but it applies to collections of programs, possibly written in different languages and running on different computers. ## Why Bazel? DevFeed: [Why Bazel?]() Original publisher: [Read original article]() Author: Marc Plano-Lesay Published: 2019-11-02T07:00:00Z Content type: tutorial Language: en Sources: [Marc Plano-Lesay]() Topics: [bazel](), [Caching](), [toolchain](), [C++](), [Compiler]() Tags: [bazel](), [build-system](), [c-plus-plus](), [caching](), [compiler](), [reproducibility](), [toolchain]() ### AI overview This tutorial explains what Bazel is, how its hermetic and reproducible build model works, and why the author chose it. It demonstrates C++ library and binary targets, dependency-driven builds, caching, toolchains, and extensibility through Starlark rules. ### Source excerpt In this post, we'll cover what Bazel is, how to use it, and why I chose to use it. What is Bazel? Bazel is a build-system released by Google in 2015. It actually is derived from the internal build-system Google uses internally for most of its own code-base, called Blaze. Building at scale Bazel has a huge focus on hermetic builds, and reproducibility. Every build step is, from a really broad perspective, defined as a list of inputs, tools, and outputs. This allows for efficient and robust caching (if no inputs nor tools changed, then this target doesn't need to be rebuilt, and this cascades through the whole build graph). Let's see a sample definition of a C++ library, as well as a C++ binary depending on it: BUILD cc_library( name = "my_feature" srcs = [ "feature_impl.cpp", "utils.cpp", ], hdrs = [ "feature.hpp", "utils.hpp", ], ) cc_binary( name = "my_app", srcs = ["main.cpp"], deps = [ ":my_feature", ], ) cc_library and cc_binary are both depending an implicit dependency on a C++ toolchain (I won't enter into any language-specific features in this post, but if you don't tell Bazel to use a specific C++ toolchain, it will try to use your system compiler - which is convenient, but loses a bit of hermeticity and reproducibility). Everything else is pretty obvious here: we defined two different build targets, one of them being a library called my_feature, and the other one a binary called my_app, depending on my_feature. If we build my_app, Bazel will automatically build my_feature first as you would expect, and then proceed to build my_app. If you change the main.cpp and re-build my_app, it will skip the compilation of my_feature entirely, as nothing changed. Bazel's cache handling is really reliable. During the past few months, I've done a lot of diverse things (writing my own rules, compiling a bunch of different languages, depending on third-party libraries and rules...), and never had a single time to run bazel clean. Now I didn't use a lot of other build systems [Next page]()