# Benchling The official blog of the Benchling engineering team. - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Fragmentation to framework: Spec-first development at Benchling DevFeed: [Fragmentation to framework: Spec-first development at Benchling]() Original publisher: [Read original article]() Author: Eli Levine Published: 2026-02-19T17:29:16Z Content type: article Language: en Sources: [Benchling]() Topics: [Development](), [Framework](), [data](), [API](), [Tech Debt](), [interoperability](), [configuration](), [migration]() Tags: [api](), [benchling](), [biotechnology](), [configuration](), [data](), [development](), [framework](), [interoperability](), [migration](), [platform-engineering](), [tech-debt]() ### AI overview Benchling describes how manually integrating scientific data objects with APIs, analytics, search, configuration migration, and other platform capabilities created an MxN maintenance burden, technical debt, and behavioral drift. The article introduces a spec-first framework approach intended to make platform functionality more uniform and scalable. ### Source excerpt Reaching the limit of manual platform development Benchling's platform handles diverse scientific data, such as DNA sequences, antibodies, notebook entries, inventory containers, workflow runs, and dozens more. Each object type carries unique domain logic: how it is validated, what relationships it holds, and what actions users can perform on it. As Benchling matured, capabilities were added that customers expected to work across all these objects including REST APIs for integration, a data warehouse for analytics, search indexing, and configuration migration tools for moving setups between tenants, among many others. Each product team is expected to expose their data in all platform surface areas. However, because this process is manual it can also be brittle and costly. With M object types and N platform capabilities, and each object requires custom integration with each capability, you're maintaining MxN integration points. Add a new object? You'll need to integrate it with every platform capability. Add a new capability? You'll need to integrate it with every object. In practice, this meant product and tech debt: some objects were available via API but missing from the warehouse, or a feature was exposed in the UI but not in other platform surface areas. It also meant behavioral drift. The same object would have slightly different field names or validation logic depending on which surface you accessed it through. As Benchling grew, so did its customers. Enterprise customers expect platforms that are designed for multi-modal integration that covers the full spectrum of Benchling's data and functionality. AI is quickly reshaping how knowledge work is done across all industries. But some fundamentals have not shifted. The same integration capabilities that make enterprise architectures more powerful are what makes agents more powerful too: both require data access and interoperability. Thus a different approach was needed. Ideally one where the cost of adding types ## The Multi-Model Playbook DevFeed: [The Multi-Model Playbook]() Original publisher: [Read original article]() Author: Sumedh Bhattacharya Published: 2026-01-16T16:02:06Z Content type: tutorial Language: en Sources: [Benchling]() Topics: [Large Language Model](), [ai-coding](), [Structured-data](), [agent observability](), [pdf]() Tags: [agentic-engineering](), [ai](), [ai-agents](), [analysis](), [benchling](), [biotechnology](), [caching](), [code](), [coding](), [comparisons](), [llm](), [model](), [parallel](), [patterns]() ### AI overview Benchling describes patterns for building production AI systems across multiple model providers. The article covers modular task decomposition, prompt structure, caching, structured data presentation, provider comparisons, and applying these principles to AI coding assistants. It reports using OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, and Amazon Nova in agent systems such as the Data Entry Agent and Compose. ### Source excerpt The Multi-Model Playbook: Patterns in Agentic Engineering Building production AI systems that work reliably across multiple model providers requires more than just swapping API keys. Over the past year, working on AI agents like the Data Entry Agent and Compose Agent at Benchling, I've learned that successful multi-provider strategies come down to understanding what's universal versus what's provider-specific, and designing around those constraints. The clearest revelation here was that the architectural principles underlying reliable software -- modularity, separation of concerns, clear interfaces -- apply just as fundamentally to AI systems as they do to traditional code. The Data Entry Agent (DEA) extracts structured data from PDFs and images, while Compose is an agent that helps scientists write electronic lab notebooks (ELNs) by extracting content from attached files, connecting that with data in Benchling's Registry, and outputting structured scientific protocols, analysis, and more. These systems currently support five different model families (OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, and Amazon Nova), typically using four in any given run. This experience has revealed patterns that hold true across providers -- patterns around task decomposition, prompt structure, caching strategies, and data presentation. While each provider has its quirks, these foundational strategies have proven consistently effective. In this post, I'll cover: How to break down problems for optimal LLM performance Why the system versus user prompt distinction matters for caching Best practices for presenting structured data as context Practical comparisons between model providers How to apply these principles when using AI coding assistants. Breaking Down Problems: Small & Complex versus Large & Simple LLMs lose accuracy when handling multiple separate tasks simultaneously or when operating on large input contexts. The sweet spot is to give them either a small, complex task ## How We Run Terraform At Scale DevFeed: [How We Run Terraform At Scale]() Original publisher: [Read original article]() Author: Christian Monaghan Published: 2025-03-04T16:32:39Z Content type: article Language: en Sources: [Benchling]() Topics: [Terraform](), [Amazon Web Services](), [Amazon S3](), [DynamoDB]() Tags: [aws](), [cloud-infrastructure](), [dynamodb](), [infrastructure](), [infrastructure-as-code](), [management](), [operations](), [s3](), [software-engineering](), [terraform]() ### AI overview Benchling describes how it manages more than 165,000 cloud resources across 625 Terraform workspaces and 38 AWS accounts. The article explains that its earlier laptop-based Terraform workflow, using S3 state files and DynamoDB locks, became inefficient as the number of workspaces grew. ### Source excerpt Managing over 165k cloud resources across hundreds of workspaces could seem daunting. But for us, it's just another day at Benchling. Here's how we do it. We currently have: 165k cloud resources under management 625 Terraform workspaces 38 AWS accounts 170 engineers (40 of whom are infra specialists) We perform: 225 infrastructure releases daily (terraform apply operations) 723 plans daily (terraform plan operations) We've been successfully operating Benchling's infrastructure release system for the past two years (spoiler, it's Terraform Cloud), over which time we've doubled our infrastructure footprint with minimal additional release overhead. Before Terraform Cloud: The Chaos Our infra release process wasn't always this smooth. Let me rewind and take you back to how it was before. As is common guidance for small Terraform projects, our team would previously apply all infrastructure changes via laptop. Also in line with common guidance, our team used S3 to store state files, with DynamoDB state locks, which prevented any apply-time collisions. This is a great strategy for a small team working on up to a dozen workspaces. However, this slowly starts to break down as the team's workspace footprint grows. It's like the proverbial frog in the pot of water, slowly heated to a boil. By the time we made the switch, Benchling was managing 350 workspaces. We were approaching the boiling point. Pain Points: Developer Toil and Inefficiency Managing 350 workspaces with this approach had several downsides: Necessitated elevated AWS access permissions for the infrastructure team. It was time-consuming as the engineer had to navigate to each directory, run terraform apply, review and approve the run, then verify it succeeded. Very commonly a single change could affect over 120 workspaces, which would mean repeating this process 120 times. (We had developed a custom python script which helped parallelize this somewhat.) Accumulated infra drift. Often an engineer would go to apply ## Building an LLM-Powered Slackbot DevFeed: [Building an LLM-Powered Slackbot]() Original publisher: [Read original article]() Author: Christian Monaghan Published: 2024-12-13T17:32:12Z Content type: tutorial Language: en Sources: [Benchling]() Topics: [Large Language Model](), [Retrieval Augmented Generation (RAG)](), [Slack](), [Terraform](), [Amazon Bedrock](), [cloud-infrastructure](), [Self-hosted]() Tags: [amazon-bedrock](), [building](), [infrastructure](), [knowledge-base](), [large-language-models](), [llm](), [rag](), [retrieval-augmented-gen](), [self-hosted](), [slackbot](), [terraform]() ### AI overview Benchling describes building an internal Slackbot that uses Retrieval-Augmented Generation and Amazon Bedrock to help engineers find answers to Terraform Cloud questions from sources including Slack, Confluence, and the web. ### Source excerpt Background At Benchling we run cloud infrastructure across several regions and environments. To coordinate and manage this complexity, our team operates a self-hosted implementation of Terraform Cloud, managing around 160,000 terraform resources across five data centers. About 50 engineers from across the engineering org release some form of infrastructure change within a given month -- some are infrastructure specialists, and others are application engineers who are completely new to Terraform Cloud. Understandably, we get a lot of questions about how to use Terraform Cloud or how to debug a specific issue, and that forum is usually in Slack. We have a glorious 20-page FAQ in Confluence that answers most questions, supplemented by numerous Slack threads documenting previous problems and their eventual solutions. So we have good documentation, but finding it is a pain. Who wants to read through a 20-page FAQ? Or go Slack spelunking to find that answer 40 messages deep into a thread? We set out to solve this problem by building a Slackbot that could dynamically answer any user question without doing any tedious searching. To accomplish this we implemented a Retrieval-Augmentated Generation (RAG) Large Language Model (LLM). Here's the story of how we did it and what we learned along the way. What we built We built an internal Slackbot that enables Benchling engineers to interact with a knowledge base to answer common Terraform Cloud questions. It also serves as a reference implementation for future LLM-powered tools at Benchling. It demonstrates how we can combine disparate information sources, both internal and public (web, Slack, Confluence), with the latest Large Language Models to expose this to the user through a familiar Slack interface. This pattern can be reused to develop Slack assistants for other specialized knowledge areas such as answering HR questions, surfacing past solutions to customer issues, or explaining software error codes. Here's what the interfa ## Scaling Scientific Data: Migrating Benchling's Schema Model for Performance at Scale DevFeed: [Scaling Scientific Data: Migrating Benchling's Schema Model for Performance at Scale]() Original publisher: [Read original article]() Author: Melody Ding Published: 2024-12-04T15:01:06Z Content type: article Language: en Sources: [Benchling]() Topics: [data](), [Structured-data](), [data-processing](), [Database](), [PostgreSQL]() Tags: [applications](), [automated](), [benchling](), [collection](), [core](), [customer](), [customers](), [data](), [data-modeling](), [database-optimization](), [model](), [performance](), [postresql](), [product](), [scalable-architecture](), [scale](), [schema](), [science](), [software-engineering](), [speed](), [storage](), [structure](), [tests]() ### AI overview Benchling describes migrating its Schema Model to a more compact structure to improve data-ingestion performance as scientific data volumes grow. The phased transition aimed to balance speed and flexibility while avoiding PostgreSQL scaling limitations, including the need for sharding. ### Source excerpt Benchling is a unified platform for scientific data. It allows scientists to collaborate on complex science, automate work, and power AI. Customers store large volumes of data on our platform, leveraging it across many applications both within Benchling and in their own infrastructure. It's critical that customer data is accessible in a performant and scalable way. In this article, we'll explore a recent shift in how we store and retrieve customer data. By migrating to a more compact structure, we've tackled key performance challenges associated with increased data volumes. This transition has required a careful balance between speed and flexibility, as well as a phased approach that minimized disruption for users. Benchling Schemas At the core of Benchling's system is Schemas, a product that allows both Benchling internal teams and customers to configure the various shapes of data, defining fields, attributes, and constraints that entities must follow. These data structures represent entities like equipment, storage, biological molecules, workflows, tasks, lab notes, and recorded results from scientific tests. Schemas reside in what we refer to as the definition layer. An example schema for defining the data structure of a molecule Each instance of a schema, referred to as a schematizable item, represents the actual data input by scientists. We call this the instance layer. These items are populated with field values conforming to the schema's defined fields. As Benchling's user base grows and the amount of schematized data ingested into the platform increases every year, optimizing the storage of field values has become crucial. Relationship between actual instances of a molecule and its defined schemaThe Challenge: Scale and Performance Historically, Benchling saw a shift from manual data upload by scientists to integrations with lab equipment, leading to automated data collection. This significantly increased the speed and volume of data ingestion. Assay results ## A behind-the-scenes look at building interactive analysis capabilities in Benchling DevFeed: [A behind-the-scenes look at building interactive analysis capabilities in Benchling]() Original publisher: [Read original article]() Author: Wonja Fairbrother Published: 2024-06-11T13:01:25Z Content type: article Language: en Sources: [Benchling]() Topics: [data-processing](), [web applications](), [Amazon EKS](), [Amazon S3](), [API](), [Back end](), [Front end](), [JSON]() Tags: [apache-arrow](), [apache-parquet](), [api](), [architecture](), [backend](), [data](), [data-analysis](), [data-transformation](), [data-visualization](), [duckdb](), [frontend](), [json](), [real-time](), [s3](), [storage](), [systems]() ### AI overview This article explains the architecture and design decisions behind Benchling's Interactive Analysis capabilities. The system lets scientists select data from multiple sources, transform and analyze it in real time, and visualize results within Benchling. Its architecture uses the Benchling web application, a stateless service running on EKS, temporary S3 storage, and a JSON-based transformation API. ### Source excerpt Authors: Wonja Fairbrother and Eli Levine Science is iterative. To design the next experiment, scientists need to analyze the results of previous ones. Interactive Analysis in Benchling allows scientists to perform real-time data transformation, visualization, and analysis without having to transfer it into other systems. In this post we will describe the architecture behind interactive analysis capabilities in Benchling and give a peek into the decision journey we took along the way¹. Interactive Analysis allows scientists to: 1. Select data from many sources: Benchling entity and results data Instrument data Notebook tables Data upload via both API and UI 2. Transform, visualize, and analyze data in real time, without leaving Benchling: Data transformations: filtering, aggregations, window functions, etc. Visualizations: line chart, bar chart, scatter plot, etc. Scientific analysis methods: IC50 and various curve fitting functions Overall architecture The architecture backing Interactive Analysis consists of: The Benchling web application An auto-scaling stateless internal service running on EKS that performs the transformations Temporary S3 storage locations for input and output data, shared between the web app and the service The frontend of the application is responsible for taking in input datasets and transformation configurations from users. The backend of the web application collects all the input data from the appropriate sources, serializes and uploads the data to S3, and sends a synchronous transformation request to the service. The service's API consists of one main endpoint that takes in a JSON payload of transformation parameters. The service can accept a single transformation, or a list of many transformations to perform. In this endpoint, the service downloads and deserializes the input data, performs the transformation with an analysis engine, and serializes and uploads the resulting data to S3. Each request spins up its own self-contained in-memor ## Signals, shells, and docker: an onion of footguns DevFeed: [Signals, shells, and docker: an onion of footguns]() Original publisher: [Read original article]() Author: raylu Published: 2024-05-22T16:01:32Z Content type: article Language: en Sources: [Benchling]() Topics: [POSIX](), [Bash](), [Processes](), [Docker]() Tags: [bash](), [docker](), [linux](), [logs](), [pipeline](), [pull-request](), [pytest](), [shell-script](), [terminal](), [testing]() ### AI overview This article investigates surprising POSIX signal behavior across shells and containers in Benchling's CI test pipeline. It explains how canceled test runs stopped forwarding logs while pytest continued running, then examines signal propagation through zsh, bash, and child processes. ### Source excerpt On a few occasions, we've needed to debug POSIX signals (SIGINT, SIGTERM, etc.). Inevitably, there's a shell involved too. One day, we were debugging some weird interaction between signals, shells, and containers and found ourselves bamboozled by some behaviors. People who consider themselves knowledgeable about Linux have found some of the details of our investigation surprising, so read on if this sort of thing doesn't make you want to defenestrate your laptop and become an alpaca-farming hermit. The scene of the crime At Benchling, we have a pretty standard testing/continuous integration (CI) setup: when you push code to a pull request branch, we run tests for you. A few years back, we added a little optimization: if you push again and tests are still running on the previous commit, we cancel the previous test run. You probably don't care about that run anyway and we save some money... or do we? The code that runs our tests is basically def test_pipeline() -> int: test_result = subprocess.run(["pytest", ...]) report_test_metrics() upload_artifacts() return test_result.returncode So our process tree is test_pipeline └──pytest subprocess.run blocks until the child process exits, so it should take almost all the time. We see in our CI logs that the tests get interrupted halfway through and then we see no more logs, so it sure looks like it's working. But we're able to get metrics and artifacts for our canceled runs, which makes no sense. We'll later discover that while we reported that the run was canceled and stopped forwarding logs, pytest just kept running. Back to basics Thinking that perhaps the problem was not forwarding a signal from test_pipeline to pytest, we thought about basic signal handling first. In a terminal running zsh, we can get the pid of zsh with $ echo $$ 20147 Then, we can run bash inside zsh and sleep infinity (like our tests, a very slow command) inside bash. $ bash $ sleep infinity From another shell, we can see the process tree. $ pstree -p 20 ## 10x faster python test iteration via fork(2) DevFeed: [10x faster python test iteration via fork(2)]() Original publisher: [Read original article]() Author: raylu Published: 2023-07-20T16:01:45Z Content type: tutorial Language: en Sources: [Benchling]() Topics: [Python](), [Testing](), [SQLAlchemy](), [modules](), [import](), [Code]() Tags: [benchling](), [code](), [dependencies](), [developer-productivity](), [engineering](), [fork](), [import](), [modules](), [python](), [sqlalchemy](), [testing]() ### AI overview This Benchling Engineering article explains how the Build team reduced Python test iteration time in a dependency-heavy codebase. It discusses the limitations of importlib.reload() and describes a fork-based approach that made the second test run start 10 times faster, reducing waiting by 90%. ### Source excerpt It's ideal to get feedback on your code faster -- to make a code change and see the result instantly. But, as projects get larger, reload times get longer. Each incremental dependency or bootstrap code block that adds 200ms feels worth it, but 50 of them later and it takes 10 seconds to see the result of a code change. On the Build team at Benchling, that's where we found ourselves one day. We used 146 packages which pull in 128 transitive dependencies for a total of 274 packages. We also spent a lot of time waiting for SQLAlchemy models to initialize. The result is our test harness took 10 seconds to set up. After making a code change, you'd start the test runner, wait a few seconds, alt+tab to your browser, get distracted for a few minutes, and then find out you had a typo in your code. This is a common challenge for a growing codebase, but it's something we knew we needed to fix. Here's the process we arrived at which allowed the second run of tests to start 10x faster -- 90% less waiting. While it'll work a little differently for your codebase depending on the language, dependencies, etc. you're using, hopefully this can inspire you on your journey to faster feedback and testing. importlib.reload() Since the problem is that we spend so long setting up a bunch of modules just right and then want to see the change in a single file we're editing, the most obvious solution is to use importlib.reload from the standard library. import importlib import sys import test_harness_stuff # takes 10 seconds import tests def rerun_tests(changed_path): for mod in sys.modules.values(): if mod.__file__ == changed_path: importlib.reload(mod) tests.run_tests() break if __name__ == '__main__': setup_file_watcher(rerun_tests) tests.run_tests() This (with some special handling for built-in modules, relative path resolution, and batching to handle editors that perform multiple filesystem operations per save) works alright when the file being changed is a test file (or any other leaf node ## Exposing AWS KMS Asymmetric Keys as a JWKS DevFeed: [Exposing AWS KMS Asymmetric Keys as a JWKS]() Original publisher: [Read original article]() Author: Brian Maloney Published: 2023-02-02T20:53:31Z Content type: tutorial Language: en Sources: [Benchling]() Topics: [OpenID connect (OIDC)](), [Authentication](), [Authorization](), [Security](), [Amazon Web Services](), [JSON Web Tokens](), [API](), [threat detection]() Tags: [api](), [authentication](), [authorization](), [aws](), [benchling](), [encryption](), [engineering](), [enterprise](), [identity](), [jwks](), [least-privilege](), [oauth](), [oauth-2-0](), [oauth2](), [oidc](), [openid-connect](), [private-key](), [public-key](), [secrets](), [security](), [threat-detection](), [token]() ### AI overview This Benchling article discusses service-to-service authentication with OAuth 2.0 and OpenID Connect. It explains why administrator-generated API tokens create secret-management, attribution, and least-privilege problems, and introduces private_key_jwt authentication with public-key cryptography while considering AWS KMS for private-key management and JWKS exposure. ### Source excerpt Here at Benchling, interaction with services is a large part of our business, from employees interacting with the software-as-a-service products with which we conduct our daily business, all the way down to interactions between the services that make up the Benchling application platform itself. Secure authentication and authorization to services is a long-standing issue in the industry, but one that has been improving in recent years due to the widespread adoption of modern standards such as OAuth 2.0 and OpenID Connect (OIDC). One specific use case for service-to-service authentication that is important to Benchling Security is connecting our Threat Detection Pipeline to our enterprise identity services vendor. We use this connection to connect log and other data provided by the vendor to our centralized Threat Detection Platform, where we correlate this with other sources of intelligence to detect risky or suspicious user activity in near real-time. Modern Authentication with OIDC Our specific identity services vendor offers two options for authenticating to its API: either an API token that an administrator can generate, or interaction by acting as an Application. API tokens, while very easy to use, are a poor choice for two reasons: First, they are a static secret that must be handled carefully and rotated frequently to mitigate the risk of a leaked key, which causes significant management overhead. Second, the identity services vendor links the privileges and identity of an API token inextricably to the administrator who generated it. This causes actions using the key to be attributed to the administrator and also makes it impossible to implement the principle of least privilege. Client authentication when acting as an Application allows the use of OIDC, and this vendor specifically requires the use of the private_key_jwt Client Authentication method. Enforcing this requirement is a good choice on the part of the vendor -- by using public-key encryption, no sec