# When Can LLMs Replace Humans in A/B Tests?

DevFeed: [When Can LLMs Replace Humans in A/B Tests?](<https://devfeed.tech/articles/when-can-llms-replace-humans-in-a-b-tests-154.md>)

Original publisher: [Read original article](<https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests/>)

Author: Spotify Engineering

Published: 2026-08-13T18:57:22Z

Content type: article

Language: en

Sources: [Spotify Engineering](<https://devfeed.tech/sources/spotify-engineering.md>), [Spotify Engineering Blog](<https://devfeed.tech/sources/spotify-engineering-blog.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [data](<https://devfeed.tech/topics/data.md>), [experiments](<https://devfeed.tech/topics/experiments.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

The article examines whether large language model predictions can replace human outcomes in A/B tests. Using the Upworthy dataset, it finds that calibrated predictions can recover treatment effects under specific assumptions, while raw predictions recovered only 39% of the observed human effect.

## Source excerpt

TL;DR: LLM predictions can stand in for human outcomes in A/B tests, but only by assumption, not by design.... The post When Can LLMs Replace Humans in A/B Tests? appeared first on Spotify Engineering.