# Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

DevFeed: [Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss](<https://devfeed.tech/articles/testing-conversation-preservation-across-llm-apis-a-rule-based-oracle-for-detecting-silent-information-loss-131717.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15543>)

Author: Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao, Lei Wang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [open-responses](<https://devfeed.tech/topics/open-responses.md>), [rule-engine](<https://devfeed.tech/topics/rule-engine.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [apis](<https://devfeed.tech/tags/apis.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [controls](<https://devfeed.tech/tags/controls.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exceptions](<https://devfeed.tech/tags/exceptions.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [gateways](<https://devfeed.tech/tags/gateways.md>), [generation](<https://devfeed.tech/tags/generation.md>), [human-review](<https://devfeed.tech/tags/human-review.md>), [information](<https://devfeed.tech/tags/information.md>), [interoperability](<https://devfeed.tech/tags/interoperability.md>), [layers](<https://devfeed.tech/tags/layers.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-apis](<https://devfeed.tech/tags/llm-apis.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

The proposed isloss oracle tests whether LLM API translations preserve conversation information, catching losses that ordinary target schema checks can miss. It maps source and translated requests and replies to a shared representation, ConvIR, and uses explicit rules to separate legitimate format changes from avoidable loss and degradation due to target API limits. It needs neither a reference translation nor extra model calls. In evaluation, it detected 76% of planted faults with a 2% false alarm rate. Of 685 completed synthetic request translations, 18.5% passed static target checks while changing information the target could preserve; 36.4% of 558 translated reply streams caused exceptions or silent corruption in at least one tested official OpenAI SDK. Experiments across six models also found behavior changes when tasks depended on translated-away information.

## Source excerpt

arXiv:2611.15543v1 Announce Type: new Abstract: LLM applications increasingly rely on translators and compatibility layers to move conversations across APIs. Yet a translated request can be accepted by the target API while silently losing information about instruction roles, tool interactions, error states, or generation controls. We introduce isloss, a deterministic oracle for testing conversation-information preservation across LLM API translations. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to distinguish legitimate representation changes from information loss. Unlike schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It also distinguishes avoidable loss from degradation caused by target API limitations. We evaluate isloss using specification-based fault injection, human review, and experiments on translators, gateways, and client libraries. On held-out tests, it detects 95/125 planted faults (76.0%) with 2/100 false alarms (2.0%). Across a synthetic request suite spanning six configurations of five translator codebases, 127/685 completed translations (18.5%) pass static target checks while silently changing prompt, role, or tool information in ways the target API could have preserved. In translated replies, 203/558 streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted experiments across six models show that translation-induced changes can alter model behaviour when tasks depend on the affected information. These results establish conversation preservation as a distinct correctness requirement for reliable LLM interoperability, agent execution, and model evaluation.