# Effect-Level Validation for Causal Discovery in Interactive Telemetry

DevFeed: [Effect-Level Validation for Causal Discovery in Interactive Telemetry](<https://devfeed.tech/articles/effect-level-validation-for-causal-discovery-in-interactive-telemetry-131878.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.08340>)

Author: Hoang Dang, Luan Pham, Minh Nguyen

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [causality](<https://devfeed.tech/topics/causality.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [ate](<https://devfeed.tech/tags/ate.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [effect](<https://devfeed.tech/tags/effect.md>), [game](<https://devfeed.tech/tags/game.md>), [gameplay](<https://devfeed.tech/tags/gameplay.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [retention](<https://devfeed.tech/tags/retention.md>), [rpg](<https://devfeed.tech/tags/rpg.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

The framework evaluates causal discovery at the target-effect level: it retains graphs that identify the effect and have adequate treatment-control overlap, then checks admissible estimates with cross-algorithm stability, placebo, subsampling, and E-value diagnostics. For the RPG telemetry query, admissible DAGs yield a positive, cohort-conditional retention effect smaller than the raw gap; its sign survives placebo and subsampling checks, while its magnitude varies with adjustment and constraints. Under minimal constraints, admissible DAG runs leave PvP without parents and estimate the confounded raw difference; under full constraints, all DAG runs pass. PAG runs are reported separately. Simulations show the checks cannot rule out unmeasured confounding, and identifiability rates do not transfer to a second game dataset.

## Source excerpt

arXiv:2602.08340v2 Announce Type: replace Abstract: Causal discovery is increasingly used to analyze telemetry from interactive systems, but a plausible graph does not guarantee an identifiable or reliable product decision. We study this problem for one query: whether early competitive gameplay increases Day-1 retention in a deployed Role-playing game (RPG). We propose an admissibility-first framework that treats discovered graphs as structural hypotheses, keeps only graphs that identify the target effect and have adequate treatment-control overlap, and validates admissible effects through cross-algorithm stability, placebo, subsampling, and E-value diagnostics. On real telemetry, under minimal constraints the admissible runs that return Directed Acyclic Graphs (DAGs) are those whose graphs leave player-versus-player (PvP) with no parents, so they estimate the confounded raw difference; under the full constraint set all are admissible. Runs returning Partial Ancestral Graphs (PAGs) are not evaluated by the DAG gate and are reported separately. The admissible DAGs agree on one adjustment set and return a positive, cohort-conditional risk-difference average treatment effect (ATE), down from a raw retention gap; the estimate keeps its sign under placebo and subsampling checks, but its magnitude changes with the adjustment set and the constraint specification. A known-truth simulation shows that the checks screen identification and stability given the measured variables and cannot certify the absence of unmeasured confounding, and a second game-telemetry dataset shows that identifiability rates do not transfer. Graph-level recovery is therefore a weak proxy for causal reliability; discovery pipelines for decision support should be evaluated at the query and effect level.