# All Dashboards Green, All Screens Black

DevFeed: [All Dashboards Green, All Screens Black](<https://devfeed.tech/articles/all-dashboards-green-all-screens-black-26982.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/all-dashboards-green-all-screens-black-bcdb4a175633?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-09-15T16:31:01Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [digital experience monitoring](<https://devfeed.tech/topics/digital-experience-monitoring.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Grafana Cloud Frontend Observability](<https://devfeed.tech/topics/grafana-cloud-frontend-observability.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [black](<https://devfeed.tech/tags/black.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [client](<https://devfeed.tech/tags/client.md>), [client-side-security](<https://devfeed.tech/tags/client-side-security.md>), [component](<https://devfeed.tech/tags/component.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [e2e](<https://devfeed.tech/tags/e2e.md>), [experience](<https://devfeed.tech/tags/experience.md>), [external](<https://devfeed.tech/tags/external.md>), [green](<https://devfeed.tech/tags/green.md>), [incident](<https://devfeed.tech/tags/incident.md>), [journey](<https://devfeed.tech/tags/journey.md>), [livestream](<https://devfeed.tech/tags/livestream.md>), [observability](<https://devfeed.tech/tags/observability.md>), [party](<https://devfeed.tech/tags/party.md>), [product](<https://devfeed.tech/tags/product.md>), [production](<https://devfeed.tech/tags/production.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

## AI overview

A live update to a third-party security script caused black video screens for about 3,000 users in the first 30 minutes, while most component dashboards stayed green. End-to-end SLO monitoring detected the broken livestream journey and paged teams within five minutes. The system models each user action as a story: correlated events must show video, auction details, and chat within 60 seconds. A worker checks completed stories each minute and emits success metrics to Datadog. To prevent retries from skewing results, each client contributes its worst result per minute; expected outcomes such as a seller blocking entry are excluded from failures. Client telemetry uses local storage, compression, and retries, but fully offline clients cannot appear in live graphs. Instrumented stories reach about 99.5% SLI, and hundreds of thousands of events per minute are pushing ClickHouse queries toward their limit.

## Source excerpt

Karol Gil | Reliability Platform (Poland) On June 8, 2026, newly loaded web clients began showing black screens instead of livestream video. For our platform, that's a serious problem: it's pretty hard to sell Pokémon cards that no one can see. It turned out that a third-party script we use for client-side security monitoring wasn't bundled with our release, but was rather fetched live from the provider's CDN. When the provider updated the script all new web clients fetched it, and it included an additional URL validation which our video provider didn't pass. The result? Black screens for users of the affected web clients, with most internal dashboards staying green. 3,000 users were impacted in the first 30 minutes of the incident. One system did catch it. Our end-to-end service-level objective (E2E SLO) monitoring was already in production and paged the owning teams within five minutes. Here's what it saw. The real problem Most of our dashboards stayed green because they monitor component-level health: a server, an endpoint, a specific function. These are all useful, but can all be healthy while the actual user experience is completely broken. This problem gets worse the more external dependencies there are, or the more sophisticated an experience you want to deliver. In complex, integrated product experiences like ours, a "small" problem can have an outsize impact on the user experience. Measuring this requires a different approach to observability, namely, to model the user journey across multiple surfaces that must be true for a customer to have a good experience. So how do we measure this in a complex distributed application? Joining a livestream is not one thing Joining a livestream sounds like one action, but the user expects at least three things: Video to be playing Auction details to be shown Chat to be visible and up to da Each of those can succeed or fail completely independently of the other two. Our video depends on third-party providers and CDN netwo