# testdev Published articles for testdev. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Digging into regressions DevFeed: [Digging into regressions]() Original publisher: [Read original article]() Author: elvis314 Published: 2019-08-23T20:53:47Z Content type: opinion Language: en Sources: [Mozilla Automation]() Topics: [bug](), [data](), [Mozilla](), [monitor](), [Web platform]() Tags: [bug](), [data](), [monitor](), [mozilla](), [platform](), [regression](), [testdev](), [tests](), [web]() ### AI overview The article examines how Mozilla tracks regressions detected by builds and tests after patches land on autoland. It proposes distinguishing unique regressions and product fixes, while considering data accuracy, backouts, and the potential impact of changing or disabling test jobs. ### Source excerpt Whenever a patch is landed on autoland, it will run many builds and tests to make sure there are no regressions. Unfortunately many times we find a regression and 99% of the time backout the changes so they can be ... Continue reading -> ## Looking at Firefox performance 57 vs 63 DevFeed: [Looking at Firefox performance 57 vs 63]() Original publisher: [Read original article]() Author: elvis314 Published: 2018-09-12T18:17:51Z Content type: opinion Language: en Sources: [Mozilla Automation]() Topics: [Firefox](), [browser](), [Benchmark](), [Hardware](), [Document Object Model (DOM)](), [Monitoring](), [Mozilla]() Tags: [benchmark](), [browser](), [firefox](), [general](), [hardware](), [monitoring](), [mozilla](), [performance](), [quantum](), [testdev]() ### AI overview This article reviews Firefox performance measurements from version 57, also known as Firefox Quantum, through version 63. It examines Speedometer, startup time, and page-load tests, noting that a hardware upgrade affected some results and that the tests showed no regressions overall, with modest improvements in several areas. ### Source excerpt Last November we released Firefox v.57, otherwise known as Firefox Quantum. Quantum was in many ways a whole new browser with the focus on speed as compared to previous versions of Firefox. As I write about many topics on my ... Continue reading -> ## running tests by bugzilla component instead of test suite DevFeed: [running tests by bugzilla component instead of test suite]() Original publisher: [Read original article]() Author: elvis314 Published: 2017-12-21T17:07:44Z Content type: article Language: en Sources: [Mozilla Automation]() Topics: [test-coverage](), [Test coverage](), [Tool]() Tags: [code-coverage](), [testdev](), [tests](), [tools](), [treeherder]() ### AI overview The article explores running tests by Bugzilla component instead of test suite or random chunks. It describes potential benefits including component-level code coverage, more deterministic SETA results, component filtering in Treeherder, and easier handling of intermittent tests, while noting that the approach is not production-ready. ### Source excerpt Over the years we have had great dreams of running our tests in many different ways. There was a dream of 'hyperchunking' where we would run everything in hundreds of chunks finishing in just a couple of minutes for all ... Continue reading -> ## Measuring the noise in Performance tests DevFeed: [Measuring the noise in Performance tests]() Original publisher: [Read original article]() Author: elvis314 Published: 2017-11-16T20:20:19Z Content type: article Language: en Sources: [Mozilla Automation]() Topics: [Hardware](), [data]() Tags: [hardware](), [metrics](), [performance](), [testdev](), [tests](), [tools]() ### AI overview The article examines noise in Talos performance-test results, defining it as greater standard deviation and explaining how it can obscure regressions, trigger false alerts, and affect hardware evaluation. It describes using repeated data points, comparison views, and variance calculations to assess reliability when introducing new hardware. ### Source excerpt Often I hear about our talos results, why are they so noisy? What is noise in this context- by noise we are referring to a larger stddev in the results we track, here would be an example: With the large ... Continue reading -> ## Keeping an eye on Performance alerts DevFeed: [Keeping an eye on Performance alerts]() Original publisher: [Read original article]() Author: elvis314 Published: 2017-11-01T20:17:46Z Content type: opinion Language: en Sources: [Mozilla Automation]() Topics: [Firefox](), [Monitoring](), [Testing](), [data]() Tags: [benchmarks](), [firefox](), [monitoring](), [performance](), [regression](), [testdev](), [testing]() ### AI overview The article describes how Firefox 57 performance monitoring expanded beyond Talos to include build metrics, AWSY, Autophone, and platform microbenchmarks. It explains how these systems track metrics, detect regressions, generate alerts, and associate sustained findings with bugs. ### Source excerpt Over the last 6 months there has been a deep focus on performance in order to release Firefox 57. Hundreds of developers sought out performance improvements and after thousands of small adjustments we see massive improvements. Last week I introduced ... Continue reading -> ## Stockwell: flowchart for triage DevFeed: [Stockwell: flowchart for triage]() Original publisher: [Read original article]() Author: elvis314 Published: 2017-10-23T19:50:29Z Content type: article Language: en Sources: [Mozilla Automation]() Topics: [bug](), [Development](), [monitor](), [Tool]() Tags: [bug](), [development](), [intermittent-failures](), [intermittents](), [monitor](), [process](), [testdev](), [treeherder]() ### AI overview This article describes Stockwell's workflow for triaging intermittent failures. Tree sheriffs associate failures with bugs in Treeherder and Orange Factor, after which the Neglected Oranges tool helps identify failures lacking recent human comments. Frequently failing bugs may be monitored and disabled when they exceed the stated failure threshold or lack resources for a fix. ### Source excerpt I gave an update 2 weeks ago on the current state of Stockwell (intermittent failures). I mentioned additional posts were coming and this is a second post in the series. First off the tree sheriffs who maintain merges between branches, ... Continue reading ->