# The CI Flake

DevFeed: [The CI Flake](<https://devfeed.tech/articles/the-ci-flake-30838.md>)

Original publisher: [Read original article](<https://hookrace.net/blog/the-ci-flake/>)

Published: 2024-08-15T22:00:00Z

Content type: article

Language: en

Sources: [Dennis Felsing](<https://devfeed.tech/sources/dennis-felsing.md>)

Topics: [ci](<https://devfeed.tech/topics/ci.md>), [Docker Compose](<https://devfeed.tech/topics/docker-compose.md>), [debug](<https://devfeed.tech/topics/debug.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [bugs](<https://devfeed.tech/tags/bugs.md>), [ci](<https://devfeed.tech/tags/ci.md>), [continuous-integration](<https://devfeed.tech/tags/continuous-integration.md>), [debug](<https://devfeed.tech/tags/debug.md>), [docker](<https://devfeed.tech/tags/docker.md>), [docker-compose](<https://devfeed.tech/tags/docker-compose.md>), [integration](<https://devfeed.tech/tags/integration.md>), [python](<https://devfeed.tech/tags/python.md>)

## AI overview

The article investigates a rare flaky test failure in Materialize CI that became more frequent after tests moved from AWS to Hetzner Cloud. It documents attempts to diagnose the issue, including flushing a generated docker-compose.yaml file, reproducing the failure, and examining parallel execution and CI setup.

## Source excerpt

I analyzed a flaky test failure in our Materialize CI today: $ docker compose up -d --scale default=0 default no such service: default mzcompose: error: running docker compose failed (exit status 1) I had seen this error already once or twice in the last year, but it was incredibly rare in our Continuous Integration (CI) runs, and never happened locally. As usual, there were more pressing product issues to debug, so I never looked into it. But last week I switched most of our CI tests to run on Hetzner Cloud instead of AWS to save some money. Suddenly this issue started occurring more often in CI, so my thinking was that it must somehow be timing-dependent. Before investigating my first instinct was that we are not writing the docker-compose.yaml file properly, which had already led to flaky calls to Docker Compose before (source code): file = self.files.get(thread_id) if not file: file = TemporaryFile(mode="w") os.set_inheritable(file.fileno(), True) yaml.dump(self.compose, file) self.files[thread_id] = file So yesterday I added a file.flush() after the yaml.dump and hoped to be done with it. This morning I woke up and the issue was still occurring! Based on the logged docker call this should be the code causing the problem (source code): def handle_composition( self, args: argparse.Namespace, composition: Composition ) -> None: if args.workflow not in composition.workflows: # Restart any dependencies whose definitions have changed. # This is Docker Compose's default behavior for `up`, but # not for `run`, which is a constant irritation that we # paper over here. The trick, taken from Buildkite's # Docker Compose plugin, is to run an `up` command that # requests zero instances of the requested service. if args.workflow: composition.invoke( "up", "-d", "--scale", f"{args.workflow}=0", args.workflow, ) super().handle_composition(args, composition) else: [...] Running the test in an endless loop locally had no success of reproducing the issue: while true; do bin/mzcom