# Signals, shells, and docker: an onion of footguns

DevFeed: [Signals, shells, and docker: an onion of footguns](<https://devfeed.tech/articles/signals-shells-and-docker-an-onion-of-footguns-20130.md>)

Original publisher: [Read original article](<https://benchling.engineering/signals-shells-and-docker-an-onion-of-footguns-ee592e2b587b?source=rss----3d4aa8fb07ea---4>)

Author: raylu

Published: 2024-05-22T16:01:32Z

Content type: article

Language: en

Sources: [Benchling](<https://devfeed.tech/sources/benchling.md>)

Topics: [POSIX](<https://devfeed.tech/topics/posix.md>), [Bash](<https://devfeed.tech/topics/bash.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [Docker](<https://devfeed.tech/topics/docker.md>)

Tags: [bash](<https://devfeed.tech/tags/bash.md>), [docker](<https://devfeed.tech/tags/docker.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logs](<https://devfeed.tech/tags/logs.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pull-request](<https://devfeed.tech/tags/pull-request.md>), [pytest](<https://devfeed.tech/tags/pytest.md>), [shell-script](<https://devfeed.tech/tags/shell-script.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

This article investigates surprising POSIX signal behavior across shells and containers in Benchling's CI test pipeline. It explains how canceled test runs stopped forwarding logs while pytest continued running, then examines signal propagation through zsh, bash, and child processes.

## Source excerpt

On a few occasions, we've needed to debug POSIX signals (SIGINT, SIGTERM, etc.). Inevitably, there's a shell involved too. One day, we were debugging some weird interaction between signals, shells, and containers and found ourselves bamboozled by some behaviors. People who consider themselves knowledgeable about Linux have found some of the details of our investigation surprising, so read on if this sort of thing doesn't make you want to defenestrate your laptop and become an alpaca-farming hermit. The scene of the crime At Benchling, we have a pretty standard testing/continuous integration (CI) setup: when you push code to a pull request branch, we run tests for you. A few years back, we added a little optimization: if you push again and tests are still running on the previous commit, we cancel the previous test run. You probably don't care about that run anyway and we save some money... or do we? The code that runs our tests is basically def test_pipeline() -> int: test_result = subprocess.run(["pytest", ...]) report_test_metrics() upload_artifacts() return test_result.returncode So our process tree is test_pipeline └──pytest subprocess.run blocks until the child process exits, so it should take almost all the time. We see in our CI logs that the tests get interrupted halfway through and then we see no more logs, so it sure looks like it's working. But we're able to get metrics and artifacts for our canceled runs, which makes no sense. We'll later discover that while we reported that the run was canceled and stopped forwarding logs, pytest just kept running. Back to basics Thinking that perhaps the problem was not forwarding a signal from test_pipeline to pytest, we thought about basic signal handling first. In a terminal running zsh, we can get the pid of zsh with $ echo $$ 20147 Then, we can run bash inside zsh and sleep infinity (like our tests, a very slow command) inside bash. $ bash $ sleep infinity From another shell, we can see the process tree. $ pstree -p 20