# Fixing the trouble with Buildbot

DevFeed: [Fixing the trouble with Buildbot](<https://devfeed.tech/articles/fixing-the-trouble-with-buildbot-21560.md>)

Original publisher: [Read original article](<http://lackingrhoticity.blogspot.com/2011/08/fixing-trouble-with-buildbot.html>)

Author: Mark Seaborn (noreply@blogger.com)

Published: 2011-08-23T15:59:00Z

Content type: article

Language: en

Sources: [Mark Seaborn](<https://devfeed.tech/sources/mark-seaborn.md>)

Topics: [Shell](<https://devfeed.tech/topics/shell.md>), [Bash](<https://devfeed.tech/topics/bash.md>), [Python](<https://devfeed.tech/topics/python.md>), [make](<https://devfeed.tech/topics/make.md>), [x86](<https://devfeed.tech/topics/x86.md>), [Chromium](<https://devfeed.tech/topics/chromium.md>)

Tags: [atomic](<https://devfeed.tech/tags/atomic.md>), [chrome](<https://devfeed.tech/tags/chrome.md>), [code](<https://devfeed.tech/tags/code.md>), [make](<https://devfeed.tech/tags/make.md>), [python](<https://devfeed.tech/tags/python.md>), [script](<https://devfeed.tech/tags/script.md>), [shell-script](<https://devfeed.tech/tags/shell-script.md>), [tests](<https://devfeed.tech/tags/tests.md>), [x86](<https://devfeed.tech/tags/x86.md>)

## AI overview

The article explains how Buildbot Annotations let checked-in build scripts divide sequential build output into separately displayed steps. It describes the benefits for maintaining and testing Native Client build logic, while noting limitations including per-step timeout handling, non-nestable steps, and an awkward syntax.

## Source excerpt

Last year I wrote a blog post, "The trouble with Buildbot", about how Buildbot creates a dilemma for complex projects because it forces you to choose between two ways of describing a project's build steps: You can describe build steps in the Buildbot config. Buildbot configs are awkward to update -- someone has to restart the Buildbot master -- and hard to test, but you get the benefit that the build steps appear as separate steps in Buildbot's display. You can write a script which runs the build steps directly, and check it into the same repository as your project. This is easier to maintain and test, but traditionally all the output from the script would appear as a single Buildbot build step, making the output hard to read. Fortunately, Brad Nelson has addressed this problem with an extension to Buildbot known as "Buildbot Annotations". The Python code for this currently lives in chromium_step.py (see AnnotatedCommand). The idea is that your checked-in script will run multiple steps sequentially but output tags between them (e.g. "@@@BUILD_STEP tests@@@") so that the output can be parsed into chunks by the Buildbot master, and displayed as separate chunks. For example, an early version of Native Client's Annotations-based buildbot script looked something like this: ... echo @@@BUILD_STEP gyp_compile@@@ make -C .. -k -j12 V=1 BUILDTYPE=${GYPMODE} echo @@@BUILD_STEP scons_compile${BITS}@@@ ./scons -j 8 -k --verbose ${GLIBCOPTS} --mode=${MODE}-host,nacl \ platform=x86-${BITS} echo @@@BUILD_STEP small_tests${BITS}@@@ ./scons -k --verbose ${GLIBCOPTS} --mode=${MODE}-host,nacl small_tests \ platform=x86-${BITS} || { RETCODE=$? && echo @@@STEP_FAILURE@@@;} ... (More recently, this shell script has been replaced with a Python script.) You can see this in use on the Native Client Buildbot page (and also on the trybot page, though that's less readable). The logic for running NaCl's many build steps -- including a clobber step, a Scons build, a Gyp build, small_tests, mediu