# How a central rsyslog server outage froze Bandcamp applications and services

DevFeed: [How a central rsyslog server outage froze Bandcamp applications and services](<https://devfeed.tech/articles/be-careful-how-you-rsyslog-19690.md>)

Original publisher: [Read original article](<https://bandcamptech.wordpress.com/2015/04/28/be-careful-how-you-rsyslog/>)

Author: Leigh Dyer

Published: 2015-04-28T18:29:22Z

Content type: opinion

Language: en

Sources: [Bandcamp](<https://devfeed.tech/sources/bandcamp.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Server](<https://devfeed.tech/topics/server.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [central](<https://devfeed.tech/tags/central.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [logging](<https://devfeed.tech/tags/logging.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [outage](<https://devfeed.tech/tags/outage.md>), [server](<https://devfeed.tech/tags/server.md>), [services](<https://devfeed.tech/tags/services.md>), [tcp](<https://devfeed.tech/tags/tcp.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

## AI overview

Bandcamp describes how maintenance on its central rsyslog server caused remote logging over TCP to block application-server logging. A shared delivery queue filled up, eventually freezing applications and system services; the team addressed the issue with a dedicated remote-logging queue that spills over to disk.

## Source excerpt

Bandcamp was offline briefly yesterday due to what I like to call an unexpected single point of failure. Good systems design is all about addressing single points of failure, making sure you have redundancies in place, but sometimes you discover single points of failure that you didn't realise you had. Yesterday's problem was caused by [...]