# Production metrics Bitly monitors beyond standard checks

DevFeed: [Production metrics Bitly monitors beyond standard checks](<https://devfeed.tech/articles/10-things-we-forgot-to-monitor-19709.md>)

Original publisher: [Read original article](<https://word.bitly.com/post/74839060954>)

Author: Wordbitly

Published: 2014-01-28T16:11:25Z

Content type: article

Language: en

Sources: [Bitly](<https://devfeed.tech/sources/bitly.md>)

Topics: [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [DevOps](<https://devfeed.tech/topics/devops.md>), [systems](<https://devfeed.tech/topics/systems.md>), [cURL](<https://devfeed.tech/topics/curl.md>), [Network Configuration](<https://devfeed.tech/topics/network-configuration.md>)

Tags: [curl](<https://devfeed.tech/tags/curl.md>), [devops](<https://devfeed.tech/tags/devops.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [network-configuration](<https://devfeed.tech/tags/network-configuration.md>), [ops](<https://devfeed.tech/tags/ops.md>), [process](<https://devfeed.tech/tags/process.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

This Bitly article describes production-monitoring lessons beyond standard disk, memory, load, and ping metrics. The supplied text discusses fork rate, network flow-control packets, swap-in/out rate, server boot notifications, and NTP clock offset, including incidents involving IPv6 configuration, curl, and dropped traffic.

## Source excerpt

There is always a set of standard metrics that are universally monitored (Disk Usage, Memory Usage, Load, Pings, etc). Beyond that, there are a lot of lessons that we've learned from operating our production systems that have helped shape the breadth of monitoring that we perform at bitly. One of my favorite all-time tweets is from @DevOps_Borat "Law of Murphy for devops: if thing can able go wrong, is mean is already wrong but you not have Nagios alert of it yet." What follows is a small list of things we monitor at bitly that have grown out of those (sometimes painful!) experiences, and where possible little snippets of the stories behind those instances. 1 - Fork Rate We once had a problem where IPv6 was intentionally disabled on a box via options ipv6 disable=1 and alias ipv6 off in /etc/modprobe.conf. This caused a large issue for us: each time a new curl object was created, modprobe would spawn, checking net-pf-10 to evaluate IPv6 status. This fork bombed the box, and we eventually tracked it down by noticing that the process counter in /proc/stat was increasing by several hundred a second. Normally you would only expect a fork rate of 1-10/sec on a production box with steady traffic. check_fork_rate.sh 2 - flow control packets TL;DR; If your network configuration honors flow control packets and isn't configured to disable them, they can temporarily cause dropped traffic. (If this doesn't sound like an outage, you need your head checked.) $ /usr/sbin/ethtool -S eth0 | grep flow_control rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 Note: Read this to understand how these flow control frames can cascade to switch-wide loss of connectivity if you use certain Broadcom NIC's. You should also trend these metrics on your switch gear. While at it, watch your dropped frames. 3 - Swap In/Out Rate It's common to check for swap usage above a threshold, but even if you have a small quantity of memory swapped, it's actually th