# Networking: Using Linux Traffic Control for Fun and Profit Loss Prevention

DevFeed: [Networking: Using Linux Traffic Control for Fun and Profit Loss Prevention](<https://devfeed.tech/articles/networking-using-linux-traffic-control-for-fun-and-profit-loss-prevention-19706.md>)

Original publisher: [Read original article](<https://word.bitly.com/post/67486390974>)

Author: Wordbitly

Published: 2013-11-19T19:43:00Z

Content type: article

Language: en

Sources: [Bitly](<https://devfeed.tech/sources/bitly.md>)

Topics: [networking](<https://devfeed.tech/topics/networking.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Network design](<https://devfeed.tech/topics/network-design.md>), [Ethernet](<https://devfeed.tech/topics/ethernet.md>), [Linux](<https://devfeed.tech/topics/linux.md>)

Tags: [ethernet](<https://devfeed.tech/tags/ethernet.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [linux](<https://devfeed.tech/tags/linux.md>), [migration](<https://devfeed.tech/tags/migration.md>), [network](<https://devfeed.tech/tags/network.md>), [networking](<https://devfeed.tech/tags/networking.md>)

## AI overview

A Bitly technical account of migrating a physical Hadoop cluster and the resulting network overload. A fast distcp transfer generated enough cross-cabinet traffic to cause errors, timeouts, and failed internal DNS queries for website users and API clients.

## Source excerpt

Here at bitly, we are big fans of data, tubes and especially tubes that carry data. This is a story about asking tubes to carry too much data. A physical hadoop cluster has been a significant part of bitly's infrastructure and a core tool of the bitly Data Science and Ops/Infra teams for some time. Long enough that we needed to cycle in a new cluster, copy data, and fold the old into the new. Branches were opened, work was done, servers provisioned and the new cluster was stood up. Time to take a step back from the story and flesh out some technical details: bitly operates at a consequential scale of data: At the time of this migration the hadoop cluster was just over 150TB consumed disk space of compressed stream data, that is data that is the valuable output of our various applications after having been manipulated and expanded on by other applications. bitly's physical presence is collocated with our data center partner. There are three physical chassis classes (application, storage and database) racked together in contiguous cabinets in rows. At the time of this story each chassis had three physical 1Gb Ethernet connections (each logically isolated by VLANs), frontlink, backlink and lights-out (for out of band management of the server chassis). Each connection, after a series of cabinet specific patch panels and switches, connects to our core switches over 10Gb glass in a hub and spoke topology. While bitly also operates at a consequential physical scale (hundreds of physical server chassis), we depend on our data center partner for network infrastructure and topology. This means that within most levels of the physical networking stack, we have severely limited control and visibility. Back to the story: The distcp tool bundled with hadoop allowed us to quickly copy data from one cluster to the other. Put simply, distcp tool creates a mapreduce job to shuffle data from one hdfs cluster to another, in a many to many node copy. Distcp was fast, which was good. bitl