# Bitly The Bitly engineering blog This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lessons Learned from a 2016 Data Center Move DevFeed: [Lessons Learned from a 2016 Data Center Move]() Original publisher: [Read original article]() Author: Wordbitly Published: 2017-11-30T19:53:27Z Content type: opinion Language: en Sources: [Bitly]() Topics: [migration](), [datacenter]() Tags: [data-center](), [eu](), [migration](), [talk]() ### AI overview The article shares lessons learned from a 2016 data center move and discusses what worked well and what to consider when planning a similar move. ### Source excerpt The Great Migration Recently we gave a talk at Velocity EU about the lessons learned from our big data center move in 2016. Check it out to learn more about what worked well and what you should consider when you are facing your own data center move. by Sean O'Connor ## Bitly Summer Intern Wrap 2015 DevFeed: [Bitly Summer Intern Wrap 2015]() Original publisher: [Read original article]() Author: Wordbitly Published: 2015-08-24T14:00:22Z Content type: opinion Language: en Sources: [Bitly]() Topics: [Front end](), [JavaScript](), [React](), [Functional programming](), [CSS](), [Ember](), [Python]() Tags: [css](), [frontend](), [functional-programming](), [javascript](), [python](), [react]() ### AI overview A Bitly summer intern describes working on the frontend team, creating a JavaScript A/B testing utility, and contributing to the BBT2 product. The article discusses using React, JSX, and Immutable JS, compares React with Ember, and mentions Python and CSS work. ### Source excerpt We've invited this year's summer interns to share their experiences from working on our engineering team. Our interns worked on the same problems as our full-time engineers and made a difference to our product and business on a massive scale. They also appeared on the Bitly Tech Podcast which you should totally check out! If you have an interest in making a difference on a massive scale then check out our job postings as we are hiring in both Denver and New York City! Nathaniel I'm Nathaniel and I'm writing this a few days before my awesome summer with Bitly comes to an end. I will be returning to Carnegie Mellon to start my 3rd year studying Computer Science but I have to say, I'm going to miss this place. I spent the summer interning with the extremely talented Frontend team here at Bitly. While here I've gotten to create a javascript AB testing utility and contribute to the soon-to-be-released BBT2 product. One of the best parts of working at Bitly was that even as an intern, I felt like I had voice at the company. This was extremely evident with my work on BBT2 where I got to contribute not just code, but also ideas and opinions. Oftentimes you hear stories about how interns get stuck in corners to work on projects that never get to see the light of day. Bitly does the exact opposite. We were given seats amongst everyone else, we were given projects that will actually affect the direction and success of the company, we got to (and were encouraged to) attend all meetings, and most importantly, we were made to feel like members of the team. I also learned a ton this summer. The new BBT2 product is being built in React JS with the help of Immutable JS. I've gotten to fully immerse myself in these tools and I've fallen in love with them. Being a TA for a functional programming class, Immutable JS brings me as close as a javascript framework could to the Eden that is functional programming. Then React combined with JSX helped us write very clean and modular code, som ## Bitly In Denver DevFeed: [Bitly In Denver]() Original publisher: [Read original article]() Author: Wordbitly Published: 2015-05-11T16:34:44Z Content type: news Language: en Sources: [Bitly]() Topics: [DevOps](), [elasticsearch](), [Messaging](), [Testing](), [API](), [Go Language](), [Kernel](), [Microservice](), [big-data](), [Hacking]() Tags: [api](), [big-data](), [devops](), [elasticsearch](), [go](), [hacking](), [linux](), [messaging](), [microservices](), [testing]() ### AI overview Bitly announces a week of Denver events from May 18-22 featuring members of its engineering team. The schedule includes talks and community events covering deployment, Elasticsearch aggregation, async Tornado testing, NSQ, APIs and microservices, messaging in Go, Linux kernel hacking, and big data, plus a free party with StatusPage.io. ### Source excerpt via flickr Want to meet and hang out with some of the Bitly engineering team? A bunch of us will be in Denver, Colorado the week of May 18-22. While we're out there we'll be catching up with our growing Denver-based team (we're hiring, by the way) and participating in a bunch of local tech events. Check out the schedule below and come join us: Monday 5/18 Sean will be talking about how we think about deployment at DevOps Boulder. Peter will be sharing some of the things we've learned about doing Aggregation in Elasticsearch at Elasticsearch Denver. Tuesday 5/19 Michael will be talking about testing async Tornado code with Mock at Front Range Pythoneers. Georgi will be talking about NSQ at Boulder Python Rob will be at All Things API talking APIs and microservices. Wednesday 5/20 Peter and Georgi will be speaking at Gluecon. We'll be hosting Boulder/Denver Big Data in partnership with our friends at MapQuest. Thursday 5/21 Peter will be talking about messaging in Go at Denver Gophers. Georgi will be talking about hacking the Linux kernel at Women Who Code. Friday 5/22 We'll be hosting a party with StatusPage.io! This party is free and open to everybody and gives you a chance to come hang out with a bunch of Denver's tech community. Please make sure to RSVP. Keep an eye on #bitlyindenver for live updates. Please come and join us at any of these events to join in the conversation, we can't wait to see you! by Sean O'Connor ## Frontend Dependency Management with Browserify DevFeed: [Frontend Dependency Management with Browserify]() Original publisher: [Read original article]() Author: Wordbitly Published: 2014-10-30T20:10:27Z Content type: tutorial Language: en Sources: [Bitly]() Topics: [frontend development](), [JavaScript](), [Package Management](), [Node.js](), [npm]() Tags: [browserify](), [dependency-management](), [frontend](), [javascript](), [node-js](), [npm](), [package-management]() ### AI overview This developer article explains how Bitly adopted Browserify for frontend dependency management while overhauling its "Your Bitlinks" interface. It describes Browserify's Node.js-style module and package workflow, including bundling dependencies from an entry point into a destination file, and discusses benefits such as fewer HTTP requests and a less-polluted global namespace. ### Source excerpt With frontend development moving as fast as it does at Bitly, things can get pretty messy. We found ourselves with piles of unmanaged script tags and little indication of what was still being used in the app's current iteration. There had to be a better way! Enter Browserify! A few months ago, we embarked upon a project to totally overhaul the "Your Bitlinks" interface, and with that came the opportunity to rethink our practices and introduce frontend dependency management. Now the page looks as good behind the scenes as it does in your browser! So, what is Browserify? Browserify is a great tool that lets you write your client-side scripts like it's node, allowing you to use node's package management and module systems. Each file should be a module and explicitly require() all its dependencies. Then you simply give Browserify your main entry point, such as an overarching app file, and the name of a destination file, e.g. browserify app.js > bundle.js This example takes your network of dependencies starting at app.js and streams all the files from your network of dependencies into bundle.js. (Side note: Being familiar with the node.js environment and NPM, node's package manager, is essential for picking up Browserify. In the weeks before we started working with Browserify, I wrote and published a node module as a side project. Going through the process, making sure all the pieces were in place-- specifically the package.json-- so that I could add it into the NPM registry made figuring out Browserify much easier.) Why use Browserify? There was no question in our minds that using a frontend dependency management tool was an important step in our app's development. We considered the field of options, ultimately settling on Browserify. We didn't need the async loading functionality of Require.js and appreciated the power of using NPM, leaving Browserify as the natural choice. Leveraging node to bundle our JavaScript provides many advantages over the traditional multiplicit ## Production metrics Bitly monitors beyond standard checks DevFeed: [Production metrics Bitly monitors beyond standard checks]() Original publisher: [Read original article]() Author: Wordbitly Published: 2014-01-28T16:11:25Z Content type: article Language: en Sources: [Bitly]() Topics: [Monitoring](), [DevOps](), [systems](), [cURL](), [Network Configuration]() Tags: [curl](), [devops](), [metrics](), [monitoring](), [network-configuration](), [ops](), [process](), [systems]() ### AI overview This Bitly article describes production-monitoring lessons beyond standard disk, memory, load, and ping metrics. The supplied text discusses fork rate, network flow-control packets, swap-in/out rate, server boot notifications, and NTP clock offset, including incidents involving IPv6 configuration, curl, and dropped traffic. ### Source excerpt There is always a set of standard metrics that are universally monitored (Disk Usage, Memory Usage, Load, Pings, etc). Beyond that, there are a lot of lessons that we've learned from operating our production systems that have helped shape the breadth of monitoring that we perform at bitly. One of my favorite all-time tweets is from @DevOps_Borat "Law of Murphy for devops: if thing can able go wrong, is mean is already wrong but you not have Nagios alert of it yet." What follows is a small list of things we monitor at bitly that have grown out of those (sometimes painful!) experiences, and where possible little snippets of the stories behind those instances. 1 - Fork Rate We once had a problem where IPv6 was intentionally disabled on a box via options ipv6 disable=1 and alias ipv6 off in /etc/modprobe.conf. This caused a large issue for us: each time a new curl object was created, modprobe would spawn, checking net-pf-10 to evaluate IPv6 status. This fork bombed the box, and we eventually tracked it down by noticing that the process counter in /proc/stat was increasing by several hundred a second. Normally you would only expect a fork rate of 1-10/sec on a production box with steady traffic. check_fork_rate.sh 2 - flow control packets TL;DR; If your network configuration honors flow control packets and isn't configured to disable them, they can temporarily cause dropped traffic. (If this doesn't sound like an outage, you need your head checked.) $ /usr/sbin/ethtool -S eth0 | grep flow_control rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 Note: Read this to understand how these flow control frames can cascade to switch-wide loss of connectivity if you use certain Broadcom NIC's. You should also trend these metrics on your switch gear. While at it, watch your dropped frames. 3 - Swap In/Out Rate It's common to check for swap usage above a threshold, but even if you have a small quantity of memory swapped, it's actually th ## Optimizing Text Processing for 1.75 Billion Lines of Hadoop Output DevFeed: [Optimizing Text Processing for 1.75 Billion Lines of Hadoop Output]() Original publisher: [Read original article]() Author: Wordbitly Published: 2014-01-21T16:34:03Z Content type: article Language: en Sources: [Bitly]() Topics: [Hadoop](), [data](), [Python]() Tags: [bash](), [data](), [hadoop](), [performance](), [python]() ### AI overview The article describes attempts to post-process 1.75 billion lines of Hadoop output from an EMR job. It examines Hadoop output plugins, secondary Hadoop jobs, and shell-based text-processing approaches for separating data by account and metric type, including their operational and performance problems. ### Source excerpt Lessons learned while post-processing 1.75 billion lines of Hadoop output. The Problem Recently, I encountered a problem. I had a nightly Hadoop job running on EMR that churned over the past 30 days' worth of Bitly redirect data in order to run reach analysis pertaining to about 1000 of our paid accounts. This job resulted in 175 gzipped "part" files, each containing at least 10 million lines of data. I needed to collate that data after the Hadoop job ran. > ls part-*.gz part-0000.gz ... part-0174.gz > zcat part-0000.gz | wc -l 10000000 The Hadoop output data inside the part files consisted of things like this: "3,g,05-02,12SIMV6" 329 "175,geo,05,US,GA,Atlanta" 9987 "10,phrase,05,egg foo young" 1093 "11,n_clicks,05" 393999 Those were comma-delimited keys with the following structure and a count: "[ACCOUNT_ID],[METRIC_TYPE],[DATE],[VALUE]" COUNT The challenge was this: How do I efficiently separate out this data by ACCOUNT_ID and METRIC_TYPE? That is, I wanted one file per ACCOUNT_ID-METRIC_TYPE combination. First, Look on the Shelf Like many people churning through volumes of data, we make use of the the mr_job python package for our Hadoop processing. At first I thought this was a no-brainer: "I'll use the oddjob plugin. Yay, a solution already exists!" The plugin's description was tailor-made for me: "oddjob.MultipleJSONOutputFormat - Writes to the directories specified by the first element in the key" - https://github.com/jblomo/oddjob Wrong. oddjob plugin wouldn't run at all on our Hadoop cluster. oddjob plugin wouldn't run consistently on EMR This approach resulted in 890 x 175 x 5 = ~800K part files. To scp 800K files from EMR is a nightmare of a long time. Secondary Hadoop Jobs After days of struggling with oddjob, I cut bait on it and looked at running a set of secondary Hadoop jobs, using the output from the first Hadoop job as input to the second ones. Something like this: for account_id in $account_ids do run_emr_job_to_extract $account_id done Even if ea ## Getting more clues from Python's logging module DevFeed: [Getting more clues from Python's logging module]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-12-05T16:01:54Z Content type: tutorial Language: en Sources: [Bitly]() Topics: [Python](), [Logging](), [web applications](), [Exception](), [Django]() Tags: [applications](), [bug](), [developers](), [development](), [django](), [exception](), [logging](), [python](), [troubleshooting](), [web-applications]() ### AI overview The article discusses diagnosing bugs and exceptions in Python web applications, especially production issues where development debug tools are unavailable. It describes how limited logging can create a repeated cycle of adding logging, deploying it, and waiting for the error to recur, and introduces more informative logging configuration as a way to gather useful context. ### Source excerpt During development of Python web applications, there are a lot of tools that can help provide clues about what caused a bug or exception. For example, Django's default debug error page prints out local variables at every stack frame when there's an unhandled exception. The popular django_debugtoolbar adds more information. In a similar vein there are things like Pyramid's pyramid_debugtoolbar and its predecessor WebError. But when troubleshooting production issues, those tools aren't available, for good reason: We need the useful information to be exposed to developers, not overwhelming our end users nor exposing sensitive information to malicious eyes. So, we turn to logging instead. But that provides a lot less information out of the box. So at bitly, we often found ourselves in an irritating cycle that went something like this: Sometime soon after a deploy, we notice an unusual number of exceptions being logged. (Let's say that old Python chestnut, 'NoneType' object is unsubscriptable.) So we know that we have some code that is expecting a dict or list and getting None instead. We may decide the error is not critical enough to roll back without first trying to diagnose the underlying cause, but of course we want to fix it quickly. The pressure is on. We find the traceback in our logs. It looks like: Python traceback example But where's the None? Is it response['data'] or is it one of the entry values? We have no way to see. (Tip: We can actually tell that it's not response['data']['subtotal']. Do you know why? Would you remember that while under pressure to fix a production bug?) We try, and fail, to provoke the error on our local development or staging environments. Then we add some logging code. We get a quick code review and deploy the logging code. We wait for the error to happen again. We find the log message and realize we forgot to log everything we wanted to see, or we need more contextual information. Go back 3 steps and repeat as needed. Finally we can ## Networking: Using Linux Traffic Control for Fun and Profit Loss Prevention DevFeed: [Networking: Using Linux Traffic Control for Fun and Profit Loss Prevention]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-11-19T19:43:00Z Content type: article Language: en Sources: [Bitly]() Topics: [networking](), [Hadoop](), [migration](), [Network design](), [Ethernet](), [Linux]() Tags: [ethernet](), [hadoop](), [infrastructure](), [linux](), [migration](), [network](), [networking]() ### AI overview A Bitly technical account of migrating a physical Hadoop cluster and the resulting network overload. A fast distcp transfer generated enough cross-cabinet traffic to cause errors, timeouts, and failed internal DNS queries for website users and API clients. ### Source excerpt Here at bitly, we are big fans of data, tubes and especially tubes that carry data. This is a story about asking tubes to carry too much data. A physical hadoop cluster has been a significant part of bitly's infrastructure and a core tool of the bitly Data Science and Ops/Infra teams for some time. Long enough that we needed to cycle in a new cluster, copy data, and fold the old into the new. Branches were opened, work was done, servers provisioned and the new cluster was stood up. Time to take a step back from the story and flesh out some technical details: bitly operates at a consequential scale of data: At the time of this migration the hadoop cluster was just over 150TB consumed disk space of compressed stream data, that is data that is the valuable output of our various applications after having been manipulated and expanded on by other applications. bitly's physical presence is collocated with our data center partner. There are three physical chassis classes (application, storage and database) racked together in contiguous cabinets in rows. At the time of this story each chassis had three physical 1Gb Ethernet connections (each logically isolated by VLANs), frontlink, backlink and lights-out (for out of band management of the server chassis). Each connection, after a series of cabinet specific patch panels and switches, connects to our core switches over 10Gb glass in a hub and spoke topology. While bitly also operates at a consequential physical scale (hundreds of physical server chassis), we depend on our data center partner for network infrastructure and topology. This means that within most levels of the physical networking stack, we have severely limited control and visibility. Back to the story: The distcp tool bundled with hadoop allowed us to quickly copy data from one cluster to the other. Put simply, distcp tool creates a mapreduce job to shuffle data from one hdfs cluster to another, in a many to many node copy. Distcp was fast, which was good. bitl ## Deploying All Day at bitly Without Breaking the Internet DevFeed: [Deploying All Day at bitly Without Breaking the Internet]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-11-12T16:05:42Z Content type: article Language: en Sources: [Bitly]() Topics: [Automation](), [systems](), [Messaging]() Tags: [architecture](), [automation](), [distributed](), [review](), [state-management](), [systems]() ### AI overview A video from Sean O'Connor's Devopsdays Portland talk about how bitly deploys continuously without breaking the Internet. The talk covers automation, state management, systems architecture, distributed messaging, and peer review. ### Source excerpt Deploying all day - Sean O Connor.mp4 from devopsdays on Vimeo. Recently we gave a talk at Devopsdays Portland about how we deploy all day at bitly without breaking the Internet. Some of the topics we covered include automation, state management, systems architecture, distributed messaging, and peer review. Check it out! by Sean O'Connor ## Z Proxy: Bitly's Independent Link-Redirect Fallback DevFeed: [Z Proxy: Bitly's Independent Link-Redirect Fallback]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-10-02T17:47:22Z Content type: article Language: en Sources: [Bitly]() Topics: [Availability](), [Amazon S3](), [Go Language](), [Amazon EC2](), [nsq]() Tags: [availability](), [ec2](), [go](), [nsq](), [s3]() ### AI overview Bitly describes Z Proxy, a Go application designed to keep short-link redirects available independently of its main infrastructure. It stores link mappings in S3, uses NSQ to populate them, runs on EC2, and caches S3 lookups locally with memcached. ### Source excerpt Here is a tale of how we leverage redundant datacenters, redundant code, and multi-tiered fallbacks in the quest for uptime. But Why? High availability is important for any site operating at scale, but at bitly it is particularly important; people expect bitly links to work, no matter what. We have enterprise customers who rely on them for key metrics, users who share them on social networks, and websites with custom short domains that trust us to serve requests with their name on them. A bitly link not working in any of these scenarios would make our users look bad, so it is something we take very seriously. No matter how redundant, distributed, and fault tolerant your main infrastructure is, things can always go wrong. Recently Google Apps and Search, probably the most distributed infrastructure in existence, went down. There are unknowns everywhere, and ultimately you have to plan for any part of your infrastructure breaking for unknown reasons. Under failure, a distributed system should degrade gracefully, not suddenly. This is why we created Z Proxy. So What is Z Proxy? Z Proxy is an application that serves decodes (this is what we call redirecting from a short bitly link to its long URL, and what happens every time you click on a bitly link) without relying on any other part of the bitly infrastructure. This means that it does not use our primary database of urls, or any of our other servers, to do lookups. So how does it work? How it Works Z Proxy is essentially a self contained wrapper around S3, written in Go. When all of bitly is running properly, every time a link is shortened, a message is put on NSQ, which a queuereader later grabs. A queuereader then writes the short and long urls into S3 so that Z Proxy can perform lookups against S3 by short url, get the long url, and serve a 301 or 302 redirect. To the browser, nothing different happened. There are multiple host running Z Proxy in EC2. This location provides proximity to S3, high availability, and m ## Building NSQ Client Libraries DevFeed: [Building NSQ Client Libraries]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-05-09T18:58:23Z Content type: tutorial Language: en Sources: [Bitly]() Topics: [Messaging](), [client](), [configuration](), [servers](), [HTTP]() Tags: [asynchronous](), [client-library](), [configuration](), [developers](), [distributed](), [guide](), [infrastructure](), [message-queue](), [messaging](), [nsq](), [tcp](), [thundering-herd]() ### AI overview A guide to the responsibilities and design expectations of NSQ client libraries, focusing on consumers. It covers configuration, discovery, TCP connections, message handling, pipelining, asynchronous processing, and techniques for maintaining cluster robustness and performance. ### Source excerpt Brace yourself, this is a long one. The following guide was originally intended for client library developers to describe in detail all the important features and functionality we expected in an NSQ client library. While writing it we began to realize that it had value beyond client library developers. It incorporates a comprehensive analysis of most of the capabilities of NSQ (both client and server) and is therefore interesting and useful for end-users as well (or anyone using or interested in infrastructure messaging platforms). If you need some background on NSQ please see our original blog post or its follow up, spray some NSQ on it. Intro NSQ's design pushes a lot of responsibility onto client libraries in order to maintain overall cluster robustness and performance. This guide attempts to outline the various responsibilities well-behaved client libraries need to fulfill. Because publishing to nsqd is trivial (just an HTTP POST to the /put endpoint), this document focuses on consumers. By setting these expectations we hope to provide a foundation for achieving consistency across languages for NSQ users. Overview Configuration Discovery (optional) Connection Handling Feature Negotiation Data Flow / Heartbeats Message Handling RDY State Backoff Configuration At a high level, our philosophy with respect to configuration is to design the system to have the flexibility to support different workloads, use sane defaults that run well "out of the box", and minimize the number of dials. A client subscribes to a topic on a channel over a TCP connection to nsqd instance(s). You can only subscribe to one topic per connection so multiple topic consumption needs to be structured accordingly. Using nsqlookupd for discovery is optional so client libraries should support a configuration where a client connects directly to one or more nsqd instances or where it is configured to poll one or more nsqlookupd instances. When a client is configured to poll nsqlookupd the polling int ## Speeding things up with Redshift DevFeed: [Speeding things up with Redshift]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-04-25T14:30:19Z Content type: opinion Language: en Sources: [Bitly]() Topics: [Amazon Redshift](), [Data analysis](), [SQL](), [Hadoop](), [Python](), [amazon]() Tags: [amazon](), [analysis](), [data-analysis](), [gotchas](), [hadoop](), [python](), [redshift](), [speed](), [sql](), [workflow]() ### AI overview The article describes bitly's experience using Amazon Redshift to speed up ad hoc analysis of large volumes of click data. It contrasts Redshift SQL queries with a slower Hadoop and Python MapReduce workflow, reporting answers within minutes or seconds and an overall positive experience, while noting some gotchas. ### Source excerpt Recently we've started to experiment with using Redshift, Amazon's new data warehousing service. More specifically, we're using it to speed up and expand our ad hoc data analysis. The Challenge bitly sees billions of clicks and shortens each month. Often we have various questions about the data generated from this activity. Sometimes these questions are driven by business needs (how much traffic do we see from a potential enterprise customer), sometimes they are more technically driven (how much traffic will a new sub-system need to deal with), and sometimes we like to just have fun (what are the top trashy celeb stories this week). Unfortunately, when working with that volume of data it can be pretty difficult to do much of anything quickly. Pre-Redshift, all of these questions were answered by writing map-reduce jobs to be run on our Hadoop cluster or on Amazon's EMR. Whenever we wanted to answer a question with our data, the process would look something like this: Write map-reduce job in Python Run it on some local test data Fix bugs. Run it on the Hadoop cluster Wait 20-30 minutes for results Get an error back from Hadoop Dig through the logs to find the error. GOTO 3 This is clearly not ideal when all you want to do is get a simple count. For a lot of the work we do Hadoop + Python make for an awesome combination, but for these ad hoc aggregation queries they're very blunt instruments. In both cases, they are general purpose tools that are super flexible, but slow and difficult to use for this specific use case. Redshift, on the other hand, is specifically built and optimized for doing aggregation queries over large sets of data. When we want to answer a question with Redshift, we just write a SQL query and get an answer within a few minutes--if not seconds. Overall, our experience with Redshift has been a positive one but we have run into some gotchas that we'll get into below. The Good News User Experience From a user perspective, we're really happy with Redsh ## Securing Internal Applications DevFeed: [Securing Internal Applications]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-04-09T17:19:51Z Content type: article Language: en Sources: [Bitly]() Topics: [Authentication](), [Authorization](), [servers](), [nginx](), [OAuth 2.0](), [Google](), [HTTP]() Tags: [authentication](), [authorization](), [google](), [http](), [nginx](), [oauth2]() ### AI overview The article describes Google Auth Proxy, an HTTP reverse proxy developed by bitly to secure internal applications with Google Account authentication through Google's OAuth2 API. It addresses the limitations of HTTP Basic Auth and application-specific authorization lists by allowing access for individual Google Accounts or an entire Google Apps domain. ### Source excerpt A common infrastructure problem is managing access to internal applications. Some patterns for solving that problem include: fronting internal apps with HTTP Basic or HTTP Digest authentication through Apache or nginx, running the apps on secret ports, and bundling an authentication system as part of each application. Many off-the-shelf applications include no built-in authentication, and count on upstream proxies/systems for access control which limits options for securing them. Google Auth Proxy is a tool we have developed to give us a better ability to secure internal applications by using Google Account authentication (like many other startups, we rely on Google Apps). At bitly, we like fronting HTTP applications with Nginx and use that approach for nearly all of our stack, from Tornado apps handling our core APIs to common internal tools (e.g. Nagios, Munin, Graphite) to homegrown tools like NSQ and our deploy system. For many of those systems, we took advantage of this by configuring HTTP Basic authentication to restrict access. Using Basic Auth however led to situations where the authentication information would be stored in configuration files and scripts, leaking it throughout the repository. Some of our homegrown tools initially relied on bitly's OAuth 2 service for authentication, which meant that sign in was managed through bitly accounts by the OAuth flow. This left each application to handle authorization by maintaining a list of bitly accounts that actually had permission to access the app. This was an improvement over HTTP Basic Auth as each application did not need to accept or store passwords, but it was less than ideal because each app still needed to store and manage its own list of authorized users. Over time, this led to a large number of disparate systems with separate authorization lists. This approach was only feasible for internal tools and thus couldn't apply to other open source applications we were using. We developed Google Auth Proxy a ## Forget Table: Tracking Changing Categorical Distributions in Data Streams DevFeed: [Forget Table: Tracking Changing Categorical Distributions in Data Streams]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-01-23T16:46:00Z Content type: article Language: en Sources: [Bitly]() Topics: [Statistics](), [data](), [Streams](), [datasets]() Tags: [analytics](), [data](), [dataset](), [statistics](), [streams]() ### AI overview Forget Table addresses the challenge of tracking categorical distributions in continuously changing data streams. It uses recent data to better represent shifting patterns and support anomaly detection, such as identifying unexpectedly high traffic from a location. ### Source excerpt Forget Table is a solution to the problem of storing the recent dynamics of categorical distributions that change over time (ie: non-stationary distributions). What does this mean? Why is this useful? What makes this the most important thing since sliced bread?! Read on! Storing distributions is crucial for doing any sort of statistics on a dataset. Normally, this problem is as easy as keeping counters of how many times we've seen certain quantities, but when dealing with data streams that constantly deliver new data, these simple counters no longer do the job. They fail because data we saw weeks ago has the same weight as data we have just seen, even though the fundamental distribution the data is describing may be changing. In addition it provides an engineering challenge since the counters would strictly grow and very soon use all the resources available to it. This is why we created Forget Table. Background A categorical distribution describes the probability of seeing an event occur out of a set of possible events. So an example of this at bitly is that every click coming through our system comes from one of a fixed number of country codes (there are about 260). We would like to maintain a categorical distribution, that assigns a probability to each country, that describes how likely any given click comes from each country. With bitly's data, this is going to give a high weight to the US and lower weights to Japan and Brazil, for example. It's very useful for bitly to store distributions like this, as it gives us a good idea as to what's considered 'normal'. Most of the time we produce analytics that show, for example, how many clicks came from a given country, or referrer. While this gives our clients a sense of where there traffic is coming from, it doesn't directly express how surprising their traffic is. This can be remedied by maintaining a distribution over countries for all of bitly, so that we can identify anomalous traffic, ie: when a particular link i ## Improving Frontend Code Quality and Workflow DevFeed: [Improving Frontend Code Quality and Workflow]() Original publisher: [Read original article]() Author: Wordbitly Published: 2012-12-11T18:45:27Z Content type: tutorial Language: en Sources: [Bitly]() Topics: [Front end](), [Code quality](), [JavaScript](), [Refactoring](), [modules](), [engineering-culture]() Tags: [code-quality](), [code-review](), [coding](), [coding-style](), [consistency](), [frontend](), [maintainability](), [refactoring]() ### AI overview A Bitly frontend engineer describes improving code quality and workflow by introducing shared JavaScript style conventions, evaluating tools and libraries, and adopting CoffeeScript for new frontend code. The article explains how consistent patterns can improve code readability, onboarding, and maintainability. ### Source excerpt When I started at Bitly as a Frontend Engineer, we were about to launch the new bitly. It was exciting to be so close to a product launch and we were cranking out lots of code each day. It took me a few days to get my bearings in the sprawling codebase and I saw lots of opportunity for refactoring, cleanup, and style normalization as well as some architectural concepts we weren't leveraging. Product launch mode kept us from doing house keeping for the next few months after we released the new product as we collected feedback and iterated on several designs/functionalities. Once we had a chance to take a step back and regroup, I saw that we could be more productive if we invested in a common JavaScript style (both syntax and module level patterns) and re-evaluated which libraries we were leveraging to build the site. When we decided the next feature work would be on save/share modal dialogs, I used this as an opportunity to evaluate my new picks for tools and libraries. The result was a success and allowed us to adopt the new tools and libraries for all new features (see Easily Save and Share the Links You Love). Keep reading to see what tools we ended up using to make us more productive! Coding style guidelines If you've seen frontend JavaScript codebases older than six months with two or more developers working on them, you've seen how hard it is to enforce style. The goal of a style guide (and code conventions in general) is to reduce the amount of friction when switching between different modules in a codebase and increase the consistency and readability of code. If every module follows the same patterns, you can easily get your bearings in code written by your coworkers. Every team should have a common style that is enforced in code review and ideally written in a document that can be referred to for new hires. For JavaScript at Bitly, we mostly use the Google Style Guide without the JSDoc and with a variation that non-method variables are lowercase_with_undersc ## Classifying Human Traffic with Random Forest Decision Trees DevFeed: [Classifying Human Traffic with Random Forest Decision Trees]() Original publisher: [Read original article]() Author: Wordbitly Published: 2012-11-16T15:38:18Z Content type: article Language: en Sources: [Bitly]() Topics: [SciKits](), [data](), [Python]() Tags: [data](), [python](), [realtime](), [script](), [stream]() ### AI overview A bitly data scientist presented a fast random forest decision-tree approach implemented in Python with scikits-learn to classify organic versus inorganic data in a realtime stream. ### Source excerpt At bitly, we study human behavior on the social web, and we often need to figure out when data is generated by a deliberate human action (organic data) or by an action taken by a script or without a human's knowledge (inorganic data). bitly data scientist Brian Eoff recently gave a talk at PyData NYC 2012 on a fast random forest decision tree approach, implemented in Python with scikits-learn, to identifying organic vs inorganic data in a realtime stream. SciKit Random Forest - Brian Eoff from Continuum Analytics on Vimeo. by hilary