# Nick Craver Software Imagineering This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Binding Redirects DevFeed: [Binding Redirects]() Original publisher: [Read original article]() Published: 2020-02-11T00:00:00Z Content type: tutorial Language: en Sources: [Nick Craver]() Topics: [.NET](), [net core](), [.NET Framework](), [migration](), [NuGet](), [Visual Studio]() Tags: [blog](), [compatibility](), [dependencies](), [framework](), [migration](), [net](), [net-core](), [net-framework](), [nuget](), [visual-studio]() ### AI overview This tutorial explains how .NET Framework assembly version conflicts produce load errors and build warnings. It recommends migrating to .NET Core as the long-term approach and describes binding redirects as the main fix for applications that remain on .NET Framework. ### Source excerpt This isn't part of the series on Stack Overflow's architecture, but is a topic that has bitten us many times. Hopefully, some of this information helps you sort out issues you hit. You're probably here because of an error like this: Could not load file or assembly 'System.<...>, Version=4.x.x.x, Culture=neutral, PublicKeyToken=<...>' or one of its dependencies. The system cannot find the file specified. And you likely saw a build warning like this: warning MSB3277: Found conflicts between different versions of "System.<...>" that could not be resolved. Whelp, you're not alone. We're thinking about starting a survivors group. The most common troublemakers here are: System.Memory (NuGet link) System.Net.Http (NuGet link) System.Numerics.Vectors (NuGet link) System.Runtime.CompilerServices.Unsafe (NuGet link) System.ValueTuple (NuGet link) If you just want out of this fresh version of DLL Hell you've found yourself in... The best fix The best fix is "go to .NET Core". Since .NET Framework (e.g. 4.5, 4.8, etc.) has a heavy backward compatibility burden, meaning that the assembly loader itself is basically made of unstable plutonium with a hair trigger coated in flesh eating bacteria behind a gate made of unobtanium above a moat of napalm filled with those jellyfish that kill you...that won't ever really be fixed. However, .NET Core's simplified assembly loading means it just works. I'm not saying a migration to .NET Core is trivial, that depends on your situation, but it is generally the best long-term play. We're almost done porting Stack Overflow to .NET Core, and this kind of pain is one of the things we're very much looking forward to not fighting ever again. The fix if you're using .NET Framework The fix for .NET Framework is "binding redirects". When you are trying to load an assembly (and if it's strongly named, like the ones in the framework and many libs are), it'll try to load the specific version you specified. Unless it's told to load another (likely newer) version. T ## Stack Overflow: How We Do App Caching - 2019 Edition DevFeed: [Stack Overflow: How We Do App Caching - 2019 Edition]() Original publisher: [Read original article]() Published: 2019-08-06T00:00:00Z Content type: tutorial Language: en Sources: [Nick Craver]() Topics: [Caching](), [Stack Overflow](), [App](), [Latency](), [API](), [cpu](), [Database](), [Web]() Tags: [api](), [app](), [architecture](), [blog](), [cache](), [caching](), [computer](), [cost](), [cycles](), [database](), [latency](), [memory](), [performance](), [processor](), [server](), [web]() ### AI overview This article explains how caching avoids repeatedly recalculating or fetching data, improving performance and reducing costs. It examines cache layers in computer processors and memory, their capacity and latency trade-offs, and the role of storage in Stack Overflow's production architecture. ### Source excerpt This is #5 in a very long series of posts on Stack Overflow's architecture. Previous post (#4): Stack Overflow: How We Do Monitoring - 2018 Edition So...caching. What is it? It's a way to get a quick payoff by not re-calculating or fetching things over and over, resulting in performance and cost wins. That's even where the name comes from, it's a short form of the "ca-ching!" cash register sound from the dark ages of 2014 when physical currency was still a thing, before Apple Pay. I'm a dad now, deal with it. Let's say we need to call an API or query a database server or just take a bajillion numbers (Google says that's an actual word, I checked) and add them up. Those are all relatively crazy expensive. So we cache the result - we keep it handy for re-use. Why Do We Cache? I think it's important here to discuss just how expensive some of the above things are. There are several layers of caching already in play in your modern computer. As a concrete example, we're going to use one of our web servers which currently houses a pair of Intel Xeon E5-2960 v3 CPUs and 2133MHz DIMMs. Cache access is a "how many cycles" feature of a processor, so by knowing that we always run at 3.06GHz (performance power mode), we can derive the latencies (Intel architecture reference here - these processors are in the Haswell generation): L1 (per core): 4 cycles or ~1.3ns latency - 12x 32KB+32KB L2 (per core): 12 cycles or ~3.92ns latency - 12x 256KB L3 (shared): 34 cycles or ~11.11ns latency - 30MB System memory: ~100ns latency - 8x 8GB Each cache layer is able to store more, but is farther away. It's a trade-off in processor design with balances in play. For example, more memory per core means (almost certainly) on average putting it farther away on the chip from the core and that has costs in latency, opportunity costs, and power consumption. How far an electric charge has to travel has substantial impact at this scale; remember that distance is multiplied by billions every second. And I ## Stack Overflow: How We Do Monitoring - 2018 Edition DevFeed: [Stack Overflow: How We Do Monitoring - 2018 Edition]() Original publisher: [Read original article]() Published: 2018-11-29T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [Monitoring](), [Stack Overflow](), [Automation](), [data](), [telemetry](), [Architecture & Design](), [health checks]() Tags: [architecture](), [automation](), [blog](), [data](), [health-checks](), [logs](), [metrics](), [monitoring](), [telemetry]() ### AI overview This article explains how Stack Overflow approaches application monitoring. It discusses monitoring concepts, automation and alert thresholds, trade-offs in operational decisions, and data types including logs, metrics, and health checks. ### Source excerpt This is #4 in a very long series of posts on Stack Overflow's architecture. Previous post (#3): Stack Overflow: How We Do Deployment - 2016 Edition What is monitoring? As far as I can tell, it means different things to different people. But we more or less agree on the concept. I think. Maybe. Let's find out! When someone says monitoring, I think of: ...but evidently some people think of other things. Those people are obviously wrong, but let's continue. When I'm not a walking zombie after reading a 10,000 word blog post some idiot wrote, I see monitoring as the process of keeping an eye on your stuff, like a security guard sitting at a desk full of cameras somewhere. Sometimes they fall asleep-that's monitoring going down. Sometimes they're distracted with a doughnut delivery-that's an upgrade outage. Sometimes the camera is on a loop-I don't know where I was going with that one, but someone's probably robbing you. And then you have the fire alarm. You don't need a human to trigger that. The same applies when a door gets opened, maybe that's wired to a siren. Or maybe it's not. Or maybe the siren broke in 1984. I know what you're thinking: Nick, what the hell? My point is only that monitoring any application isn't that much different from monitoring anything else. Some things you can automate. Some things you can't. Some things have thresholds for which alarms are valid. Sometimes you'll get those thresholds wrong (especially on holidays). And sometimes, when setting up further automation isn't quite worth it, you just make using human eyes easier. What I'll discuss here is what we do. It's not the same for everyone. What's important and "worth it" will be different for almost everyone. As with everything else in life, it's full of trade-off decisions. Below are the ones we've made so far. They're not perfect. They are evolving. And when new data or priorities arise, we will change earlier decisions when it warrants doing so. That's how brains are supposed to work. A ## HTTPS on Stack Overflow: The End of a Long Road DevFeed: [HTTPS on Stack Overflow: The End of a Long Road]() Original publisher: [Read original article]() Published: 2017-05-22T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [Stack Overflow](), [HTTP](), [Network](), [WebSocket](), [Latency](), [Open Source]() Tags: [apis](), [blog](), [code](), [http](), [infrastructure](), [latency](), [network](), [networks](), [open-source](), [performance](), [proxy]() ### AI overview Stack Overflow deployed HTTPS by default across its Q&A websites in 2017 after several years of work. The article explains the infrastructure, application, performance, WebSocket, proxy, domain, and embedded-content challenges involved in the rollout. ### Source excerpt Today, we deployed HTTPS by default on Stack Overflow. All traffic is now redirected to https:// and Google links will change over the next few weeks. The activation of this is quite literally flipping a switch (feature flag), but getting to that point has taken years of work. As of now, HTTPS is the default on all Q&A websites. We've been rolling it out across the Stack Exchange network for the past 2 months. Stack Overflow is the last site, and by far the largest. This is a huge milestone for us, but by no means the end. There's still more work to do, which we'll get to. But the end is finally in sight, hooray! Fair warning: This is the story of a long journey. Very long. As indicated by your scroll bar being very tiny right now. While Stack Exchange/Overflow is not unique in the problems we faced along the way, the combination of problems is fairly rare. I hope you find some details of our trials, tribulations, mistakes, victories, and even some open source projects that resulted along the way to be helpful. It's hard to structure such an intricate dependency chain into a chronological post, so I'll break this up by topic: infrastructure, application code, mistakes, etc. I think it's first helpful to preface with a list of problems that makes our situation somewhat unique: We have hundreds of domains (many sites and other services) Many second-level domains (stackoverflow.com, stackexchange.com, askubuntu.com, etc.) Many 4th level domains (e.g. meta.gaming.stackexchange.com) We allow user submitted & embedded content (e.g. images and YouTube videos in posts) We serve from a single data center (latency to a single origin) We have ads (and ad networks) We use websockets, north of 500,000 active at any given (connection counts) We get DDoSed (proxy) We have many sites & apps communicating via HTTP APIs (proxy issues) We're obsessed with performance (maybe a little too much) Since this post is a bit crazy, links for your convenience: The Beginning Quick Specs Infrast ## Stack Overflow: How We Do Deployment - 2016 Edition DevFeed: [Stack Overflow: How We Do Deployment - 2016 Edition]() Original publisher: [Read original article]() Published: 2016-05-03T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [Deployment](), [Stack Overflow](), [Development](), [Git](), [Databases](), [ci](), [C#](), [Localization (l10n)]() Tags: [article](), [blog](), [c-sharp](), [ci](), [code](), [database](), [deployment](), [development](), [git]() ### AI overview This article explains how Stack Overflow deployed code in 2016, covering the repository, human workflow, branches, Git, build steps, database migrations, localization, deployment tiers, and techniques for building without breaking production. It reports roughly 25 daily deployments to development, 5-10 daily production deployments, and a full deployment time of under nine minutes. ### Source excerpt This is #3 in a very long series of posts on Stack Overflow's architecture. Previous post (#2): Stack Overflow: The Hardware - 2016 Edition We've talked about Stack Overflow's architecture and the hardware behind it. The next most requested topic was Deployment. How do we get code a developer (or some random stranger) writes into production? Let's break it down. Keep in mind that we're talking about deploying Stack Overflow for the example, but most of our projects follow almost an identical pattern to deploy a website or a service. I'm going ahead and inserting a set of section links here because this post got a bit long with all of the bits that need an explanation: Source & Context The Human Steps Branches Git On-Premises The Build System What's In The Build? Steps 1 & 2: Migrations Step 3: Finding Moonspeak (Translation) Step 4: Translation Dump (JavaScript Edition) Step 5: MSBuild Step 6: Translation Dump (C# Edition) Step 7: Importing English Strings Step 8: Deploy Website Step 9: New Strings Hook Tiers Database Migrations Localization/Translations (Moonspeak) Building Without Breaking Extra resources because I love you all GitHub Gist (scripts) GitHub Gist (logs) Source This is our starting point for this article. We have the Stack Overflow repository on a developer's machine. For the sake of discussing the process, let's say they added a column to a database table and the corresponding property to the C# object -- that way we can dig into how database migrations work along the way. A Little Context We deploy roughly 25 times per day to development (our CI build) just for Stack Overflow Q&A. Other projects also push many times. We deploy to production about 5-10 times on a typical day. A deploy from first push to full deploy is under 9 minutes (2:15 for dev, 2:40 for meta, and 3:20 for all sites). We have roughly 15 people pushing to the repository used in this post. The repo contains the code for these applications: Stack Overflow (every single Q&A site), sta ## Stack Overflow: The Hardware - 2016 Edition DevFeed: [Stack Overflow: The Hardware - 2016 Edition]() Original publisher: [Read original article]() Published: 2016-03-29T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [Hardware](), [Stack Overflow](), [Server]() Tags: [architecture](), [blog](), [capacity](), [hardware](), [infrastructure](), [performance](), [server]() ### AI overview This article describes the hardware foundation of Stack Overflow's 2016 architecture. It presents an inventory of the site's infrastructure for reference and comparison, and explains how servers are specified according to their intended workloads and their role in the broader architecture. ### Source excerpt This is #2 in a very long series of posts on Stack Overflow's architecture. Previous post (#1): Stack Overflow: The Architecture - 2016 Edition Next post (#3): Stack Overflow: How We Do Deployment - 2016 Edition Who loves hardware? Well, I do and this is my blog so I win. If you don't love hardware then I'd go ahead and close the browser. Still here? Awesome. Or your browser is crazy slow, in which case you should think about some new hardware. I've repeated many, many times: performance is a feature. Since your code is only as fast as the hardware it runs on, the hardware definitely matters. Just like any other platform, Stack Overflow's architecture comes in layers. Hardware is the foundation layer for us, and having it in-house affords us many luxuries not available in other scenarios...like running on someone else's servers. It also comes with direct and indirect costs. But that's not the point of this post, that comparison will come later. For now, I want to provide a detailed inventory of our infrastructure for reference and comparison purposes. And pictures of servers. Sometimes naked servers. This web page could have loaded much faster, but I couldn't help myself. In many posts through this series I will give a lot of numbers and specs. When I say "our SQL server utilization is almost always at 5-10% CPU," well, that's great. But, 5-10% of what? That's when we need a point of reference. This hardware list is meant to both answer those questions and serve as a source for comparison when looking at other platforms and what utilization may look like there, how much capacity to compare to, etc. How We Do Hardware Disclaimer: I don't do this alone. George Beech (@GABeech) is my main partner in crime when speccing hardware here at Stack. We carefully spec out each server for its intended purpose. What we don't do is order in bulk and assign tasks later. We're not alone in this process though; you have to know what's going to run on the hardware to spec it optimally. ## Stack Overflow: The Architecture - 2016 Edition DevFeed: [Stack Overflow: The Architecture - 2016 Edition]() Original publisher: [Read original article]() Published: 2016-02-17T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [Stack Overflow](), [ASP.NET](), [Redis](), [SQL](), [Hardware](), [servers](), [elasticsearch](), [Server]() Tags: [architecture](), [asp-net](), [blog](), [cloudflare](), [elasticsearch](), [hardware](), [http](), [load-balancer](), [performance](), [redis](), [servers](), [sql]() ### AI overview This article describes Stack Overflow's 2016 architecture using daily traffic and performance statistics from February 9, 2016, compared with November 2013. It covers HTTP requests, page loads, SQL queries, Redis hits, Elasticsearch searches, Tag Engine requests, and processing times. The author attributes reduced ASP.NET processing time to a 2015 hardware upgrade and application performance tuning, then lists the main server and load-balancer hardware. ### Source excerpt This is #1 in a very long series of posts on Stack Overflow's architecture. Welcome. Previous post (#0): Stack Overflow: A Technical Deconstruction Next post (#2): Stack Overflow: The Hardware - 2016 Edition To get an idea of what all of this stuff "does," let me start off with an update on the average day at Stack Overflow. So you can compare to the previous numbers from November 2013, here's a day of statistics from February 9th, 2016 with differences since November 12th, 2013: 209,420,973 (+61,336,090) HTTP requests to our load balancer 66,294,789 (+30,199,477) of those were page loads 1,240,266,346,053 (+406,273,363,426) bytes (1.24 TB) of HTTP traffic sent 569,449,470,023 (+282,874,825,991) bytes (569 GB) total received 3,084,303,599,266 (+1,958,311,041,954) bytes (3.08 TB) total sent 504,816,843 (+170,244,740) SQL Queries (from HTTP requests alone) 5,831,683,114 (+5,418,818,063) Redis hits 17,158,874 (not tracked in 2013) Elastic searches 3,661,134 (+57,716) Tag Engine requests 607,073,066 (+48,848,481) ms (168 hours) spent running SQL queries 10,396,073 (-88,950,843) ms (2.8 hours) spent on Redis hits 147,018,571 (+14,634,512) ms (40.8 hours) spent on Tag Engine requests 1,609,944,301 (-1,118,232,744) ms (447 hours) spent processing in ASP.Net 22.71 (-5.29) ms average (19.12 ms in ASP.Net) for 49,180,275 question page renders 11.80 (-53.2) ms average (8.81 ms in ASP.Net) for 6,370,076 home page renders You may be wondering about the drastic ASP.Net reduction in processing time compared to 2013 (which was 757 hours) despite 61 million more requests a day. That's due to both a hardware upgrade in early 2015 as well as a lot of performance tuning inside the applications themselves. Please don't forget: performance is still a feature. If you're curious about more hardware specifics than I'm about to provide--fear not. The next post will be an appendix with detailed hardware specs for all of the servers that run the sites (I'll update this with a link when it's liv ## Stack Overflow: A Technical Deconstruction DevFeed: [Stack Overflow: A Technical Deconstruction]() Original publisher: [Read original article]() Published: 2016-02-03T00:00:00Z Content type: opinion Language: en Sources: [Nick Craver]() Topics: [Stack Overflow]() Tags: [blog](), [developers](), [technical]() ### AI overview Nick Craver introduces a planned series about Stack Overflow's technical practices and invites readers to suggest and prioritize future topics. He presents the series as an experiment in openness intended to share lessons and improve how the team works. ### Source excerpt As new posts in the series appear, I'll add them here to serve as a master list: #1: Stack Overflow: The Architecture - 2016 Edition #2: Stack Overflow: The Hardware - 2016 Edition #3: Stack Overflow: How We Do Deployment - 2016 Edition #4: Stack Overflow: How We Do Monitoring - 2018 Edition #5: Stack Overflow: How We Do App Caching - 2019 Edition One of the reasons I love working at Stack Overflow is we're allowed encouraged to talk about almost anything out in the open. Except for things companies always keep private like financials and the nuclear launch codes, everything else is fair game. That's an awesome thing that we haven't taken advantage of on the technical side lately. I think it's time for an experiment in extreme openness. By sharing what we do (and I mean all of us), we better our world. Everyone that works at Stack shares at least one passion: improving life for all developers. Sharing how we do things is one of the best and biggest ways we can do that. It helps you. It helps me. It helps all of us. When I tell you how we do , a few things happen: You might learn something cool you didn't know about. We might learn we're doing it wrong. We'll both find a better way, together...and we share that too. It helps eliminate the perception that "the big boys" always do it right. No, we screw up too. There's nothing to lose here and there's no reason to keep things to yourself unless you're afraid of being wrong. Good news: that's not a problem. We get it wrong all the time anyway, so I'm not really worried about that one. Failure is always an option. The best any of us can do is live, learn, move on, and do it better next time. Here's where I need your help I need you to tell me: what do you want to hear about? My intention is to get to a great many things, but it will take some time. What are people most interested in? How do I decide which topic to blog about next? The answer: I don't know and I can't decide. That's where you come in. Please, tel ## Why you should wait on upgrading to .Net 4.6 DevFeed: [Why you should wait on upgrading to .Net 4.6]() Original publisher: [Read original article]() Published: 2015-07-27T00:00:00Z Content type: article Language: en Sources: [Nick Craver]() Topics: [.NET Framework](), [bug](), [JIT](), [Compiler](), [Microsoft](), [Stack Overflow]() Tags: [blog](), [bug](), [compiler](), [jit](), [microsoft](), [net-framework]() ### AI overview The article warns against upgrading to .NET Framework 4.6 because a bug in the 64-bit RyuJIT implementation can cause methods to receive parameter values different from those passed. Microsoft later released a patch through Security Bulletin MS15-092 and KB3086251. ### Source excerpt Update (August 11th): A patch for this bug has been released by Microsoft. Here's their update to the advisory: We released an updated version of RyuJIT today, which resolves this advisory. The update was released as Microsoft Security Bulletin MS15-092 and is available on Windows Update or via direct download as KB3086251. The update resolves: CoreCLR #1296, CoreCLR #1299, and VisualFSharp #536. Major thanks to the developers who reported these issues. Thanks to everyone for their patience. Original Post What follows is the work of several people: Marc Gravell and I have taken lead on this at Stack Overflow and we continue to coordinate with Microsoft on a resolution. They have fixed the bug internally, but not for users. Given the severity, we can't in good conscience let such a subtle yet high-impact bug linger silently. We are not upgrading Stack Overflow to .Net 4.6, and you shouldn't upgrade yet either. You can find the issue we opened on GitHub (for public awareness) here. A fix has been released, see Update 5 below. Update #1 (July 27th): A pull request has been posted by Matt Michell (Microsoft). Update #2 (July 28th): There are several smaller repros now (including a small console app). Microsoft has confirmed they are working on an expedited hotfix release but we don't have details yet. Update #3 (July 28th): Microsoft's Rich Lander has posted an update: RyuJIT Bug Advisory in the .NET Framework 4.6. Update #4 (July 29th): There's another subtle bug found by Andrey Akinshin and the F# Engine Exception is confirmed to be a separate issue. I still recommend disabling RyuJIT in production given the increasing bug count. Update #5 (August 11th): A patch for this bug has been released by Microsoft, see above. This critical bug is specific to .Net 4.6 and RyuJIT (64-bit). I'll make this big and bold so we get to the point quickly: The methods you call can get different parameter values than you passed in. The JIT (Just-in-Time compiler) in .Net (and many platfo ## Life at Stack Overflow: Operations, Scale and Team Coordination DevFeed: [Life at Stack Overflow: Operations, Scale and Team Coordination]() Original publisher: [Read original article]() Published: 2015-07-24T00:00:00Z Content type: opinion Language: en Sources: [Nick Craver]() Topics: [Stack Overflow](), [HTTP](), [Redis](), [SQL](), [elasticsearch]() Tags: [developers](), [http](), [redis](), [sql](), [stack-overflow](), [talks]() ### AI overview A Stack Overflow talk presents operational scale metrics and discusses capacity, rendering, databases, Redis, Elasticsearch, and team coordination with DBAs. ### Source excerpt My Developers Are Smarter Than Your DBAs by @Nick_Craver What would you say...you do here? Last month at Stack Overflow: 1,468,389,303 Page Views 5,183,954,727 HTTP Hits 71,562,833,811,315 Bytes Sent 3,202,505,376 CDN Hits 54,400,000,000,000 CDN Bytes 19,532,899,854 SQL Queries 81,505,688,410 Redis Ops 18.2ms Average Render Time ...at roughly 5-10% capacity How do we do that? Go that way, really fast. If something gets in your way...turn. We're not ready to turn yet. Being part of a team Disadvantages: You no longer know all the things You don't have time to learn all the things Merge conflicts Advantages: You have help Lots of help More victims for the wheel of blame Pipelines Most interactions with a DBA require 2 people This is true of anyone you're asking for a service Perspective & Scope We know the things we know Except knowing what we know Insight and decisions are based on our world Specifically, the world as we see it Fog of war What does this allow us to do? Tag Engine Elasticsearch Moving /users into redis Anything we want Q&A? We love Q&A.