# Brendan Gregg's Blog Brendan Gregg's Blog This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why I joined OpenAI DevFeed: [Why I joined OpenAI]() Original publisher: [Read original article]() Published: 2026-02-06T13:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [OpenAI](), [Artificial Intelligence](), [datacenter](), [ChatGPT]() Tags: [ai](), [ai-adoption](), [blog](), [chatgpt](), [datacenter](), [openai](), [performance-engineering]() ### AI overview The author explains joining OpenAI to work on datacenter and ChatGPT performance, motivated by the rapidly growing cost and scale of AI infrastructure. The article also describes conversations that changed the author's view of how widely people use ChatGPT in everyday life. ### Source excerpt The staggering and fast-growing cost of AI datacenters is a call for performance engineering like no other in history; it's not just about saving costs - it's about saving the planet. I have joined OpenAI to work on this challenge directly, with an initial focus on ChatGPT performance. The scale is extreme and the growth is mind-boggling. As a leader in datacenter performance, I've realized that performance engineering as we know it may not be enough - I'm thinking of new engineering methods so that we can find bigger optimizations than we have before, and find them faster. It's the opportunity of a lifetime and, unlike in mature environments of scale, it feels as if there are no obstacles - no areas considered too difficult to change. Do anything, do it at scale, and do it today. Why OpenAI exactly? I had talked to industry experts and friends who recommended several companies, especially OpenAI. However, I was still a bit cynical about AI adoption. Like everyone, I was being bombarded with ads by various companies to use AI, but I wondered: was anyone actually using it? Everyday people with everyday uses? One day during a busy period of interviewing, I realized I needed a haircut (as it happened, it was the day before I was due to speak with Sam Altman). Mia the hairstylist got to work, and casually asked what I do for a living. "I'm an Intel fellow, I work on datacenter performance." Silence. Maybe she didn't know what datacenters were or who Intel was. I followed up: "I'm interviewing for a new job to work on AI datacenters." Mia lit up: "Oh, I use ChatGPT all the time!" While she was cutting my hair - which takes a while - she told me about her many uses of ChatGPT. (I, of course, was a captive audience.) She described uses I hadn't thought of, and I realized how ChatGPT was becoming an essential tool for everyone. Just one example: She was worried about a friend who was travelling in a far-away city, with little timezone overlap when they could chat, but she c ## Leaving Intel DevFeed: [Leaving Intel]() Original publisher: [Read original article]() Published: 2025-12-04T13:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [intel](), [Artificial Intelligence](), [GPU](), [eBPF](), [Cloud](), [cloud-computing](), [Linux](), [Open Source](), [Monitoring]() Tags: [ai](), [blog](), [cloud-computing](), [cpu](), [ebpf](), [gpu](), [intel](), [leaving](), [linux](), [monitoring](), [open-source](), [performance]() ### AI overview The author announces his departure from Intel after 3.5 years and reflects on his work there, including open-source AI flame graphs, GPU performance visualization, Linux stack walking, eBPF security monitoring, conference leadership, customer meetings, and a company-wide cloud strategy. ### Source excerpt InnovatiON 2022 AI Flame Graphs GPU Flame Scope Harshad Sane SREcon APAC Cloud strategy Last day I've resigned from Intel and accepted a new opportunity. If you are an Intel employee, you might have seen my fairly long email that summarized what I did in my 3.5 years. Much of this is public: AI flame graphs and released them as open source GPU subsecond-offset heatmap Worked with Linux distros to enable stack walking Was interviewed by the WSJ about eBPF for security monitoring Provided leadership on the eBPF Technical Steering Committee (BSC) Co-chaired USENIX SREcon APAC 2023 Gave 6 conference keynotes It's still early days for AI flame graphs. Right now when I browse CPU performance case studies on the Internet, I'll often see a CPU flame graph as part of the analysis. We're a long way from that kind of adoption for GPUs (and it doesn't help that our open source version is Intel only), but I think as GPU code becomes more complex, with more layers, the need for AI flame graphs will keep increasing. I also supported cloud computing, participating in 110 customer meetings, and created a company-wide strategy to win back the cloud with 33 specific recommendations, in collaboration with others across 6 organizations. It is some of my best work and features a visual map of interactions between all 19 relevant teams, described by Intel long-timers as the first time they have ever seen such a cross-company map. (This strategy, summarized in a slide deck, is internal only.) I always wish I did more, in any job, but I'm glad to have contributed this much especially given the context: I overlapped with Intel's toughest 3 years in history, and I had a hiring freeze for my first 15 months. My fond memories from Intel include meeting Linus at an Intel event who said "everyone is using fleme graphs these days" (Finnish accent), meeting Pat Gelsinger who knew about my work and introduced me to everyone at an exec all hands, surfing lessons at an Intel Australia and HP offsite ( ## On "AI Brendans" or "Virtual Brendans" DevFeed: [On "AI Brendans" or "Virtual Brendans"]() Original publisher: [Read original article]() Published: 2025-11-27T13:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [Artificial Intelligence](), [eBPF](), [systems]() Tags: [ai](), [ai-agents](), [blog](), [ebpf](), [opinions](), [performance]() ### AI overview Brendan Gregg discusses AI performance-engineering agents and "virtual Brendan" systems trained on his publications. He distinguishes agents that interpret flame graphs and eBPF metrics from systems intended to emulate his expertise, and argues that these tools have limited scope, may become outdated, and are difficult to price and evaluate. ### Source excerpt There are now multiple AI performance engineering agents that use or are trained on my work. Some are helper agents that interpret flame graphs or eBPF metrics, sometimes privately called AI Brendan; others have trained on my work to create a virtual Brendan that claims it can tune everything just like the real thing. These virtual Brendans sound like my brain has been uploaded to the cloud by someone who is now selling it (yikes!). I've been told it's even "easy" to do this thanks to all my publications available to train on: >90 talks, >250 blog posts, >600 open source tools, and >3000 book pages. Are people allowed to sell you, virtually? And am I the first individual engineer to be AI'd? (There is a 30-year-old precedent for this, which I'll get to later.) This is an emerging subject, with lots of different people, objectives, and money involved. Note that this is a personal post about my opinions, not an official post by my employer, so I won't be discussing internal details about any particular project. I'm also not here to recommend you buy any in particular. Summary There are two types: AI agents. I've sometimes heard them called an AI Brendan because it does Brendan-like things: systems performance recommendations and interpretation of flame graphs and eBPF metrics. There are already several of these and this idea in general should be useful. Virtual Brendan can refer to something not just built on my work, but trained on my publications to create a virtual me. These would only automate about 15% of what I do as a performance engineer, and will go out of date if I'm not training it to follow industry changes. Pricing is hard, in-house is easier. With a typical pricing model of $20 per instance per month, customers may just use such an agent on one instance and then copy-and-paste any tuning changes to their entire fleet. There's no practical way to keep tuning changes secret, either. These projects are easier as internal in-house tools. Some claim a lot but ## How to Give Constructive Feedback to Intel and Hardware Vendors DevFeed: [How to Give Constructive Feedback to Intel and Hardware Vendors]() Original publisher: [Read original article]() Published: 2025-11-21T13:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [intel](), [Hardware]() Tags: [blog](), [hardware](), [intel](), [opinion]() ### AI overview A firsthand opinion piece argues that customers can influence Intel and other hardware vendors through firm, documented, constructive technical feedback. It offers practical advice on preparation, intellectual-property terms, meeting minutes, and avoiding personal attacks. ### Source excerpt Intel's new CEO, Lip-Bu Tan, has made listening to customers a top priority, saying at Intel Vision earlier this year: "Please be brutally honest with us. This is what I expect of you this week, and I believe harsh feedback is most valuable." I'd been in regular meetings with Intel for several years before I joined, and I had been giving them technical direction on various projects, including at times some brutal feedback. When I finally interviewed for a role at Intel I was told something unexpected: that I had already accomplished so much within Intel that I qualified to be an Intel Fellow candidate. I then had to pass several extra interviews to actually become a Fellow (and was told I may only be the third person in Intel's history to be hired as a Fellow) but what stuck with me was that I had already accomplished so much at a company I'd never worked for. If you are in regular meetings with a hardware vendor as a customer (or potential customer) you can accomplish a lot by providing firm and tough feedback, particularly with Intel today. This is easier said than done, however. Now that I've seen it from the other side I realize I could have accomplished more, and you can too. I regret the meetings where I wasn't really able to have my feedback land as the staff weren't really getting it, so I eventually gave up. After the meeting I'd crack jokes with my colleagues about how the product would likely fail. (Come on, at least I tried to tell them!) Here's what I wish I had done in any hardware vendor meeting: Prep before meetings: study the agenda items and look up attendees on LinkedIn and note what they do, how many staff they say they manage, etc. Be aware of intellectual property risks: Don't accept meetings covered by some agreement that involves doing a transfer of intellectual property rights for your feedback (I wrote a post on this); ask your legal team for help. Make sure feedback is documented in the meeting minutes (e.g., a shared Google doc) and that ## Third Stage Engineering DevFeed: [Third Stage Engineering]() Original publisher: [Read original article]() Published: 2025-11-16T13:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [Hardware](), [benchmarking](), [Software](), [intel](), [Tensorflow]() Tags: [benchmark](), [blog](), [engineering](), [evaluation](), [hardware](), [intel](), [performance](), [software](), [tensorflow]() ### AI overview The article argues that production hardware performance depends on three stages: hardware, software, and tuning. It presents tuning, or "third-stage engineering," as involving people, training, tools, and capabilities, and recommends evaluating all hardware options with appropriate software and tuning for realistic workload performance. ### Source excerpt The real performance of any computer hardware in production is the result of the hardware, software, and tuning; the investment and sequence of these efforts can be pictured as a three-stage rocket: I recently presented this embarrassingly simple diagram to Intel's executive leadership, and at the time realized the value of sharing it publicly. The Internet is awash with comparisons about Intel (and other vendors') product performance based on hardware performance alone, but the performance of software and then tuning can make a huge difference for your particular workload. You need all three stages to reach the highest, and most competitive, performance. It's obvious why this is important for HW vendors to understand internally - they, like the Internet, can get overly focused on HW alone. But customers need to understand it as well. If a benchmark is comparing TensorFlow performance between HW vendors, was the Intel hardware tested using the Intel Extension for TensorFlow Software, and was it then tuned? The most accurate and realistic evaluation for HW involves selecting the best software and then tuning it, and doing this for all HW options. I spend a lot of time on the final stage, tuning - what I call third-stage engineering. It's composed of roughly four parts: People, training, tools, and capabilities. You need staff, you need them trained to understand performance methodologies and SW and HW internals, they need tools to analyze the system (both observational and experimental), and finally they need capabilities to tune (tunable parameters, settings, config, code changes, etc.). I see too many HW evaluations that are trying to understand customer performance but are considering HW alone, which is like only testing the first stage of a rocket. This doesn't help vendors or customers. I hope that's what my simple diagram makes obvious: We need all three stages to reach the highest altitude. ## When to Hire a Computer Performance Engineering Team (2025) part 1 of 2 DevFeed: [When to Hire a Computer Performance Engineering Team (2025) part 1 of 2]() Original publisher: [Read original article]() Published: 2025-08-03T14:00:00Z Content type: tutorial Language: en Sources: [Brendan Gregg's Blog]() Topics: [benchmarking](), [Latency](), [Scalability](), [Development]() Tags: [blog](), [infrastructure](), [intel](), [latency](), [netflix](), [observability](), [performance](), [performance-engineering](), [scalability]() ### AI overview This advice article explains what computer performance engineering teams do and when technology companies should consider forming one. It describes potential benefits including infrastructure cost savings, lower latency, improved scalability and reliability, and faster engineering, with a focus on non-vendor companies using significant back-end compute and AI. ### Source excerpt As a leader in computer performance I've been asked by companies about how (and why) to form a performance engineering team, and as this is broadly useful I'll share my advice here. Large tech companies in the US hire performance engineers (under that or other titles) to ensure that infrastructure costs and service latency don't grow too high, and that their service is reliable under peak load. A new performance team can likely find enough optimizations to halve infrastructure spend in their first couple of years, even for companies that have been using commercial performance or observability tools. Performance engineers do much more than those tools, working with development teams and vendors to build, test, debug, tune, and adopt new performance solutions, and to find deep optimizations that those tools can miss. I previously worked on the performance engineering team for Netflix, a large tech consumer running on hundreds of thousands of AWS instances. I'm now doing similar work at Intel (a large tech vendor) for Intel and their customers. As a leader in this space I've also interacted with other performance teams and staff doing performance work at many companies. In this post I'll explain what these teams do and when you should consider forming one. In part 2 I'll provide sample job descriptions, specialties, advice, pitfalls, comments on AI, and what to do if you can't hire a performance team. It's easy for hardware vendors like Intel to justify hiring performance engineers, as the number one factor in sales is beating a competitor's performance. However, my focus in this post is on non-vendor tech-heavy companies who hire these staff to drive down costs and latency outliers (e.g., banks, telecomms, defence, AI, tech-based companies, and anyone else who is spending more than $1M/year on back-end compute and AI). What is the ROI of performance engineering? The main ROIs are infrastructure cost savings, latency reductions, improved scalability and reliability, an ## Doom GPU Flame Graphs DevFeed: [Doom GPU Flame Graphs]() Original publisher: [Read original article]() Published: 2025-04-30T14:00:00Z Content type: article Language: en Sources: [Brendan Gregg's Blog]() Topics: [GPU](), [shaders](), [intel](), [Open Source](), [Code]() Tags: [blog](), [doom](), [dust](), [flame-graph](), [games](), [gaming](), [gpu](), [intel](), [open-source](), [performance](), [plugin](), [profile](), [shaders](), [svg]() ### AI overview The article demonstrates full-stack GPU flame graphs and GPU FlameScope using GZDoom, including Intel Battlemage GPU support. It shows how synchronized CPU and GPU profiling can correlate workload periods and identify GPU shader compilation and NIR preprocessing as sources of CPU activity. ### Source excerpt AI Flame Graphs are now open source and include Intel Battlemage GPU support, which means it can also generate full-stack GPU flame graphs for providing new insights into gaming performance, especially when coupled with FlameScope (an older open source project of mine). Here's an example of GZDoom, and I'll start with flame scopes for both CPU and GPU utilization, with details annotated: (Here are the raw CPU and GPU versions.) FlameScope shows a subsecond-offset heatmap of profile samples, where each column is one second (in this example, made up of 50 x 20ms blocks) and the color depth represents the number of samples, revealing variance and perturbation that you can select to generate a flame graph just for that time range. Update: the row size can be ajusted (it is limited by the sample rate captured in the profile), e.g., you could generate 60 rows to match 60fps games. Putting these CPU and GPU flame scopes side by side has enabled your eyes to do pattern matching to solve what would otherwise be a time-consuming task of performance correlation. The gaps in the GPU flame scope on the right - where the GPU was not doing much work - match the heavier periods of CPU work on the left. CPU Analysis FlameScope lets us click on the interesting periods. By selecting one of the CPU shader compilation stripes we get the flame graph just for that range: This is brilliant, and we can see exactly why the CPUs were busy for about 180 ms (the vertical length of the red stripe): it's doing compilation of GPU shaders and some NIR preprocessing (optimizations to the NIR intermediate representation that Mesa uses internally). If you are new to flame graphs, you look for the widest towers and optimize them first. Here is the interactive SVG. CPU flame graphs and CPU flame scope aren't new (from 2011 and 2018, both open source). What is new is full-stack GPU flame graphs and GPU flame scope. GPU Analysis Interesting details can also be selected in the GPU FlameScope for generating ## AI Flame Graphs DevFeed: [AI Flame Graphs]() Original publisher: [Read original article]() Published: 2024-10-28T13:00:00Z Content type: article Language: en Sources: [Brendan Gregg's Blog]() Topics: [Artificial Intelligence](), [GPU](), [intel](), [telemetry](), [Instrumentation](), [eBPF](), [C++](), [Code]() Tags: [ai](), [blog](), [c-plus-plus](), [ebpf](), [flame-graph](), [flamegraph](), [gpu](), [instrumentation](), [intel](), [performance](), [profiling](), [svg](), [visualization]() ### AI overview The article presents AI Flame Graphs, an Intel analyzer that visualizes AI accelerator or GPU instruction activity together with the software stack. It explains how the profiler uses Intel EU stall profiling and eBPF instrumentation to identify costly code paths, illustrating optimization differences in a SYCL matrix-multiply benchmark. ### Source excerpt Imagine halving the resource costs of AI and what that could mean for the planet and the industry -- based on extreme estimates such savings could reduce the total US power usage by over 10% by 20301. At Intel we've been creating a new analyzer tool to help reduce AI costs called AI Flame Graphs: a visualization that shows an AI accelerator or GPU hardware profile along with the full software stack, based on my CPU flame graphs. Our first version is available to customers in the Intel Tiber AI Cloud as a preview for the Intel Data Center GPU Max Series (previously called Ponte Vecchio). Here is an example: Simple example: SYCL matrix multiply microbenchmark (Click for interactive SVG.) The green frames are the actual instructions running on the AI or GPU accelerator, aqua shows the source code for these functions, and red (C), yellow (C++), and orange (kernel) show the CPU code paths that initiated these AI/GPU programs. The gray "-" frames just help highlight the boundary between CPU and AI/GPU code. The x-axis is proportional to cost, so you look for the widest things and find ways to reduce them. Layers This flame graph shows a simple program for SYCL (a high-level C++ language for accelerators) that tests three implementations of matrix multiply, running them with the same input workload. The flame graph is dominated by the slowest implementation, multiply_basic(), which doesn't use any optimizations and consumes at 72% of stall samples and is shown as the widest tower. On the right are two thin towers for multiply_local_access() at 21% which replaces the accessor with a local variable, and multiply_local_access_and_tiling() at 6% which also adds matrix tiling. The towers are getting smaller as optimizations are added. This flame graph profiler is a prototype based on Intel EU stall profiling for hardware profiling and eBPF for software instrumentation. It's designed to be easy and low-overhead, just like a CPU profiler. You should be able to generate a flame gr ## How eBPF Could Prevent Kernel Crashes from Security Software Updates DevFeed: [How eBPF Could Prevent Kernel Crashes from Security Software Updates]() Original publisher: [Read original article]() Published: 2024-07-21T14:00:00Z Content type: opinion Language: en Sources: [Brendan Gregg's Blog]() Topics: [eBPF](), [Security](), [Linux](), [Kernel](), [Windows](), [browsers]() Tags: [blog](), [browsers](), [ebpf](), [kernel](), [linux](), [outage](), [outages](), [security](), [windows]() ### AI overview This opinion article examines a major Windows outage caused by a security software configuration update that made a kernel driver read invalid memory. It argues that eBPF's verifier and sandboxed execution model could reduce the risk of kernel crashes, including for Windows once Microsoft's eBPF support is production-ready. ### Source excerpt In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code. Friday July 19th provided an unprecedented example of the inherent dangers of kernel programming, and has been called the largest outage in the history of information technology. Windows computers around the world encountered blue-screens-of-death and boot loops, causing outages for hospitals, airlines, banks, grocery stores, media broadcasters, and more. This was caused by a config update by a security company for their widely used product that included a kernel driver on Windows systems. The update caused the kernel driver to try to read invalid memory, an error type that will crash the kernel. For Linux systems, the company behind this outage was already in the process of adopting eBPF, which is immune to such crashes. Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well. These security agents will then be safe and unable to cause a Windows kernel crash. eBPF (no longer an acronym) is a secure kernel execution environment, similar to the secure JavaScript runtime built into web browsers. If you're using Linux, you likely already have eBPF available on your systems whether you know it or not, as it was included in the kernel several years ago. eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox. If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage. Some eBPF-based security startups (e.g., Oligo, Uptycs)