# Mark Seaborn Published articles for Mark Seaborn. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Observing interrupts from userland on x86 DevFeed: [Observing interrupts from userland on x86]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2018-01-08T20:39:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [x86](), [cpu](), [Kernel](), [Linux](), [Processes]() Tags: [architecture](), [c](), [cpu](), [kernel](), [linux](), [process](), [side-channel](), [x86]() ### AI overview This article explains how a userland process on x86 can detect that it has been interrupted without timing measurements. Setting %fs or %gs to 1 allows the x86 IRET instruction to reset the register to 0 when returning from an interrupt handler; a C program demonstrates the behavior on Linux. ### Source excerpt In 2016, I noticed a quirk of the x86 architecture that leads to an interesting side channel. On x86, it is possible for a userland process to detect when it has been interrupted by an interrupt handler, without resorting to timing. This is because the usual mechanism for handling interrupts (without using virtualisation) doesn't always preserve all userland registers across an interrupt handler. If a process sets a segment selector register such as %fs or %gs to 1, the register will get set to 0 when the process gets interrupted by an interrupt. Specifically, the x86 IRET instruction will reset the register to 0 when returning to userland -- this is the instruction that kernels use for returning from an interrupt handler. I have not seen this quirk explicitly documented anywhere, so I thought it was worthwhile documenting it via this blog post. The following C program demonstrates the effect: #include #include void set_gs(uint16_t value) { __asm__ volatile("mov %0, %%gs" : : "r"(value)); } uint16_t get_gs() { uint16_t value; __asm__ volatile("mov %%gs, %0" : "=r"(value)); return value; } int main() { uint16_t orig_gs = get_gs(); set_gs(1); unsigned int count = 0; /* Loop until %gs gets reset by an interrupt handler. */ while (get_gs() == 1) ++count; /* Restore register so as not to break TLS on x86-32 Linux. This is not necessary on x86-64 Linux, which uses %fs for TLS. */ set_gs(orig_gs); printf("%%gs was reset after %u iterations\n", count); return 0; } This works on x86-32 or x86-64. I tested it on Linux. It will print a non-deterministic number of iterations. For example: %gs was reset after 1807364 iterations Why this happens x86 segment registers are a bit weird, because each one has two parts: A program-visible 16-bit "segment selector" value, which can be read and written by the MOV instruction. A hidden part. When you write to a segment register using the MOV instruction, the CPU also fills out the hidden part. The hidden part includes ## PassMark received offer to not release rowhammer test DevFeed: [PassMark received offer to not release rowhammer test]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-10-20T19:36:00Z Content type: opinion Language: en Sources: [Mark Seaborn]() Topics: [bug](), [Software](), [systems](), [Maintainers]() Tags: [banking](), [bug](), [devices](), [maintainers](), [medical-devices](), [memory](), [passmark](), [rowhammer]() ### AI overview The article reports that PassMark received an anonymous offer to suppress a rowhammer test for its MemTest86 tool in exchange for payment, but released the software anyway. It discusses the potential implications of rowhammer-related memory unreliability for systems including medical devices, banking systems, and flight control systems. ### Source excerpt Here's an interesting report of skulduggery related to the rowhammer bug. PassMark say they received an offer to not release a rowhammer test in their MemTest86 tool, in return for payment: "We had anonymous contact offering to act as a go between between us and unnamed memory companies, with a view to paying us not release the new version of MemTest86. Who knows how serious the offer was. Needless to say we didn't take up that option, and just released the software anyway. But the issue is a BIG issue. The lack of publicity up to now is somewhat surprising considering the implications. Many computers are fundamentally (slightly) unreliable in a random ways. Maybe this doesn't matter for home use, but for medical devices, banking systems, flight control systems, etc.. it is a big deal." The quoted post is from 20th February 2015 - after the rowhammer bug was publicised by the CMU paper but before we published about the exploitability of the bug. Some background: PassMark are the maintainers of MemTest86. (MemTest86 should not be confused with MemTest86+ which is an alternative, open source fork of the same original codebase.) ## Passing FDs/handles between processes on Unix and Windows -- a comparison DevFeed: [Passing FDs/handles between processes on Unix and Windows -- a comparison]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-05-24T20:57:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Unix](), [Windows](), [Processes](), [API](), [systems](), [Programming]() Tags: [api](), [comparison](), [processes](), [programming](), [systems](), [unix](), [windows]() ### AI overview This article compares how Unix file descriptors and Windows handles are represented and passed between processes. It explains their shared distinction between numeric identifiers and underlying objects, then contrasts Unix socket-based descriptor passing with Windows use of the DuplicateHandle API. ### Source excerpt Handles on Windows are analogous to file descriptors (FDs) on Unix, and both can be passed between processes. However, the way in which handles/FDs can be passed between processes is quite different on Unix and Windows. In this blog post I'll explain the difference. You might find this useful if you are familiar with systems programming on either Unix or Windows but not both. Similarities I'll first explain what's the same on Unix and Windows. Both OSes have a distinction between FD/handle numbers and FD/handle objects. On both Windows and Unix, each process has its own FD/handle table which maps from FD/handle numbers to FD/handle objects: FD numbers are indexes into the FD table. On Unix, an FD number is an int. Windows uses the HANDLE type for handle numbers. Though HANDLE is typedef'd to void *, a HANDLE is really just a 32-bit index. (Windows does not use a HANDLE as a pointer into the process's address space.) FD objects are what FD numbers map to. User code never gets to see FD objects directly: it can only manipulate them via FD numbers. Multiple FD numbers can map to the same FD object. FD objects are (mostly) reference counted. All of this applies to handles on Windows too. Note: Some people use the alternative terminology that a "file description" refers to the underlying object while "file descriptor" refers to the number. I prefer to add "number" or "object" as a suffix as the way to disambiguate -- it is more explicit, and the term "file description" is not often used. Differences The key difference between Unix and Windows is this: On Unix, FD objects can be sent via sockets in messages. On Windows, handle objects cannot be sent in messages; only handle numbers can. Windows fills this gap by allowing one process to read or modify another process's handle table synchronously using the DuplicateHandle() API. Using this API involves one process dealing with another process's handle numbers. In contrast, Unix has no equivalent to DuplicateHandle(). A Unix ## Can cached memory accesses do double-sided row hammering? DevFeed: [Can cached memory accesses do double-sided row hammering?]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-05-11T22:35:00Z Content type: opinion Language: en Sources: [Mark Seaborn]() Topics: [Cache](), [cpu]() Tags: [cache](), [cpu](), [dram](), [mapping](), [rowhammer](), [xor]() ### AI overview This article examines whether double-sided Rowhammer attacks can be performed using cached memory accesses instead of CLFLUSH. For the author's Sandy Bridge test machine, cache-set, DRAM-bank, and row-address constraints make selecting suitable addresses impossible, leaving single-sided hammering or two sets of 13 addresses as possible approaches. ### Source excerpt There are indications that it is possible to cause bit flips in memory by row hammering without using CLFLUSH, using normal cached memory accesses. This makes me wonder: Is it possible to do double-sided row hammering using cached memory accesses, or only single-sided row hammering? The former is more likely to cause bit flips, and might be the only way to cause bit flips on some machines, such as those using a 2x refresh rate -- i.e. those configured to refresh DRAM every 32ms instead of every 64ms. (See the rowhammer blog post for more background.) The answer appears to be "no" -- at least on my test machine. For this machine (which has a Sandy Bridge CPU), I figured out how physical addresses map to cache sets and to banks and rows in DRAM. We can use these mappings to answer questions about what kinds of row hammering are possible using cached memory accesses. More specifically, my question is this: For a machine with an N-way L3 cache, is it possible to pick N+1 addresses that map to the same cache set, where at least two of these addresses map to rows R-1 and R+1 in one bank (for some neighbouring row R)? If so, repeatedly accessing these addresses would cause cache misses that cause rows R-1 and R+1 to be repeatedly activated. That puts more stress on row R (the victim row) than repeatedly activating only row R-1 or row R+1. The answer to this is "no": It's not possible to pick two such physical addresses. Here's why: Suppose we have two such addresses, A and B. Then: The addresses map to the same bank, so: (1): A[14:17] ^ A[18:21] = B[14:17] ^ B[18:21] (using the bank/row XOR scheme I described previously) The addresses are 2 rows apart, so: (2): A[18:32] + 2 = B[18:32] The addresses map to the same cache set, so: (3): A[6:17] = B[6:17] (also, SliceHash(A[17:32]) = SliceHash(B[17:32]), but we don't need this property) (2) implies that A[19] = ~B[19]. (3) implies that A[14:17] = B[14:17]. Combining that with (1) gives A[18:21] = B[18:21]. That implies A[19] = ## How physical addresses map to rows and banks in DRAM DevFeed: [How physical addresses map to rows and banks in DRAM]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-05-04T22:57:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [cpu](), [intel](), [bug](), [Testing]() Tags: [cpu](), [dram](), [intel](), [memory](), [rowhammer](), [testing]() ### AI overview The article examines how Intel Sandy Bridge memory controllers map physical addresses to DRAM rows, banks, and columns. Using a vulnerable test machine, it relates the mapping to Rowhammer testing and describes inferring and checking the mapping from observed aggressor and victim addresses. ### Source excerpt In my previous blog post, I discussed how Intel Sandy Bridge CPUs map physical addresses to locations in the L3 cache. Now I'll discuss how these CPUs' memory controllers map physical addresses to locations in DRAM -- specifically, to row, bank and column numbers in DRAM modules. Let's call this the DRAM address mapping. I'll use one test machine as a case study. Motivation: the rowhammer bug I am interested in the DRAM address mapping because it is relevant to the "rowhammer" bug. Rowhammer is a problem with some DRAM modules whereby certain pessimal memory access patterns can cause memory corruption. In these DRAMs, repeatedly activating a row of memory (termed "row hammering") can produce electrical disturbances that produce bit flips in vulnerable cells in adjacent rows of memory. These repeated row activations can be caused by repeatedly accessing a pair of DRAM locations that are in different rows of the same bank of DRAM. Knowing the DRAM address mapping is useful because it tells us which pairs of addresses satisfy this "same bank, different row" (SBDR) property. Guessing and checking an address mapping For my case study, I have a test machine containing DRAM that is vulnerable to the rowhammer problem. Running rowhammer_test on this machine demonstrates bit flips. I'd like to know what the DRAM address mapping is for this machine, but apparently it isn't publicly documented: This machine has a Sandy Bridge CPU, but Intel don't document the address mapping used by these CPUs' memory controllers. rowhammer_test does not actually need to identify SBDR address pairs. rowhammer_test just repeatedly tries hammering randomly chosen address pairs. Typically 1/8 or 1/16 of these pairs will be SBDR pairs, because our machine has 8 banks per DIMM (and 16 banks in total). So, while we don't need to know the DRAM address mapping to cause bit flips on this machine, knowing it would help us be more targeted in our testing. Though the address mapping isn't documented, I fo ## L3 cache mapping on Sandy Bridge CPUs DevFeed: [L3 cache mapping on Sandy Bridge CPUs]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-04-27T21:59:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Cache](), [cpu](), [intel](), [systems]() Tags: [cache](), [cpu](), [hash](), [intel](), [kernel](), [memory](), [paper](), [protocol](), [rowhammer](), [side-channel](), [slices](), [xor]() ### AI overview The article explains how physical addresses map to cache sets and slices in the L3 cache of Intel Sandy Bridge CPUs. It reports a previously published mapping for four-core CPUs and presents the author's mapping for two-core CPUs, with applications to kernel ASLR analysis and row-hammering research. ### Source excerpt In 2013, some researchers reverse-engineered how Intel Sandy Bridge CPUs map physical addresses to cache sets in the L3 cache (the last-level cache). They were interested in the cache mapping because it can be used to defeat kernel ASLR. I'm interested because the cache mapping can be used to test whether cached memory accesses can do row hammering (which can cause exploitable bit flips in some DRAM devices). The researchers published the details in the paper "Practical Timing Side Channel Attacks Against Kernel Space ASLR" (Ralf Hund, Carsten Willems and Thorsten Holz). They only published the mapping for 4-core CPUs, but I have figured out the mapping for 2-core CPUs as well. Some background: On Sandy Bridge CPUs, the L3 cache is divided into slices. Physical addresses are hashed to determine which slice of the L3 cache they will be stored in. The L3 cache is distributed and ring-based. There is one slice per core, but all the cores in a CPU can access all the cache slices via a ring bus which connects all the cores and their caches together. When a core accesses a memory location, the location will be slightly slower to access if it maps to a different core's cache slice, because it would take one or two hops around the ring bus to access it. The protocol used on the ring bus is based on QPI (Intel's QuickPath Interconnect). (QPI is a protocol used for connecting multiple CPUs together on high-end multi-socket systems.) Each cache slice contains 2048 cache sets. On lower-end CPUs, cache sets are 12-way associative, so a cache slice is 1.5MB in size (2048 sets * 12 ways * 64 bytes per cache line = 1.5MB). On higher-end CPUs, cache sets are 16-way associative, so a cache slice is 2MB in size (2048 sets * 16 ways * 64 bytes per cache line = 2MB). Cache mapping The researchers (Hund et al) figured out that the L3 cache uses the bits of a physical address as follows: Bits 0-5: These give the 6-bit byte offset within a 64-byte cache line. Bits 6-16: These give the 11-b ## The DRAM rowhammer bug is exploitable DevFeed: [The DRAM rowhammer bug is exploitable]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-03-10T21:53:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [bug](), [Security](), [Kernel]() Tags: [blog](), [bug](), [dram](), [kernel](), [project](), [rowhammer](), [security]() ### AI overview The article discusses the security implications of the DRAM rowhammer bug and notes published findings on exploiting it to gain kernel privileges. ### Source excerpt I've been researching the DRAM rowhammer issue and its security implications for a while. We've finally published our findings on the Project Zero blog: Exploiting the DRAM rowhammer bug to gain kernel privileges. ## Conditionalising C/C++ code: "#ifdef FOO" vs. "#if FOO" DevFeed: [Conditionalising C/C++ code: "#ifdef FOO" vs. "#if FOO"]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2015-01-21T20:14:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [C++](), [Code](), [Compiler](), [Chromium](), [MSVC](), [gcc](), [Windows]() Tags: [architecture](), [c-plus-plus](), [chromium](), [code](), [compiler](), [gcc](), [mistakes](), [msvc](), [windows]() ### AI overview The article compares #ifdef with #if for conditional compilation based on operating system and CPU architecture. It explains that defined macros can enable warnings for misspelled names when GCC/Clang warnings are enabled, while MSVC's corresponding warning is difficult to use because of system-header warnings. It also notes that runtime if statements can compile-test code across platforms, provided the code does not depend on platform-specific functions. ### Source excerpt Is it better to use #ifdef PLATFORM or #if PLATFORM when writing code that needs to be conditionalised according to OS, CPU architecture, etc.? Chromium's codebase uses the former. For example, it uses #ifdef OS_WIN or #if defined(OS_WIN), where OS_WIN is #defined on Windows (by build/build_config.h) and undefined elsewhere. In contrast, NaCl's codebase uses the latter. It uses #if NACL_WINDOWS or if (NACL_WINDOWS), where NACL_WINDOWS is always #defined: as 1 on Windows and 0 elsewhere. This latter approach has the benefit of catching some mistakes: If you mistype the macro name in #if NACL_WINDOWS, you can get a compiler warning. However, this only works if you enable GCC/Clang's -Wundef warning. Microsoft's compiler (MSVC) has a similar warning, /wd4668, but it's effectively unusable because it produces warnings about system header files. You can sometimes write if (PLATFORM) instead of #if PLATFORM. The if() version has the advantage that the code in the if() block will be compile-tested on all platforms, not just those where PLATFORM == 1. This can help catch mistakes earlier. This doesn't work if the code block uses functions that are only defined on PLATFORM, though. See also: The Great -Wundef purge ## Implementing fork() on the Mill CPU DevFeed: [Implementing fork() on the Mill CPU]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2014-07-23T03:12:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [cpu](), [Processes](), [Unix](), [systems]() Tags: [architecture](), [cache](), [cpu](), [performance](), [processes](), [unix]() ### AI overview This article examines how the Mill CPU architecture might implement Unix fork(). It explains the architecture's Single Address Space OS model and speculates that forked processes could require flushing affected TLB and cache ranges during context switches. The performance impact would depend on the Mill's cache and TLB design, while copy-on-write and the typical short lifetime of forked processes could limit the cost. ### Source excerpt The Mill is a new CPU architecture that claims to provide high performance but at a much better performance-per-watt than conventional CPUs that use out-of-order execution. The Mill achieves this by making various architectural simplifications. One of those is to remove the TLB from the fast path of memory accesses. Rather than having the TLB between the CPU core and the cache, the Mill's TLB is between the cache and main memory. OS models This means the Mill is best suited for running Single Address Space operating systems (SASOS). The intent is that different processes will live at different addresses within a shared 64-bit address space. A process runs with permissions to access restricted ranges of this address space. Switching between processes is therefore just a matter of switching those permissions, which is fast on the Mill. It doesn't involve an expensive flush of the TLB (as on a conventional OS). It doesn't involve flushing the Mill's virtual-address-tagged (VIVT) cache. This runs into a problem if we want to run Unix programs that use fork(), though. Use of fork() assumes that multiple processes will want to use the same virtual addresses. The Mill developers have said they have a scheme for handling fork(), but they haven't said what it is, so I'm going to speculate. :-) Context switching If you create forked processes that share a range of virtual address space, a Mill OS can just flush the TLB and cache for those ranges whenever it needs to context switch between the two processes. Basically, a Mill OS can act like a Separate Address Space OS when handling forked processes. Switching costs How expensive that would be depends on how the TLB and cache work. In conventional CPUs, the TLB is per-core. The Mill's TLB might be shared between cores, though, if the Mill has higher-level caches that are both virtually-tagged and shared between cores. If that's the case, forked processes wouldn't be able to run concurrently on multiple cores. Flushing the TLB ## How to do history-sensitive merges in Git DevFeed: [How to do history-sensitive merges in Git]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2014-03-26T17:17:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [Git](), [LLVM]() Tags: [git](), [history](), [how-to](), [llvm](), [merge](), [patches](), [refactoring](), [upgrade](), [version-control]() ### AI overview This tutorial explains how to perform history-sensitive merges in Git by applying upstream changes one at a time. It uses the merge of LLVM 3.4 into a patched PNaCl LLVM branch to show how considering commit history can reduce conflicts and provide more context when resolving them. ### Source excerpt Merging in Git is usually not history-sensitive. By this I mean: if you're merging branches A and B together, Git looks at the content at the tips of branches A and B, and the content of the common ancestor commit(s) of A and B, but it doesn't look at any commits inbetween. Git just does a 3-way merge. This can make merging more painful than it needs to be. History-sensitive merging is a merge algorithm that does look at the commit history. It can automatically resolve conflicts in cases where non-history-sensitive merging won't. It can also present conflicts more readably: It can reduce the size of a conflict and show which upstream change conflicted with your local change. This is a short "how to" for doing a history-sensitive merge in Git. My case study is: Merging LLVM 3.4 into PNaCl's branch of LLVM, which I did recently. The problem PNaCl has a branch of LLVM with various patches applied. (As an aside, we should really upstream those patches, to reduce the difficulty of merging from upstream.) Before the merge, this branch was based on LLVM 3.3 (commit $FROM). We wanted to upgrade it to being based on LLVM 3.4 (commit $TO). Simply doing git merge $TO produced about 160 conflicts, as counted by "git grep '<<<<<<<' | wc -l". A large number of those conflicts occurred because pnacl-llvm contained various changes cherry-picked from upstream LLVM. Often those changes touched code that was later refactored upstream. That will produce a conflict if we do a 3-way merge. For example, pnacl-llvm cherry-picked various changes that add test files with "CHECK:" lines. Upstream later refactored these to "CHECK-LABEL:". If we do "git merge $TO", Git will see that the two branches added a file with the same name but different contents. Bam: that's a conflict. Git's 3-way merge doesn't look at the history, so it doesn't notice that both branches contain identical commits that add the file with a "CHECK:" line, and just one of the two branches later modifies that file. A soluti ## Handling crashes on Mac OS X: ordering of Mach exceptions versus POSIX signals DevFeed: [Handling crashes on Mac OS X: ordering of Mach exceptions versus POSIX signals]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2013-08-07T20:25:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [Operating system](), [Kernel](), [Processes](), [Exception](), [POSIX](), [out-of-process](), [Remote Procedure Call (RPC)]() Tags: [crash](), [dialog](), [exception](), [exception-handling](), [experimentation](), [handler](), [kernel](), [mac](), [mac-os](), [memory](), [ordering](), [os](), [out-of-process](), [posix](), [process](), [processes](), [rpc](), [signal]() ### AI overview This article explains how Mac OS X handles hardware exceptions through POSIX signals and Mach exceptions. It reports that Mach exception handlers receive priority, while the kernel uses a first-chance Mach handler, a POSIX signal handler, and a second-chance Mach handler when handling faults and crashes. ### Source excerpt Mac OS X is a curious operating system because its kernel is derived from two kernel codebases -- the Mach kernel and a BSD kernel -- that have been glued together. From these two ancestors, OS X inherits two different mechanisms for processes to handle hardware exceptions (a.k.a. faults): POSIX signals: In-process only. Registered per-process. The handler is always called on the thread that produced the fault. Mach exceptions: Allow both in-process and out-of-process handling. Can be registered per-thread and per-process. The handler is invoked via Mach RPC. On OS X, it's possible for a process to have both POSIX signal handlers and Mach exception handlers registered. It's not immediately obvious which of the two handlers will take priority and get invoked first, but experimentation shows that it's the Mach handler. If a process faults with a memory access error, the Mach exception handler for EXC_BAD_ACCESS gets invoked first, and if this handler returns KERN_FAILURE, the POSIX signal handler for SIGBUS will then be invoked. However, that's not the full story. OS X has a built-in Crash Reporter service which will create a crash dump if an application crashes and pop up a dialog box. Crash Reporter works via Mach exception handling and runs out-of-process. An application will normally have Crash Reporter's crash handler registered as one of its default Mach exception handlers. But what stops this handler from interfering with normal use of POSIX signals, if Mach exception handlers take priority over POSIX signals? The answer is that OS X's kernel has three steps for handling a hardware exception: First-chance Mach exception handler: The kernel tries to invoke the Mach exception handler registered for the type of fault that occurred, e.g. EXC_BAD_ACCESS. If there's no handler registered, it skips this step. If the handler returns KERN_SUCCESS, the kernel resumes the thread that faulted. POSIX signal handler: If the Mach exception handler was absent or returned KERN_ ## Simplifying LLVM IR for PNaCl DevFeed: [Simplifying LLVM IR for PNaCl]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2013-06-29T19:08:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [LLVM](), [c/c++](), [browser](), [Web](), [x86]() Tags: [browser](), [c-plus-plus](), [clang](), [code](), [encoding](), [intermediate](), [llvm](), [native](), [optimisation](), [portable](), [x86]() ### AI overview This article explains how PNaCl simplifies LLVM IR to create a maintainable, architecture-neutral executable format for native code in web browsers. It describes removing types, names, metadata, and complex instructions, and compares the approach with Emscripten and asm.js. ### Source excerpt Lately I've been working on Portable Native Client ("PNaCl" for short). Native Client (NaCl) is a sandboxing system that allows safe execution of native code in a web browser -- typically C/C++ code compiled to x86-32, x86-64 or ARM native code (with some support for MIPS too). The problem with NaCl has always been that you have to compile your program multiple times, once for each target architecture. PNaCl aims to solve that, so that your program can be compiled once to an architecture-neutral executable which is then translated to x86 or ARM inside the web browser. PNaCl is based on LLVM, and PNaCl's executable format is based on LLVM's IR ("Intermediate Representation") language and its binary encoding, LLVM bitcode. LLVM's IR language is quite complex, so we're pruning out and expanding out a lot of features in order to ensure that PNaCl will be maintainable in the long term and to reduce executable sizes. As an example, consider this fragment of C code: struct foo { int x; int y; }; void func(struct foo *ptr) { ptr->y = 123; } Normally in LLVM this compiles to the following LLVM assembly code: %struct.foo = type { i32, i32 } define void @func(%struct.foo* nocapture %ptr) #0 { entry: %y = getelementptr inbounds %struct.foo* %ptr, i32 0, i32 1 store i32 123, i32* %y, align 4, !tbaa !0 ret void } attributes #0 = { nounwind } !0 = metadata !{metadata !"int", metadata !1} With PNaCl, we strip out the types, names and metadata so that this becomes: define internal void @67(i32) { %2 = add i32 %0, 4 %3 = inttoptr i32 %2 to i32* store i32 123, i32* %3, align 1 ret void } The definition of the struct type goes away. The "getelementptr" instruction is how LLVM handles pointer arithmetic for indexing into structs and arrays -- this gets expanded out and is replaced with pointer arithmetic on integers. The "%struct.foo*" pointer type is replaced with the i32 type. The "inttoptr" instruction remains as a vestige of LLVM's type system: every "load" or "store" instruction in ## Native Client's NTDLL patch on x86-64 Windows DevFeed: [Native Client's NTDLL patch on x86-64 Windows]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2012-09-29T02:54:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [Security](), [Windows](), [x86](), [Exception](), [Kernel](), [Chrome](), [Processes](), [POSIX](), [Unix]() Tags: [bug](), [chrome](), [code](), [exception](), [kernel](), [native-client](), [posix](), [process](), [security](), [unix](), [windows](), [x86-64]() ### AI overview This technical article explains a security hole in Native Client on 64-bit Windows that could allow escape from the Native Client sandbox. It describes how differences between Windows vectored exception handling and Unix signal handling created the problem, and notes that Native Client used a process-local in-memory patch to NTDLL to prevent it. ### Source excerpt Last year, I found a security hole in Native Client on 64-bit Windows that could be used to escape from the Native Client sandbox. Fortunately I found the hole before Native Client was enabled by default in Chrome. I recently wrote up the document below to explain the bug and how we fixed it. Native Client currently relies on a small, process-local, in-memory patch to NTDLL on 64-bit versions of Windows in order to prevent a problem that would create a hole in Native Client's x86-64 sandbox. The problem The problem arises because Windows does not have an equivalent of Unix's sigaltstack() system call. On Unix, a POSIX signal handler may be registered using signal() or sigaction(). When a process faults (e.g. as a result of a memory access error or an illegal instruction), the kernel passes control to the signal handler that is registered for the process. Normally, the signal handler is run on the stack of the thread that faulted. However, if an "alternate signal stack" has been registered using sigaltstack(), the signal handler will run on that stack instead. On Windows, "vectored exception handlers" are similar to Unix signal handlers, except that a vectored exception handler always gets run on the thread's current stack. This might have been OK, except that Windows has a different division of responsibility between kernel and userland compared with Unix. Windows does more in userland, and this creates problems for NaCl. On Unix, sigaction() is a syscall that user code calls that records a user-code signal handler function in a kernel data structure. If no signal handler is registered by user code, then when a fault occurs in a process, the kernel never passes control back to userland code in that process. On Windows, AddVectoredExceptionHandler() is provided in userland by a DLL rather than being provided by the kernel. AddVectoredExceptionHandler() adds the handler function to a list in userland memory. When a fault occurs in the process, the Windows kernel passe ## Stack unwinding risks on 64-bit Windows DevFeed: [Stack unwinding risks on 64-bit Windows]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2011-11-19T20:24:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [x86](), [Assembly](), [Exception](), [Windows](), [Code]() Tags: [assembly](), [code](), [exception](), [layout](), [processes](), [programming](), [state](), [windows](), [x86]() ### AI overview The article examines a fallback rule in x86-64 Windows stack unwinding when a return address lacks unwind information. It argues that repeatedly applying the rule can make the unwinder interpret invalid stack layouts as valid frames, potentially increasing exploitation risks in programs with corrupted stacks or assembly code without unwind information. ### Source excerpt Recently, I've been looking at how x86-64 Windows does stack unwinding in 64-bit processes, and I've found some odd behaviour. If the stack unwinder finds a return address on the stack that does not have associated unwind info, it applies a fallback unwind rule that does not make much sense. I've been wondering if this could make some x86-64 programs more easily exploitable if they corrupt the stack or if they use x86-64 assembly code that does not have unwind info. In pseudocode, the current unwind logic looks something like this: void unwind_stack(struct register_state regs) { while (???) { unwind_info = get_unwind_info(regs.rip); if (unwind_info) { if (has_exception_handler(unwind_info) { // Run exception handler... } regs = unwind_stack_frame(unwind_info, regs); } else { // Fallback case for leaf functions: regs.rip = *(uint64_t *) regs.rsp; regs.rsp += 8; } } } The issue is that the fallback case only makes sense for the first iteration of the loop. The fact that it is applied to later iterations is probably just sloppy programming. The fallback case is intended to handle "leaf functions". This means functions that: do not adjust %rsp, and do not call other functions. These two properties are related: if a function calls other functions, it must adjust %rsp first, otherwise it does not conform to the x86-64 ABI. Since the fallback case is applied repeatedly, the unwinder will happily interpret the stack as a series of return addresses with no gaps between them: ... 8 bytes return address 8 bytes return address 8 bytes return address ... However, those are not valid stack frames in the x86-64 Windows ABI. A valid stack frame for a non-leaf function Foo() (i.e. a function that calls other functions) looks like this: ------------ 16-byte aligned 32 bytes "shadow space" (scratch space for Foo()) 8 bytes return address (points into Foo()'s caller) 8 bytes scratch space for Foo() 16*n bytes scratch space for Foo() (for some n >= 0) 32 bytes "shadow space" (scratch sp ## ARM cache flushing & doubly-mapped pages DevFeed: [ARM cache flushing & doubly-mapped pages]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2011-11-17T08:40:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Cache](), [Arm](), [Linux](), [Kernel](), [JIT]() Tags: [architecture](), [arm](), [cache](), [implementation](), [jit](), [kernel](), [linux](), [memory]() ### AI overview The article explains why ARM Linux code using doubly mapped pages must flush the cache for both the writable and executable virtual-address mappings. The cacheflush syscall clears the instruction cache and commits data-cache writes, so flushing only the executable mapping is insufficient for Native Client's dynamic code loading. ### Source excerpt If you're familiar with the ARM architecture you'll probably know that self-modifying code has to be careful to flush the instruction cache on ARM. (Back in the 1990s, the introduction of the StrongARM with its split instruction and data caches broke a lot of programs on RISC OS.) On ARM Linux there's a syscall, cacheflush, for flushing the instruction cache for a range of virtual addresses. This syscall works fine if you map some code as RWX (read+write+execute) and execute it from the same virtual address that you use to modify it. This is how JIT compilers usually work. In the Native Client sandbox, though, for dynamic code loading support, we have code pages that are mapped twice, once as RX (read+execute) and once as RW (read+write). Naively you'd expect that after you write instructions to the RW address, you just have to call cacheflush on the RX address. However, that's not enough. cacheflush doesn't just clear the i-cache. It also flushes the data cache to ensure that writes are committed to memory so that they can be read back by the i-cache. The two parts are clear if you look at the kernel implementation of cacheflush, which does two different MCRs on the address range. I guess the syscall interface was not designed with double-mapped pages in mind, since it doesn't allow the i-cache and d-cache to be flushed separately. For the time being, Native Client will have to call cacheflush on both the RW and RX mappings. See NaCl issue 2443 for where this came up. ## Fixing the trouble with Buildbot DevFeed: [Fixing the trouble with Buildbot]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2011-08-23T15:59:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Shell](), [Bash](), [Python](), [make](), [x86](), [Chromium]() Tags: [atomic](), [chrome](), [code](), [make](), [python](), [script](), [shell-script](), [tests](), [x86]() ### AI overview The article explains how Buildbot Annotations let checked-in build scripts divide sequential build output into separately displayed steps. It describes the benefits for maintaining and testing Native Client build logic, while noting limitations including per-step timeout handling, non-nestable steps, and an awkward syntax. ### Source excerpt Last year I wrote a blog post, "The trouble with Buildbot", about how Buildbot creates a dilemma for complex projects because it forces you to choose between two ways of describing a project's build steps: You can describe build steps in the Buildbot config. Buildbot configs are awkward to update -- someone has to restart the Buildbot master -- and hard to test, but you get the benefit that the build steps appear as separate steps in Buildbot's display. You can write a script which runs the build steps directly, and check it into the same repository as your project. This is easier to maintain and test, but traditionally all the output from the script would appear as a single Buildbot build step, making the output hard to read. Fortunately, Brad Nelson has addressed this problem with an extension to Buildbot known as "Buildbot Annotations". The Python code for this currently lives in chromium_step.py (see AnnotatedCommand). The idea is that your checked-in script will run multiple steps sequentially but output tags between them (e.g. "@@@BUILD_STEP tests@@@") so that the output can be parsed into chunks by the Buildbot master, and displayed as separate chunks. For example, an early version of Native Client's Annotations-based buildbot script looked something like this: ... echo @@@BUILD_STEP gyp_compile@@@ make -C .. -k -j12 V=1 BUILDTYPE=${GYPMODE} echo @@@BUILD_STEP scons_compile${BITS}@@@ ./scons -j 8 -k --verbose ${GLIBCOPTS} --mode=${MODE}-host,nacl \ platform=x86-${BITS} echo @@@BUILD_STEP small_tests${BITS}@@@ ./scons -k --verbose ${GLIBCOPTS} --mode=${MODE}-host,nacl small_tests \ platform=x86-${BITS} || { RETCODE=$? && echo @@@STEP_FAILURE@@@;} ... (More recently, this shell script has been replaced with a Python script.) You can see this in use on the Native Client Buildbot page (and also on the trybot page, though that's less readable). The logic for running NaCl's many build steps -- including a clobber step, a Scons build, a Gyp build, small_tests, mediu ## Cookies versus the Chrome sandbox DevFeed: [Cookies versus the Chrome sandbox]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2011-02-10T01:34:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Chrome](), [browser](), [vulnerability](), [Exploit](), [HTML]() Tags: [browser](), [chrome](), [cookies](), [exploit](), [html](), [sandbox]() ### AI overview The article examines how a Chrome renderer-process vulnerability could allow a malicious site to access login cookies from another site through the interaction of cookies and framed pages. It explains that Chrome's sandbox offers limited cross-site protection in this scenario and discusses mitigation through separate browser profiles or avoiding cookies. ### Source excerpt Although Chrome's sandbox does not protect one web site from another in general, it can provide such protection in some cases. Those cases are ones in which HTTP cookies are either reduced in scope or not used at all. One lesson we could draw from this is that cookies reduce the usefulness of Chrome's sandbox. The scenario we are exploring supposes that there is a vulnerability in Chrome's renderer process, and that the vulnerability lets a malicious site take control of the renderer process. This means that all the restrictions that are normally enforced on the malicious site by the renderer process are stripped away, and all we are left with are the restrictions enforced on the renderer process by the Chrome browser process and the Chrome sandbox. In my previous blog post, I explained how an attacker site, evil.com, that manages to exploit the renderer process could steal the login cookies from another site, mail.com, and so gain access to the user's e-mail. The attack is made possible by the combination of two features: cookies frames Chrome currently runs a framed page in the same renderer process as the parent page. HTML standards allow framed pages to access cookies, so the browser process has to give the renderer process access to the cookies for both pages. Because this problem arises from the interaction of these features, one site is not always vulnerable to other sites. There should be a couple of ways that users and sites can mitigate the problem, without changing Chrome. Firstly, the user can change how cookies are scoped within the browser by setting up multiple profiles. Secondly, a site can skirt around the problem by not using cookies at all. We discuss these possibilities below. Use multiple profiles: As a user, you can create multiple browser profiles, and access mail.com and evil.com in separate profiles. Chrome does not make this very easy at the moment. It provides a command line option (--user-data-dir) for creating more profiles, but this fea ## A common misconception about the Chrome sandbox DevFeed: [A common misconception about the Chrome sandbox]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-12-21T00:34:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Chrome](), [Security](), [Processes](), [Document Object Model (DOM)](), [Memory Safety]() Tags: [browser](), [chrome](), [chromium](), [memory-safety](), [processes](), [security]() ### AI overview The article explains that Chrome's sandbox does not isolate one website from another when sites share an exploited renderer process. A malicious site may use an iframe to cause another site to run in the same renderer, potentially allowing access to that site's cookies. Chrome's multi-process architecture instead provides renderer-crash containment and protection of the wider user system. ### Source excerpt A common misconception about the Chrome web browser is that its sandbox protects one web site from another. For example, suppose you are logged into your e-mail account on mail.com in one tab, and have evil.com open in another tab. Suppose evil.com finds an exploit in the renderer process, such as a memory safety bug, that lets it run arbitrary code there. Can evil.com get hold of your HTTP cookies for mail.com, and thereby access your e-mail account? Unfortunately, the answer is yes. The reason is that mail.com and evil.com can be assigned to the same renderer process. The browser does not only do this to save memory. evil.com can cause this to happen by opening an iframe on mail.com. With mail.com's code running in the same exploited renderer process, evil.com can take it over and read the cookies for your mail.com account and use them for its own ends. There are a couple of reasons why the browser puts a framed site in the same renderer process as the parent site. Firstly, if the sites were handled by separate processes, the browser would have to do costly compositing across renderer processes to make the child frame appear inside the parent frame. Secondly, in some cases the DOM allows Javascript objects in one frame to obtain references to DOM objects in other frames, even across origins, and it is easier for this to be managed within one renderer process. I don't say this to pick on Chrome, of course. It is better to have the sandbox than not to have it. Chrome has never claimed that the sandbox protects one site against another. In the tech report "The Security Architecture of the Chromium Browser" (Barth, Jackson, Reis and the Chrome Team; 2008), "Origin isolation" is specifically listed under "Out-of-scope goals". They state that "an attacker who compromises the rendering engine can act on behalf of any web site". There are a couple of ways that web sites and users can mitigate this problem, which I'll discuss in another post. However, in the absence of tho ## When printf debugging is a luxury DevFeed: [When printf debugging is a luxury]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-12-18T19:06:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [debugging](), [Code](), [Linux](), [Chromium]() Tags: [chromium](), [code](), [debugging](), [glibc](), [libc](), [linux](), [native-client](), [sandbox]() ### AI overview This article explains why printf() and assert() may be unavailable in certain low-level programming contexts, including signal handlers, limited-stack environments, and code that cannot use libc. It presents a simple assertion implementation that constructs failure messages at compile time and discusses bypassing libc to invoke Linux system calls when TLS register state makes libc wrappers unusable. ### Source excerpt Inserting printf() calls is often considered to be a primitive fallback when other debugging tools are not available, such as stack backtraces with source line numbers. But there are some situations in low-level programming where most libc calls don't work and so even printf() and assert() are unavailable luxuries. This can happen: when libc is not properly initialised yet; when we writing code that is called by libc and cannot re-enter libc code; when we are in a signal handler; when only limited stack space is available; when we cannot allocate memory for some reason; or when we are not even linked to libc. Here's a fragment of code that has come in handy in these situations. It provides a simple assert() implementation: #include #include static void debug(const char *msg) { write(2, msg, strlen(msg)); } static void die(const char *msg) { debug(msg); _exit(1); } #define TO_STRING_1(x) #x #define TO_STRING(x) TO_STRING_1(x) #define assert(expr) { \ if (!(expr)) die("assertion failed at " __FILE__ ":" TO_STRING(__LINE__) \ ": " #expr "\n"); } By using preprocessor trickery to construct the assertion failure string at compile time, it avoids having to format the string at runtime. So it does not need to allocate memory, and it doesn't need to do multiple write() calls (which can become interleaved with other output in the multi-threaded case). Sometimes even libc's write() is a luxury. In some builds of GNU libc on Linux, glibc's syscall wrappers use the TLS register (%gs on i386) to fetch the address of a routine for making syscalls. However, if %gs is not set up properly for some reason, this will fail. For example, for Native Client's i386 sandbox, %gs is set to a different value whenever sandboxed code is running, and %gs stays in this state if sandboxed code faults and triggers a signal handler. In Chromium's seccomp-sandbox, %gs is set to zero in the trusted thread. In those situations we have to bypass libc and do the system calls ourselv ## An introduction to FreeBSD-Capsicum DevFeed: [An introduction to FreeBSD-Capsicum]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-11-04T12:48:00Z Content type: article Language: en Sources: [Mark Seaborn]() Topics: [Unix](), [Authorization](), [Processes]() Tags: [features](), [files](), [filesystem](), [introduction](), [permission](), [process](), [unix]() ### AI overview An overview of FreeBSD Capsicum, a capability-based security framework for sandboxing. It describes capability mode, restricted file-descriptor permissions, process descriptors, message-based sockets, and fexecve(). ### Source excerpt In my last blog post, I described one of the features in FreeBSD-Capsicum: process descriptors. Now it's time for an overview of Capsicum. Capsicum is a set of new features for FreeBSD that adds better support for sandboxing, using a capability model in which the capabilities are Unix file descriptors (FDs). Capsicum takes a fairly conservative approach, in that it does not make operations on file descriptors virtualisable. This approach has some limitations -- we do not get the advantages of having purely message-passing syscalls. However, it does mean that the new features are orthogonal. The main new features are: A per-process "capability mode", which is turned on via a new cap_enter() syscall. This mode disables any system call that provides ambient authority. So it disables system calls that use global namespaces, including the file namespace (e.g. open()), the PID namespace (e.g. kill()) and the network address namespace (e.g. connect()). This is not just a syscall filter, though. Some system calls optionally use a global namespace. For example, sendmsg() and sendto() optionally take a socket address. For openat(), the directory FD can be omitted. Capability mode disables those cases. Furthermore, capability mode disallows the use of ".." (parent directory) in filenames for openat() and the other *at() calls. This changes directory FDs to be limited-authority objects that convey access to a specific directory and not the whole filesystem. (It is interesting that this appears to be a property of the process, via capability mode, rather than of the directory FD itself.) Capability mode is inherited across fork and exec. Finer-grained permissions for file descriptors. Each FD gets a large set of permission bits. A less-permissive copy of an FD can be created with cap_new(). For example, you can have read-only directory FDs, or non-seekable FDs for files. Process descriptors. Capsicum doesn't allow kill() inside the sandbox because kill() uses a global namespace ## Process descriptors in FreeBSD-Capsicum DevFeed: [Process descriptors in FreeBSD-Capsicum]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-10-23T15:10:00Z Content type: opinion Language: en Sources: [Mark Seaborn]() Topics: [Processes](), [POSIX](), [Unix](), [Kernel](), [debugging]() Tags: [argument](), [fork](), [kernel](), [linux](), [memory](), [posix](), [process](), [processes](), [race-condition](), [sandbox](), [signal](), [unix]() ### AI overview The article examines Capsicum process descriptors in FreeBSD. It explains how pdfork(), pdwait(), and pdkill() replace PID-based process operations with file-descriptor-based operations, improving delegation, polling, and protection against PID-reuse race conditions. It also criticizes the design in which closing the last process descriptor terminates the process. ### Source excerpt Capsicum is a set of new features for FreeBSD that adds better support for sandboxing, adding a capability mode in which the capabilities are Unix file descriptors (FDs). The features Capsicum adds are orthogonal, which is nice. One of the new features is process descriptors. Capsicum adds a replacement for fork() called pdfork(), which returns a process descriptor (a new type of FD) rather than a PID. Similarly, there are replacements for wait() and kill() -- pdwait() and pdkill() -- which take FDs as arguments instead of PIDs. The reason for the new interface is that kill() is not safe to allow in Capsicum's sandbox, because it provides ambient authority: it looks up its PID argument in a global namespace. But even if you ignore sandboxing issues, this new interface is a significant improvement on POSIX process management: It allows the ability to wait on a process to be delegated to another process. In contrast, with wait()/waitpid(), a process's exit status can only be read by the process's parent. Process descriptors can be used with poll(). This avoids the awkwardness of having to use SIGCHLD, which doesn't work well if multiple libraries within the same process want to wait() for child processes. It gets rid of the race condition associated with kill(). Sending a signal to a PID is dodgy because the original process with this PID could have exited, and the kernel could have recycled the PID for an unrelated process, especially on a system where processes are spawned and exit frequently. kill() is only really safe when used by a parent process on its child, and only when the parent makes sure to use it before wait() has returned the child's exit status. pdkill() gets rid of this problem. In future, process descriptors can be extended to provide access to the process's internal state for debugging purposes, e.g. for reading registers and memory, or modifying memory mappings or the FD table. This would be an improvement on Linux's ptrace() interface. However, th ## My workflow with git-cl + Rietveld DevFeed: [My workflow with git-cl + Rietveld]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-08-11T11:49:00Z Content type: tutorial Language: en Sources: [Mark Seaborn]() Topics: [Git](), [Code review](), [Development]() Tags: [code-review](), [git](), [latency](), [workflow]() ### AI overview The article describes a Git workflow for managing dependent changes submitted separately for review through Rietveld. It uses one branch per change, rebases later changes onto revised earlier ones, and squashes commits before uploading or committing upstream. The author also explains problems caused by git-cl's limited support for dependent patch series, including manual tracking, conflicts, and lost history. ### Source excerpt Git's model of changes (which is shared by Mercurial, Bazaar and Monotone) makes it awkward to revise earlier patches. This can make things difficult when you are sending out multiple, dependent changes for code review. Suppose I create changes A and B. B depends functionally on A, i.e. tests will not pass for B without A also being applied. There might or might not be a textual dependency (B might or might not modify lines of code modified by A). Because code review is slow (high latency), I need to be able to send out changes A and B for review and still be able to continue working on further changes. But I also need to be able to revisit A to make changes to it based on review feedback, and then make sure B works with the revised A. What I do is create separate branches for A and B, where B branches off of A. To revise change A, I "git checkout" its branch and add further commits. Later I can update B by checking it out and rebasing it onto the current tip of A. Uploading A or B to the review system or committing A or B upstream (to SVN) involves squashing their branch's commits into one commit. (This squashing means the branches contain micro-history that reviewers don't see and which is not kept after changes are pushed upstream.) The review system in question is Rietveld, the code review web app used for Chromium and Native Client development. Rietveld does not have any special support for patch series -- it is only designed to handle one patch at a time, so it does not know about dependencies between changes. The tool for uploading changes from Git to Rietveld and later committing them to SVN is "git-cl" (part of depot_tools). git-cl is intended to be used with one branch per change-under-review. However, it does not have much support for handling changes which depend on each other. This workflow has a lot of problems: When using git-cl on its own, I have to manually keep track that B is to be rebased on to A. When uploading B to Rietveld, I must do "git cl u ## CVS's problems resurface in Git DevFeed: [CVS's problems resurface in Git]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-08-06T20:12:00Z Content type: opinion Language: en Sources: [Mark Seaborn]() Topics: [Git](), [Data structures](), [Graphs](), [systems]() Tags: [data-structure](), [git](), [graph](), [systems]() ### AI overview The article argues that modern version-control systems address important CVS limitations by storing history per repository as a directed acyclic graph of commit objects, but that repository-level branching, full-history checkouts, and the difficulty of combining existing repositories leave a fundamental problem unresolved. ### Source excerpt Although modern version control systems have improved a lot on CVS, I get the feeling that there is a fundamental version control problem that the modern VCSes (Git, Mercurial, Bazaar, and I'll include Subversion too!) haven't solved. The curious thing is that CVS had sort of made some steps towards addressing it. In CVS, history is stored per file. If you commit a change that crosses multiple files, CVS updates each file's history separately. This causes a bunch of problems: CVS does not represent changesets or snapshots as first class objects. As a result, many operations involve visiting every file's history. Reconstructing a changeset involves searching all files' histories to match up the individual file changes. (This was just about possible, though I hear there are tricky corner cases. Later CVS added a commit ID field that presumably helped with this.) Creating a tag at the latest revision involves adding a tag to every file's history. Reconstructing a tag, or a time-based snapshot, involves visiting every file's history again. CVS does not represent file renamings, so the standard history tools like "cvs log" and "cvs annotate" are not able to follow a file's history from before it was renamed. In the DAG-based decentralised VCSes (Git, Mercurial, Monotone, Bazaar), history is stored per repository. The fundamental data structure for history is a Directed Acyclic Graph of commit objects. Each commit points to a snapshot of the entire file tree plus zero or more parent commits. This addresses CVS's problems: Extracting changesets is easy because they are the same thing as commit objects. Creating a tag is cheap and easy. Recording any change creates a commit object (a snapshot-with-history), so creating a tag is as simple as pointing to an already-existing commit object. However, often it is not practical to put all the code that you're interested in into a single Git repository! (I pick on Git here because, of the DAG-based systems, it is the one I am most ## The trouble with Buildbot DevFeed: [The trouble with Buildbot]() Original publisher: [Read original article]() Author: Mark Seaborn (noreply@blogger.com) Published: 2010-05-05T21:42:00Z Content type: opinion Language: en Sources: [Mark Seaborn]() Topics: [configuration](), [Testing](), [ci](), [Script](), [Logging](), [make](), [Ubuntu]() Tags: [automated](), [config](), [configuration](), [logging](), [logs](), [make](), [script](), [test](), [testing](), [ubuntu]() ### AI overview The article argues that complex Buildbot configurations are difficult to test and maintain outside Buildbot. It recommends moving build and test steps into version-controlled scripts, while noting that this reduces Buildbot's structured logging visibility. ### Source excerpt The trouble with Buildbot is that it encourages you to put rules into a Buildbot-specific build configuration that is separate from the normal configuration files that you might use to build a project (configure scripts, makefiles, etc.). This is not a big problem if your Buildbot configuration is simple and just consists of, say, "svn up", "./configure", "make", "make test", and never changes. But it is a problem if your Buildbot configuration becomes non-trivial and ever has to be updated, because the Buildbot configuration cannot be tested outside of Buildbot. The last time I had to maintain a Buildbot setup, it was necessary to try out configuration changes directly on the Buildbot master. This doesn't work out well if multiple people are responsible for maintaining the setup! Whoever makes a change has to remember to check it in to version control after they've got it working, which of course doesn't always happen. It's a bit ironic that Buildbot is supposed to support automated testing but doesn't follow best practices for testing itself. There is a simple way around this though: Instead of putting those separate steps -- "./configure", "make", "make test" -- into the Buildbot config, put them into a script, check the script into version control, and have the Buildbot config run that script. Then the Buildbot config just consists of doing "svn up" and running the script. It is then possible to test changes to the script before checking it in. I've written scripts like this that go as far as debootstrapping a fresh Ubuntu chroot to run tests in, which ensures your package dependency list is up to date. Unfortunately, Buildbot's logging facilities don't encourage having a minimal Buildbot config. If you use a complicated Buildbot configuration with many Buildbot steps, Buildbot can display each step separately in its HTML-formatted logs. This means: you can see progress; you can see which steps have failed; you'd be able to see how long the steps take if Buildbo [Next page]()