# Bad Concurrency Misadventures in Concurrent and Parallel programming, plus random comments on software performance and various OSS contributions. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Aeron Adds Name Re-Resolution for Changing Peer IP Addresses DevFeed: [Aeron Adds Name Re-Resolution for Changing Peer IP Addresses]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2020-04-11T20:01:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Network](), [Cloud](), [Kubernetes](), [Protocol (disambiguation)](), [Latency]() Tags: [cloud](), [dns](), [kubernetes](), [latency](), [network](), [protocol](), [pull-request]() ### AI overview The article explains Aeron's approach to resolving endpoint names again when peer nodes disappear and return with different IP addresses, a situation that can occur in cloud and Kubernetes environments. It describes using existing protocol messages to detect missing peers and moving potentially blocking name-resolution work from sender and receiver message-path threads to the conductor. It also introduces name resolvers that can use sources beyond system DNS. ### Source excerpt Where Aeron catches up on the goss. The Need For Naming A few months ago a pull request appeared on Aeron's Github site that added the ability to request Aeron to resolve or re-resolve host names to IP addresses. In cloud environments, especially when using Kubernetes, when nodes fail and restart it is not uncommon for a node with the same host name to restart with a different IP address. Unfortunately for Aeron this could make life difficult as it would resolve IP addresses up front and stick with it for the life time of the media driver. This is particularly bad when we consider nodes that are part of Aeron Cluster, where we expect nodes to come and go from the cluster over time. It became very clear that we needed a plan that would allow Aeron to use logical names instead of IP addresses as endpoint identifiers and re-resolve those addresses appropriately. We didn't end up using the supplied pull request and came with an alternative solution that was a better fit with some of Aeron's longer term goals (I say we, it was mostly Todd Montgomery - I just did the C port). As DNS can often be a source odd network latency issues, we didn't want a name resolution solution that was entirely reliant on default system name resolution. So we have also included a mechanism for resolving names that works entirely within Aeron. Re-Resolution The first thing we needed to tackle was re-resolving IP addresses when peer nodes went away and came back with a different address. Fortunately we already have a indicators within the existing protocol that allows the media driver to detect when nodes have died. Aeron continually sends data frames or heartbeats (sender to receiver) and status messages (receiver to sender) during normal running. We can use the absence of these messages as a mechanism to detect that a node (that is identified by name rather than IP address) needs to be re-resolved. Periodically the sender and receiver will scan their endpoints to see if any having been missin ## Flow Control in Aeron DevFeed: [Flow Control in Aeron]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2020-03-19T22:20:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [networking](), [Messaging](), [Protocol (disambiguation)](), [Development]() Tags: [distributed-system](), [messaging](), [multicast](), [networking](), [protocol](), [udp]() ### AI overview This article explains flow control in Aeron, focusing on sliding-window behavior over UDP. It compares Aeron's receiver status messages with TCP acknowledgements and describes how receiver-window limits help prevent senders from overrunning receivers and causing message loss. ### Source excerpt One of my more recent projects has led me to become more involved in the Aeron project. If you are unaware of Aeron, then head over to the Github site and check it out. At its core is an reliable messaging system that works over UDP, Multicast UDP and IPC. It also contains an archiving feature for recording and replay and (still under active development) an implementation of the Raft protocol for clustering. Did I mention that it was fast too. I've spent the last few weeks buried in the various strategies the Aeron has for flow control. Specifically modifying the existing flow control strategies and adding more flexible configuration on a per channel basis. Before I jump into that it would be useful to cover a little background first. What is flow control? Within a distributed system the purpose of flow control is to limit the rate of a sender so that is does not overrun it's associated receiver. UDP does not come with any form of flow control, therefore it is easy to create a sender that will out pace the receiver, leading to message loss. There are a number of different forms of flow control, but I'm going to focus on the sliding window flow control protocol used by TCP and Aeron. The sliding window protocol requires that the sender maintain a buffer of data (referred to as a window). The size of this window will typically communicated from the receiver to the sender as part of the protocol. With a bi-directional protocol like TCP the size of the window is communicated in each TCP segment header. This is the amount of data that the sender can transmit to the receiver before having to wait until an acknowledgement is received. If the application thread on the receiver side is busy and does not read the data from the socket and the sender continues to transmit, the window size value will decrease until it reaches 0, at which time the sender must stop and wait for an acknowledgement with a non-zero window size before sending again. There is a lot more networking theo ## Resurrecting my blog after a six-year hiatus DevFeed: [Resurrecting my blog after a six-year hiatus]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2020-03-19T04:14:00Z Content type: opinion Language: en Sources: [Bad Concurrency]() Topics: [Software](), [Latency](), [Scalability](), [Open Source]() Tags: [blog](), [independent](), [latency](), [open-source](), [performance](), [scalability](), [software]() ### AI overview The author announces the return of their blog after a six-year gap, explains that they are now an independent software consultant, and signals upcoming posts about recent work, including open-source projects. ### Source excerpt If you have been a long time follower of my blog you will have noticed that is has been a really long time since I posted any new content. Since my last post six years ago at lot has changed. I no longer work at LMAX an have started out as an independent consultant. If you are part of an organisation looking for assistance in building software, especially if you facing challenges around performance (throughput, latency, scalability, or efficiency) then I might be able to help. My services and contact details are available as http://ephemeris.tech. I am hoping to start posting again about some of the work I've doing more recently and good potion of it will be open source, so there should be plenty to share. ## Investigating L3 Cache Contention in Micro-Benchmarks and Larger Applications DevFeed: [Investigating L3 Cache Contention in Micro-Benchmarks and Larger Applications]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2014-12-03T01:13:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Cache](), [Benchmark](), [cpu](), [Performance Testing](), [Concurrency]() Tags: [benchmarks](), [cache](), [cpu](), [performance](), [performance-testing](), [thread]() ### AI overview The article investigates why a LevelDB micro-benchmark reached about 400K operations per second while the same code could not sustain a 160K operations-per-second requirement in an end-to-end environment. It attributes the difference to shared L3 cache contention among concurrent threads in the larger application. ### Source excerpt While preparing for my talk at QCon SF 2014, I wanted to investigate a theory around how micro-benchmarks are not a useful reflection of how software may behave when run as part of a larger application. Specifically due contention in the last-level cache (L3* in current Intel CPUs). Read more " ## Speaking in October and November DevFeed: [Speaking in October and November]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2014-09-23T03:38:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Test automation](), [QCon Software Development Conference]() Tags: [jvm](), [performance](), [speaking](), [talks](), [test-automation]() ### AI overview The author announces talks in Auckland and at QCon San Francisco in October and November 2014, covering test automation and performance. ### Source excerpt I'll be giving some talks over the next few months: 8 Oct: Auckland Software Craftsmanship - 6 Years of test automation. 16 Oct: Auckland JVM Group - Stuff I learned about performance. 5 Nov: QCon San Francisco 2014 - Stuff I learned about performance. ## Linux Alternatives and Oracle Java DevFeed: [Linux Alternatives and Oracle Java]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2014-01-29T19:39:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [Java](), [Linux](), [openjdk](), [Fedora](), [redhat]() Tags: [command](), [configure](), [development-tools](), [java](), [jdk](), [linux](), [openjdk](), [oracle](), [setup]() ### AI overview A guide to using Linux Alternatives to switch between Oracle JDK and OpenJDK implementations. It explains the separate java and javac master configurations and shows how to select the active JRE or JDK from the command line. ### Source excerpt If, like me, you prefer to run the Oracle version of Java on your Linux machine as the default JDK, you will often find that the Linux distro will have other ideas. Fedora for example has a number of Java based applications as part of the distribution which will include a dependency on the OpenJDK. When the distro installs OpenJDK is will generally be setup as the default for executing the various Java binaries (e.g. 'java', 'javac'). However, the team at Redhat built a system called alternatives which maintains a set of symbolic links that allows the user to switch between multiple implementations of a package the supports the same functionality. I've managed to understand enough about the alternatives package that I can now easily switch between the Oracle JDK and the OpenJDK. Read more " ## An Alternative Multi-Producer Approach DevFeed: [An Alternative Multi-Producer Approach]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2013-12-15T19:42:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Concurrency](), [Concurrent Programming](), [Code]() Tags: [cpu](), [messages](), [optimisations](), [performance](), [structure](), [threads]() ### AI overview The article explores an alternative multi-producer design for the Disruptor by using one ring buffer with a SingleProducerSequencer per producer and a custom event processor that polls multiple buffers. A throughput test with three producers and one consumer suggests the approach has merit, while the author notes unresolved sequence-value and wait-strategy design issues. ### Source excerpt Recently on InfoQ, Aliaksei Papou posted an article on some of his experiments with high performance interchange of messages between threads. There were a number of examples within the article, but I am going to focus on the multi-producer case. One of the optimisations that the article showed was that if you knew the number of producers that you have at initialisation time you can build a structure that significantly reduces contention. The existing MultiProducerSequencer does not have this constraint, which is essential for a large number of use cases. However, I wanted to see what we could achieve if I applied this approach to this Disruptor. Read more " ## Fonts on Fedora Core 19 DevFeed: [Fonts on Fedora Core 19]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2013-10-10T21:13:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [Fedora](), [Font](), [Google Chrome](), [canonical](), [Unity]() Tags: [chrome](), [fonts](), [gnome](), [google-chrome](), [ubuntu](), [unity]() ### AI overview A step-by-step guide to improving font rendering on Fedora Core 19, including configuration for Google Chrome. ### Source excerpt I recently upgraded my personal workstation from Ubuntu 12.04 to Fedora Core 19. I'm very happy with the change, I prefer the standard Gnome 3 to Canonical's Unity desktop. Probably something to do with my workstation not being a tablet. However, one of the annoyances with Fedora Core has always been its font rendering. It was never as nice a default Ubuntu install. However after a couple of days of digging I've managed to figure out the magic incantation required to get all of my fonts looking good, including in Google Chrome. Read more " ## Release of Disruptor 3.0.0 DevFeed: [Release of Disruptor 3.0.0]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2013-04-11T06:22:00Z Content type: release Language: en Sources: [Bad Concurrency]() Topics: [Concurrency](), [Concurrent Programming](), [Benchmark](), [API]() Tags: [api](), [benchmark](), [cpu](), [release]() ### AI overview This release article announces Disruptor 3.0.0 and describes its main changes, including a unified MultiProducerSequencer, removal of the ClaimStrategy interface, improved single-producer throughput, and expanded EventTranslator API variants. ### Source excerpt I've decided that I'm a bit bored of the whole putting a beta tag on various versions of the Disruptor so I've decide to send forth Disruptor 3.0.0 into the world. The big challenges of this release were to clean up the code and come up with a better algorithm for handling multiple producers. If I was lucky, make it even faster. I went down a couple of dark alleys initially with this release, but have come up again for air with a version that is less different to the 2.x version, but still brings some nice benefits. I had wanted to implement a few more functional tests and improve the documentation, but I could be waiting forever to those aspects to a level where I was 100% satisfied. Read more " ## Speaking at Tech Mesh DevFeed: [Speaking at Tech Mesh]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-11-07T23:15:00Z Content type: opinion Language: en Sources: [Bad Concurrency]() Topics: [Development](), [systems]() Tags: [announce](), [challenges](), [concurrent](), [disruptor](), [hardware](), [high-performance](), [history](), [jvm](), [performance](), [perspectives](), [speaking](), [systems](), [tech](), [time]() ### AI overview The author announces an upcoming Tech Mesh talk about the Disruptor, covering its history and motivations, the challenges of building high-performance concurrent systems, and how JVM and hardware changes could support them. ### Source excerpt I'm happy to announce that I will speaking at Tech Mesh in December. I'll be speaking about the Disruptor from two perspectives, firstly looking briefly back at some of the history and motivations behind the Disruptor. Then spending some time explaining at the challenges of building high performance concurrent systems (like the Disruptor) and delving into how the JVM and hardware could change to support the development of these systems. ## Talk from JAX London DevFeed: [Talk from JAX London]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-10-19T20:53:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Concurrency]() Tags: [concurrency](), [video]() ### AI overview A blog post announces a talk on non-blocking concurrency at JAX London and says that slides and a video are available. ### Source excerpt Last week I gave a talk on non-blocking concurrency at JAX London. Here are the slides: The video is also available ## Arithmetic Overflow and Intrinsics DevFeed: [Arithmetic Overflow and Intrinsics]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-08-30T12:05:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [Java](), [JIT](), [x86](), [Assembly](), [Arm]() Tags: [arm](), [bits](), [java](), [jit](), [jvm](), [x86]() ### AI overview This article explains JVM intrinsics: recognizable code patterns that the JVM can compile into more efficient machine-specific assembly. It uses Java's Integer.bitCount() and Intel's POPCNT instruction to show how an intrinsic can replace a larger sequence of CPU instructions with one instruction, and outlines how HotSpot identifies and JIT-compiles frequently used methods. ### Source excerpt During a recent conversation on the LJC mailing list around a proposal for adding a library to the JDK that would add support for handling integer overflow a question arose. Would the JVM be able to optimise this code to make efficient use of the hardware support for overflow detection if this functionality was implemented as a library. I made the comment that this is problem is probably one best solved using intrinsics, but in the course of writing an explanation I thought it would be better explained in a blog post, so here goes... What is an Intrinsic? From the JVM perspective an intrinsic an identifiable code pattern (typically a method) where the JVM understands the intent, such that it can be complied to more optimal machine specific assembly. This is really useful when you have multiple target platforms and a subset of those targets contain instructions that may not be available on the others. E.g. Intel's X86 instruction set is quite rich when compared to a RISC-type processor, such as ARM. An Example using POPCNT One of the simplest examples of an intrinsic is the Integer.bitCount() method and the optimisation into Intel's POPCNT instruction (available on Nehalem and later), partially because it can be disabled and the effects of it not being applied are easy to observe. Lets start with some simple code that calls the Integer.bitCount() method: The implementation of the Integer.bitCount() is a reasonably complex combination of arithmetic and bit shifting in order to calculate the number of bits set to 1 within a given int1. If we run the PopCntTest class and print out the assembler generated by hotspot, we can see that this will result in quite a large number of instructions that need to be issued to the CPU. Running the class using following command line (-XX:-UsePopCountInstruction disables the intrinsic): java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:-UsePopCountInstruction PopCntTest. Generates the following assembly code: Now, lets look at ## Disruptor v3 reports more than twice the throughput in a 1P1C test DevFeed: [Disruptor v3 reports more than twice the throughput in a 1P1C test]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-07-27T20:13:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Refactoring](), [test-coverage](), [cpu](), [intel]() Tags: [coverage](), [cpu](), [intel](), [performance](), [refactoring](), [test-coverage](), [tests](), [version]() ### AI overview The article describes ongoing Disruptor v3 development, including refactoring, test coverage, and DSL work. It reports more than twice the throughput of the current release in a 1P1C test and mentions a planned multiple-producer algorithm. ### Source excerpt I've be working sporadically on the next major revision of the Disruptor, but still making steady progress. I've merged my experimental branch into the main line and I'm working on ensure comprehensive test coverage and re-implement the Disruptor DSL. As a matter of course I've been running performance tests to ensure that we don't regress performance. While I've not been focusing on performance, just some refactoring and simplification I got a nice surprise. The new version is over twice as fast for the 1P1C simple test case; approximately 85M ops/sec versus 35M ops/sec. This is on my workstation which is an Intel(R) Xeon(R) CPU E5620@2.40GHz. Current released version (2.10.1) [barkerm@snake disruptor-2]$ taskset -c 4-8,12-15 ant througput:single-test througput:single-test: [junit] Running com.lmax.disruptor.OnePublisherToOneProcessorUniCastThroughputTest [junit] Started Disruptor run 0 [junit] Disruptor=21,141,649 ops/sec [junit] Started Disruptor run 1 [junit] Disruptor=20,597,322 ops/sec [junit] Started Disruptor run 2 [junit] Disruptor=33,233,632 ops/sec [junit] Started Disruptor run 3 [junit] Disruptor=32,883,919 ops/sec [junit] Started Disruptor run 4 [junit] Disruptor=33,852,403 ops/sec [junit] Started Disruptor run 5 [junit] Disruptor=32,819,166 ops/sec [junit] Started Disruptor run 6 Current trunk. [barkerm@snake disruptor]$ taskset -c 4-8,12-15 ant througput:single-test througput:single-test: [junit] Running com.lmax.disruptor.OnePublisherToOneProcessorUniCastThroughputTest [junit] Started Disruptor run 0 [junit] Disruptor=23,288,309 ops/sec [junit] Started Disruptor run 1 [junit] Disruptor=23,573,785 ops/sec [junit] Started Disruptor run 2 [junit] Disruptor=86,805,555 ops/sec [junit] Started Disruptor run 3 [junit] Disruptor=87,183,958 ops/sec [junit] Started Disruptor run 4 [junit] Disruptor=86,956,521 ops/sec [junit] Started Disruptor run 5 [junit] Disruptor=87,260,034 ops/sec [junit] Started Disruptor run 6 [junit] Disruptor=88,261,253 ops/sec [junit] ## Disruptor 2.10 Release DevFeed: [Disruptor 2.10 Release]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-05-13T16:59:00Z Content type: release Language: en Sources: [Bad Concurrency]() Topics: [version](), [Maven](), [Maven Central](), [implementation](), [deprecated](), [generics](), [bug]() Tags: [bug](), [deprecated](), [generics](), [implementation](), [maven](), [maven-central](), [performance](), [release](), [version]() ### AI overview Disruptor 2.10 has been released. The release removes deprecated timeout methods, adds OSGI metadata and a Sequencer remainingCapacity() call, changes Sequence usage and implementation, fixes generics warnings, and reports an approximately 10% performance improvement related to an IBM JDK workaround. ### Source excerpt The Disruptor version 2.10 has been released. It is available from the Google Code download page and has been submitted to Maven central. Changes Remove deprecated timeout methods. Added OSGI metadata to jar file. Removed PaddedAtomicLong and use Sequence in all places. Fix various generics warnings. Change Sequence implementation to work around IBM JDK bug and improve performance by ~10%. Add a remainingCapacity() call to the Sequencer class. ## Speaking at QCon London DevFeed: [Speaking at QCon London]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-02-14T00:35:00Z Content type: news Language: en Sources: [Bad Concurrency]() Topics: [QCon Software Development Conference](), [Concurrency]() Tags: [concurrency](), [london](), [speaking]() ### AI overview The author announces that they will speak with Martin Thompson about non-blocking concurrency at QCon London in March. ### Source excerpt Along with Martin Thompson, I'm speaking about non-blocking concurrency at QCon London in March. ## Slides From Recent Presentations DevFeed: [Slides From Recent Presentations]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-02-10T06:39:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Concurrency]() Tags: [beginners](), [disruptor](), [guide](), [video]() ### AI overview The article shares presentation slides and videos from JAX London and Devoxx. The presentations cover hardcore concurrency for beginners and the Disruptor tool in action. ### Source excerpt My slides from JAX London - Beginner's Guide to Hardcore Concurrency: Beginners guide-concurrency View more presentations from Michael Barker Video: LJC@Playfish, JAX London My slides from Devoxx - Disruptor Tools In Action: Disruptor tools in action View more presentations from Michael Barker Video: Devoxx (Payment required) ## Building a CPU Topology on MacOS X DevFeed: [Building a CPU Topology on MacOS X]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2012-01-08T19:44:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [cpu](), [macOS](), [Assembly](), [C++](), [x86](), [Hardware](), [Kernel]() Tags: [assembly](), [c-plus-plus](), [cpu](), [hardware](), [kernel](), [macos](), [x86]() ### AI overview A tutorial on building a CPU topology on MacOS X to map operating-system processors to hardware threads, cores, and sockets. It describes using Intel's CPUID instruction with C++, x86 inline assembly, and a MacOS kernel extension. ### Source excerpt Within a project I've been working on I've had the need to simulate the capabilities of Linux's /proc/cpuinfo on Mac OS. Specifically I needed to build a topology of the CPUs on a given system. I.e. I need to map the operating system's processors to hardware threads, then build a picture of which cores and sockets those threads reside. For example my Mac looks something like: CPU0 (Thread 0) ---+ |---> Core 0 ---+ CPU1 (Thread 1) ---+ | | ----> Socket 0 CPU2 (Thread 2) ---+ | |---> Core 1 ---+ CPU3 (Thread 3) ---+ While this sounds very simple, it's actually fraught with a number of little niggles. Not only did it require getting down and dirty with a bit of C++ and X86 Assembly, it also required writing a MacOS kernel extension. The first step was to understand what information was available from the CPU. Intel exposes an instruction called CPUID. The is the primary mechanism for getting information about the CPU. There is a raft of information available from listing of the CPU features available (e.g. hyperthreading) to sizes of the various levels of cache and the associated cache lines. To access the CPUID instruction we need a little bit of inline assembler. The code shows how to get the vendor string from the CPU. On my Mac I get the following: // Output: Vendor String: GenuineIntel For those unfamiliar with Intel inline assembly, the Intel CPU defines a number of registers. The ones used for the CPUID instruction are EAX, EBX, ECX, and EDX (referenced as RAX, RBX, etc if using 64 bit instructions via the REX extension). These used for both input and output. An inline asm segment consists of 3 parts. The first part is the instruction to be executed. In this case the "cpuid" instruction. The second line defines the output parameters. The snippet "=a" (data[0]) means store the result in the EAX register in the variable data[0]. The "=a" refers to the 2nd letter of the register designation. The 3rd and final section are the input parameters. The CPUID instruction ## Concurrent Sequencing DevFeed: [Concurrent Sequencing]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2011-12-31T10:37:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Concurrency](), [Concurrent Programming](), [Java](), [cpu]() Tags: [concurrent](), [cpu](), [event](), [java](), [process](), [processors](), [sequences]() ### AI overview The article examines poor Disruptor performance under heavy contention when there are fewer available processor cores than busy threads. It attributes the problem to a busy-spin publishing loop that can starve the thread updating the cursor, and discusses Java and processor-level constraints affecting possible solutions. ### Source excerpt A few weeks ago one of the users of the Disruptor posted some worrying benchmarks: ThreePublisherToOneProcessorSequencedThroughputTest run 0: BlockingQueue=645,161 Disruptor=1,772 ops/sec run 1: BlockingQueue=1,250,000 Disruptor=20,000,000 ops/sec run 2: BlockingQueue=1,250,000 Disruptor=56 ops/sec It appears under heavy contention with fewer available cores than busy threads the Disruptor can perform terribly. After a bit of investigation I managed to isolate the problem. One of the most complex parts of the Disruptor is the multi-threaded claim strategy. It is the only place in the Disruptor where - out of necessity - we break the single-writer principal. The approach that we used was very simple. Each thread claims a slot in the ring buffer using AtomicLong.incrementAndGet(). This ensures that each claim will return a unique sequential value. The complexity arrives when the multiple threads try to publish their sequence. We require that all events placed in the ring buffer must be made available to event processors in a strictly sequential order. To ensure this behaviour we have a method called serialisePublishing(). Our simple implementation would have the thread that is publishing busy spin until the last published sequence (the cursor) is one less the value being published. This works because each sequence published is unique and strictly ascending. For example if one thread wants to publish the value 8, it will spin until the cursor value reaches 7. Because no other thread will be trying publish value 8 it can make progress and ensuring the sequential publishing behaviour in the process. However, this busy spin loop causes problems when there are more threads than cores. The threads that need wait for the prior sequences to be published can starve out the thread that should be updating the cursor. This leads to the unpredictable results shown above. We need a better solution. In an ideal world there would be a Java API that would compile down to the Intel MON ## Blog Rename and Video Links DevFeed: [Blog Rename and Video Links]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2011-12-23T07:15:00Z Content type: opinion Language: en Sources: [Bad Concurrency]() Topics: [Concurrency](), [coding]() Tags: [blog](), [concurrency](), [conferences](), [disruptor](), [links]() ### AI overview The author announces a blog rename and says future posts will focus primarily on concurrency. The article also shares links to conference videos about concurrency and the Disruptor, including a live-coding experiment that did not work well. ### Source excerpt I've decided to rename my blog. I plan to focus my blog efforts more on concurrency than any other particular subject and I thought it would be fun to pay homage to the bad science blog by Ben Goldacre. It will probably kill my traffic for a while, until I get the few blog aggregators that carry my blog to update their feeds. So welcome to my rebranded blog, I'll start posting again in the next few weeks. In the mean time here are some links to videos from the various conferences that I spoke at over the past month or 2. LJC @Playfish - Beginner's Guide to Concurrency (the trial run) JAX London - Beginner's Guide to Concurrency Devoxx - A tools in action session on the Disruptor (this was a experiment that didn't work very well, an attempt at live coding) ## Listing HotSpot VM Flags and Their Default Values DevFeed: [Listing HotSpot VM Flags and Their Default Values]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2011-08-18T09:58:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [Java]() Tags: [hotspot](), [java]() ### AI overview A short tip showing how to list HotSpot VM flags and their default values using java -XX:+PrintFlagsFinal. ### Source excerpt Ever wanted to know what flags could be set and their default values for the Hotspot VM? Try this: java -XX:+PrintFlagsFinal ## Now Offically an OpenJDK Contributor DevFeed: [Now Offically an OpenJDK Contributor]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2011-07-20T20:19:00Z Content type: opinion Language: en Sources: [Bad Concurrency]() Topics: [Java](), [openjdk](), [Code](), [Open Source](), [C#](), [Go Language]() Tags: [design-patterns](), [go](), [java](), [jvm](), [open-source](), [openjdk](), [type-safety]() ### AI overview The author argues that Java and the JVM would benefit from efficient stack-allocated value types, tuples, and related structures. These could improve type safety, enable efficient tiny types, improve cache locality, and reduce garbage-collection pressure. The author also reports that initial implementations have been accepted into the MLVM project, while noting uncertainty about adoption in the JDK or Java language. ### Source excerpt Being someone who's interested in Open Source and very focused on performance, I've been looking at the Java & the JVM and wondering about what I would most like to see added. IMHO one of the fundamental missing pieces is support for complex stack allocated types. This is not a new idea. C# calls them structs. In C, C++ and Go stack & heap allocation is orthogonal to the definition of the type. An implementation for Java is described by John Rose (some time ago) on his blog. While this is presented as a feature that would benefit alternative languages on the JVM, I think this could bring something significant to Java as well (not that I actually know what such a language feature would look like, I'm not a language designer). It would improve the efficiency of a some very useful design patterns that tend to be avoided (by me anyway) due to the cost of heap allocation. E.g. a number of classes in LMAX's code base consist of aggregations of 64 bit integers. public class OrderInstruction { public OrderInstruction(long accountId, long instrumentId, long clientOrderId, long stopPrice, long price, long quantity) { // ... } } While very efficient, it can lead to a number of errors as it is very easy to screw up an assignment wouldn't prevent assigning the accountId to the price. However, with support for tuples/value types/structs then it would be possible to do the following (I've borrowed from the C# syntax): public struct AccountId { public AccountId(long id) { // ... } } public struct InstrumentId { public InstrumentId(long id) { // ... } } public class OrderInstruction { public OrderInstruction(AccountId accountId, InstrumentId instrumentId, ClientOrderId clientOrderId, Price stopPrice, Price price, Quantity quantity) { // ... } } Given any reasonable implementation the structs Account, InstrumentId, etc, would compile down to a long (and be no less efficient). It would enforce type safety preventing accidental incorrect assignment. Another benefit is that is allows se ## Disruptor, Now Open Source! DevFeed: [Disruptor, Now Open Source!]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2011-06-22T15:07:00Z Content type: article Language: en Sources: [Bad Concurrency]() Topics: [Open Source](), [Concurrency](), [Code](), [Latency](), [Java](), [QCon Software Development Conference]() Tags: [code](), [concurrency](), [implementation](), [java](), [latency](), [open-source](), [performance](), [queue]() ### AI overview The article announces that the Disruptor, a ring-buffer-based design pattern used at the core of an LMAX financial exchange, has been open sourced. It describes the Disruptor as an alternative to a queue or a simple actor-like concurrency framework, and reports higher throughput and lower latency than Java's ArrayBlockingQueue. ### Source excerpt The Disruptor, the design pattern at the core of the financial exchange I'm helping build at LMAX has been open sourced. If you've seen the presentation that Martin Thompson and I gave at QCon or the one I did for the LJC @Skillsmatter, this is the ring-buffer based code that we've been banging on about for a while. What is the Disruptor? At its simplest, it's an alternative to a queue, but can also can be thought of as a simple actor-like concurrency framework. Why? Performance, it's about 8-9 times higher throughput and 3 orders of magnitude lower latency than Java's ArrayBlockingQueue. There's a technical article on the Google code site with more details of the implementation and comprehensive performance test results.