# Adrian Colyer a random walk through Computer Science research, by Adrian Colyer This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The ants and the pheromones DevFeed: [The ants and the pheromones]() Original publisher: [Read original article]() Author: adriancolyer Published: 2021-02-08T14:46:00Z Content type: opinion Language: en Sources: [Adrian Colyer]() Topics: [Algorithms](), [Computer science](), [Learning]() Tags: [algorithms](), [bbc](), [computer-science](), [covid](), [learn](), [podcast](), [uncategorized]() ### AI overview The final edition of The Morning Paper reflects on how ant pheromone trails and reward equations can illustrate the disruption of established solutions. It also discusses how Covid lockdowns disrupted the author's routines and may lead to different patterns of life and work, while the author considers broadening their studies beyond computer science. ### Source excerpt TLDR; this is the last edition of The Morning Paper for now. Plus: one strand of research you won't want to miss! I was listening to a BBC Radio 4 podcast recently (More or Less: Behind the Stats - Ants and Algorithms) in which the host Tim Harford is interviewing David Sumpter about his recent ... Continue reading The ants and the pheromones ## An overview of end-to-end entity resolution for big data DevFeed: [An overview of end-to-end entity resolution for big data]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-12-14T14:37:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Entity resolution](), [big-data](), [data]() Tags: [algorithms-and-data-structures](), [article](), [big-data](), [blocking](), [clustering](), [comparisons](), [overview](), [uncategorized]() ### AI overview This article summarizes an ACM Computing Surveys survey on end-to-end entity resolution for big data. It explains the main pipeline stages: blocking to reduce comparisons, block processing to remove redundant and superfluous comparisons, matching entity-description pairs, and clustering matches into resolved entities. It also outlines classification dimensions including schema awareness, matching process, and batch or incremental processing. ### Source excerpt An overview of end-to-end entity resolution for big data, Christophides et al., ACM Computing Surveys, Dec. 2020, Article No. 127 The ACM Computing Surveys are always a great way to get a quick orientation in a new subject area, and hot off the press is this survey on the entity resolution (aka record linking) problem. It's an ... Continue reading An overview of end-to-end entity resolution for big data ## Bias in word embeddings DevFeed: [Bias in word embeddings]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-12-08T14:32:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Embeddings](), [Machine learning](), [Algorithms, Complexity]() Tags: [algorithm](), [bias](), [dataset](), [embeddings](), [machine-learning](), [model](), [train](), [uncategorized](), [word-embeddings]() ### AI overview This article summarizes research on bias in word embeddings, explaining how bias in training text can be encoded in embeddings, transferred to later algorithms, and produce socially discriminatory decisions. It also discusses detecting, measuring, and mitigating that bias. ### Source excerpt Bias in word embeddings, Papakyriakopoulos et al., FAT*'20 There are no (stochastic) parrots in this paper, but it does examine bias in word embeddings, and how that bias carries forward into models that are trained using them. There are definitely some dangers to be aware of here, but also some cause for hope as we ... Continue reading Bias in word embeddings ## Seeing is believing: a client-centric specification of database isolation DevFeed: [Seeing is believing: a client-centric specification of database isolation]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-11-30T14:08:00Z Content type: opinion Language: en Sources: [Adrian Colyer]() Topics: [Database](), [systems]() Tags: [database](), [datastores](), [paper](), [state](), [storage](), [uncategorized](), [visibility]() ### AI overview This article reviews "Seeing is believing," a paper that proposes defining database isolation guarantees in terms of effects visible to clients. Its state-based formalization treats storage systems as black boxes and aims to clarify observable behavior for application developers while preserving implementation freedom. ### Source excerpt Seeing is believing: a client-centric specification of database isolation, Crooks et al., PODC'17. Last week we looked at Elle, which detects isolation anomalies by setting things up so that the inner workings of the database, in the form of the direct serialization graph (DSG), can be externally recovered. Today's paper choice, 'Seeing is believing' also deals ... Continue reading Seeing is believing: a client-centric specification of database isolation ## Elle: inferring isolation anomalies from experimental observations DevFeed: [Elle: inferring isolation anomalies from experimental observations]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-11-23T13:56:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Databases](), [Transactions](), [Concurrency](), [test](), [SQL](), [graph-database](), [NoSQL]() Tags: [concurrency](), [database](), [evaluation](), [graph-database](), [sql](), [systems](), [test](), [transactions](), [uncategorized]() ### AI overview This article discusses the Elle checker from the Jepsen project, which infers database isolation anomalies from experimental execution histories. Elle can check histories involving hundreds of thousands of transactions in tens of seconds and was evaluated on SQL, document, and graph databases, where it found unexpected anomalies. The article notes that Elle supports many anomaly types and reproducible bug reports but does not guarantee completeness. ### Source excerpt Elle: inferring isolation anomalies from experimental observations, Kingsbury & Alvaro, VLDB'20 Is there anything more terrifying, and at the same time more useful, to a database vendor than Kyle Kingsbury's Jepsen? As the abstract to today's paper choice wryly puts it, "experience shows that many databases do not provide the isolation guarantees they claim." Jepsen captures ... Continue reading Elle: inferring isolation anomalies from experimental observations ## Achieving 100Gbps intrusion prevention on a single server DevFeed: [Achieving 100Gbps intrusion prevention on a single server]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-11-16T13:42:00Z Content type: opinion Language: en Sources: [Adrian Colyer]() Topics: [servers](), [Scalability](), [Architecture & Design](), [Network](), [Latency](), [Low Latency](), [systems]() Tags: [capacity](), [efficiency](), [hardware](), [latency](), [low-latency](), [network](), [networking](), [protocol](), [requirements](), [scalability](), [server](), [systems]() ### AI overview This article discusses Pigasus, an intrusion detection and prevention system designed to achieve 100 Gbps intrusion prevention on a single server. It explains the system's focus on high throughput and low latency, and highlights an architecture in which an FPGA coordinates processing while the CPU provides support. ### Source excerpt Achieving 100 Gbps intrusion prevention on a single server, Zhao et al., OSDI'20 Papers-we-love is hosting a mini-event this Wednesday (18th) where I'll be leading a panel discussion including one of the authors of today's paper choice: Justine Sherry. Please do join us if you can. We always want more! This stems from a combination of Jevon's paradox ... Continue reading Achieving 100Gbps intrusion prevention on a single server ## Virtual consensus in Delos DevFeed: [Virtual consensus in Delos]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-11-09T13:34:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [distributed-systems](), [consistency](), [Protocol (disambiguation)](), [Availability](), [upgrade](), [maintenance]() Tags: [availability](), [consensus](), [consistency](), [distributed-systems](), [durability](), [implementation](), [maintenance](), [protocol](), [uncategorized](), [upgrade]() ### AI overview This article discusses Delos, a system designed for Facebook's control-plane services. It describes an API-based abstraction over consensus using a shared log, with a runtime that supports hot-swapping consensus implementations so the system can be upgraded without interrupting operation. ### Source excerpt Virtual consensus in Delos, Balakrishnan et al. (Facebook, Inc.), OSDI'2020 Before we dive into this paper, if you click on the link above and then download and open up the paper pdf you might notice the familiar red/orange splash of USENIX, and appreciate the fully open access. USENIX is a nonprofit organisation committed to making content and ... Continue reading Virtual consensus in Delos ## Helios: hyperscale indexing for the cloud & edge (part II) DevFeed: [Helios: hyperscale indexing for the cloud & edge (part II)]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-11-02T13:20:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Cloud](), [data-processing](), [data](), [Microsoft](), [real-time](), [Edge]() Tags: [azure](), [cloud](), [data-processing](), [microsoft](), [monitoring](), [parsing](), [process](), [time](), [uncategorized]() ### AI overview This article explains Helios, a distributed system used at Microsoft for ingesting, indexing, and aggregating large streams of real-time data. It describes Helios's separation of ingestion and indexing, bottoms-up index construction, relational query-engine integration, and edge computation. The article also outlines use cases such as incident investigation, performance monitoring, reporting, and GDPR deletion requests, along with its stream-table and agent-based design. ### Source excerpt Helios: hyperscale indexing for the cloud & edge, Potharaju et al., PVLDB'20 Last time out we looked at the motivations for a new reference blueprint for large-scale data processing, as embodied by Helios. Today we're going to dive into the details of Helios itself. As a reminder: Helios is a distributed, highly-scalable system used at Microsoft for ... Continue reading Helios: hyperscale indexing for the cloud & edge (part II) ## Helios: hyperscale indexing for the cloud & edge - part 1 DevFeed: [Helios: hyperscale indexing for the cloud & edge - part 1]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-10-26T20:22:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [data-processing](), [big-data](), [Cloud](), [distributed-systems](), [data](), [Edge](), [Microsoft](), [client](), [servers]() Tags: [architecture](), [big-data](), [cloud](), [data](), [data-processing](), [datastores](), [distributed](), [distributed-computing](), [edge](), [idc](), [latency](), [microsoft](), [paper](), [real-time](), [server](), [streams](), [systems](), [uncategorized]() ### AI overview This first part of a discussion of Helios examines its role as a reference architecture for distributed big-data systems. It describes the pressures driving cloud platforms toward new approaches, including growing data volume, variable arrival rates, lower latency, reduced data-center costs, and stronger privacy considerations. ### Source excerpt Helios: hyperscale indexing for the cloud & edge, Potharaju et al., PVLDB'20 On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, ... Continue reading Helios: hyperscale indexing for the cloud & edge - part 1 ## The case for a learned sorting algorithm DevFeed: [The case for a learned sorting algorithm]() Original publisher: [Read original article]() Author: adriancolyer Published: 2020-10-19T19:11:00Z Content type: article Language: en Sources: [Adrian Colyer]() Topics: [Sorting](), [Algorithm](), [Machine learning](), [systems](), [Computer science]() Tags: [algorithm](), [arrays](), [computer-science](), [machine-learning](), [ml](), [sorting](), [systems](), [uncategorized]() ### AI overview This article discusses Learned Sort, a sorting method that uses a model to approximate the cumulative distribution function and place items near their sorted positions before finishing with Insertion Sort. On a 1 billion item dataset, it reportedly outperformed RadixSort by a factor of 1.49x, including model-training time. ### Source excerpt The case for a learned sorting algorithm, Kristo, Vaidya, et al., SIGMOD'20 We've watched machine learning thoroughly pervade the web giants, make serious headway in large consumer companies, and begin its push into the traditional enterprise. ML, then, is rapidly becoming an integral part of how we build applications of all shapes and sizes. But what about systems ... Continue reading The case for a learned sorting algorithm