# Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data

DevFeed: [Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data](<https://devfeed.tech/articles/open-sourcing-bullet-yahoo-s-forward-looking-query-engine-for-streaming-data-20479.md>)

Original publisher: [Read original article](<https://yahooeng.tumblr.com/post/161855616651>)

Author: mikesefanov

Published: 2017-06-15T15:55:29Z

Content type: release

Language: en

Sources: [Yahoo](<https://devfeed.tech/sources/yahoo.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [data](<https://devfeed.tech/topics/data.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [GitHub](<https://devfeed.tech/topics/github.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [data](<https://devfeed.tech/tags/data.md>), [github](<https://devfeed.tech/tags/github.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [storm](<https://devfeed.tech/tags/storm.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [yahoo](<https://devfeed.tech/tags/yahoo.md>), [yahoo-engineering](<https://devfeed.tech/tags/yahoo-engineering.md>)

## AI overview

Yahoo announces the open sourcing of Bullet, a general-purpose query engine for streaming data. Bullet queries data as it flows through the system, supports filtering, record retrieval, aggregations, and distributions, and does not use a persistence layer.

## Source excerpt

By Michael Natkovich, Akshai Sarma, Nathan Speidel, Marcus Svedman, and Cat Utah Big Data is no longer just Apache server logs. Nowadays, the data may be user engagement data, performance metrics, IoT (Internet of Things) data, or something else completely atypical. Regardless of the size of the data, or the type of querying patterns on it (exploratory, ad-hoc, periodic, long-term, etc.), everyone wants queries to be as fast as possible and cheap to run in terms of resources. Data can be broadly split into two kinds: the streaming (generally real-time) kind or the batched-up-over-a-time-interval (e.g., hourly or daily) kind. The batch version is typically easier to query since it is stored somewhere like a data warehouse that has nice SQL-like interfaces or an easy to use UI provided by tools such as Tableau, Looker, or Superset. Running arbitrary queries on streaming data quickly and cheaply though, is generally much harder... until now. Today, we are pleased to share our newly open sourced, forward-looking general purpose query engine, called Bullet, with the community on GitHub. With Bullet, you can: Powerful and nested filtering Fetching raw data records Aggregating data using Group Bys (Sum, Count, Average, etc.), Count Distincts, Top Ks Getting distributions of fields like Percentiles or Frequency histograms One of the key differences between how Bullet queries data and the standard querying paradigm is that Bullet does not store any data. In most other systems where you have a persistence layer (including in-memory storage), you are doing a look-back when you query the layer. Instead, Bullet operates on data flowing through the system after the query is started - it's a look-forward system that doesn't need persistence. On a real-time data stream, this means that Bullet is querying data after the query is submitted. This also means that Bullet does not query any data that has already passed through the stream. The fact that Bullet does not rely on a persistence