# storm Published articles for storm. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Generating scenarios for extreme events, without extreme data DevFeed: [Generating scenarios for extreme events, without extreme data]() Original publisher: [Read original article]() Author: Jennifer Chu | MIT News Published: 2026-08-24T18:00:00Z Content type: news Language: en Sources: [MIT AI News]() Topics: [Machine learning](), [Algorithms](), [Critical Infrastructure](), [data]() Tags: [algorithm](), [algorithms](), [artificial-intelligence](), [center-for-computational-science-and-engineering](), [climate](), [climate-risk-assessment](), [computer-modeling](), [critical-infrastructure](), [data](), [extreme-event-aware](), [extreme-weather](), [fire](), [heat](), [idss](), [kai-chang](), [learning-fefb62e9fa83](), [machine-learning](), [mechanical-engineering](), [mit-meche](), [mit-schwarzman-college-of-computing](), [natural-disasters](), [research](), [risk](), [school-of-engineering](), [storm](), [sustainability](), [themis-sapsis](), [weather](), [weather-prediction]() ### AI overview MIT engineers developed a machine-learning algorithm that generates plausible future extreme-event scenarios without requiring past extreme events in the training data. It learns from available records, filters out implausible weather scenarios, and estimates events' frequency, size, intensity, duration, and area of impact to help planners prepare. ### Source excerpt A new algorithm learns to anticipate the unprecedented scenarios that critical infrastructure and global supply chains are least prepared for. ## Transformers for software engineers DevFeed: [Transformers for software engineers]() Original publisher: [Read original article]() Author: Nelson Elhage Published: 2022-04-01T20:00:00Z Content type: tutorial Language: en Sources: [Nelson Elhage]() Topics: [Transformer architecture](), [Programming](), [Software Engineering](), [Reverse Engineering](), [Transformers]() Tags: [anthropic](), [architecture](), [codex](), [deep-learning](), [gpt-3](), [model-architecture](), [programming](), [reverse-engineering](), [software-engineering](), [storm](), [transformer-architecture](), [transformers]() ### AI overview This tutorial explains Transformer architecture for software engineers, using software engineering and programming perspectives to discuss how GPT-style Transformer models work. It also connects the architecture to interpretability and reverse-engineering efforts. ### Source excerpt Ever since its introduction in the 2017 paper, Attention is All You Need, the Transformer model architecture has taken the deep-learning world by storm. Initially introduced for machine translation, it has become the tool of choice for a wide range of domains, including text, audio, video, and others. Transformers have also driven most of the massive increases in model scale and capability in the last few years. OpenAI's GPT-3 and Codex models are Transformers, as are DeepMind's Gopher models and many others. ## Analyzing Twitter Location Data with Heron, Machine Learning, Google's NLP, and BigQuery DevFeed: [Analyzing Twitter Location Data with Heron, Machine Learning, Google's NLP, and BigQuery]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2018-03-18T00:00:00Z Content type: tutorial Language: en Sources: [Christian Hollinger]() Topics: [streaming-data-processing](), [stream-processing](), [data](), [Natural language processing](), [Machine Learning & Artificial Intelligence](), [BigQuery](), [Google Cloud Platform (GCP)](), [Google]() Tags: [analytics](), [article](), [big-data](), [bigquery](), [cloud](), [data](), [distributed](), [gcp](), [google](), [google-cloud](), [hbase](), [heron](), [java](), [machine-learning](), [natural-language-processing](), [nlp](), [programming](), [storm](), [stream](), [stream-processing]() ### AI overview A tutorial describing a pipeline that processes public Twitter data with Heron, Google's NLP tools, Nominatim reverse geocoding, machine learning clustering, and Google Cloud storage and visualization services. The project maps inferred user locations and explores privacy implications. ### Source excerpt In this article, we will use Heron, the distributed stream processing and analytics engine from Twitter, together with Google's NLP toolkit, Nominatim and some Machine Learning as well as Google's BigTable, BigQuery, and Data Studio to plot Twitter user's assumed location across the US. ## Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data DevFeed: [Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-06-15T15:55:29Z Content type: release Language: en Sources: [Yahoo]() Topics: [Streaming](), [data](), [real-time](), [GitHub](), [Persistence]() Tags: [big-data](), [data](), [github](), [open-source](), [persistence](), [real-time](), [storm](), [streaming](), [yahoo](), [yahoo-engineering]() ### AI overview Yahoo announces the open sourcing of Bullet, a general-purpose query engine for streaming data. Bullet queries data as it flows through the system, supports filtering, record retrieval, aggregations, and distributions, and does not use a persistence layer. ### Source excerpt By Michael Natkovich, Akshai Sarma, Nathan Speidel, Marcus Svedman, and Cat Utah Big Data is no longer just Apache server logs. Nowadays, the data may be user engagement data, performance metrics, IoT (Internet of Things) data, or something else completely atypical. Regardless of the size of the data, or the type of querying patterns on it (exploratory, ad-hoc, periodic, long-term, etc.), everyone wants queries to be as fast as possible and cheap to run in terms of resources. Data can be broadly split into two kinds: the streaming (generally real-time) kind or the batched-up-over-a-time-interval (e.g., hourly or daily) kind. The batch version is typically easier to query since it is stored somewhere like a data warehouse that has nice SQL-like interfaces or an easy to use UI provided by tools such as Tableau, Looker, or Superset. Running arbitrary queries on streaming data quickly and cheaply though, is generally much harder... until now. Today, we are pleased to share our newly open sourced, forward-looking general purpose query engine, called Bullet, with the community on GitHub. With Bullet, you can: Powerful and nested filtering Fetching raw data records Aggregating data using Group Bys (Sum, Count, Average, etc.), Count Distincts, Top Ks Getting distributions of fields like Percentiles or Frequency histograms One of the key differences between how Bullet queries data and the standard querying paradigm is that Bullet does not store any data. In most other systems where you have a persistence layer (including in-memory storage), you are doing a look-back when you query the layer. Instead, Bullet operates on data flowing through the system after the query is started - it's a look-forward system that doesn't need persistence. On a real-time data stream, this means that Bullet is querying data after the query is submitted. This also means that Bullet does not query any data that has already passed through the stream. The fact that Bullet does not rely on a persistence ## Storm vs. Heron - Part 2 - Why Heron? A developer's view DevFeed: [Storm vs. Heron - Part 2 - Why Heron? A developer's view]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2016-12-02T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [Streaming](), [big-data](), [Development](), [Resilience](), [debugging](), [Monitoring](), [maintenance](), [Deployment](), [Server](), [Processes]() Tags: [big-data](), [debugging](), [deployment](), [developers](), [development](), [distributed](), [hadoop](), [heron](), [java](), [maintenance](), [monitoring](), [performance](), [processes](), [servers](), [spark](), [storm](), [streaming]() ### AI overview This article examines Apache Storm from a developer's perspective and explains why Twitter's Heron may be preferable for enterprise-ready distributed streaming and analytics workloads. It highlights concerns around resource allocation, performance, resilience, debugging, monitoring, deployment, and maintenance. ### Source excerpt This article is part 2 of an upcoming article series, Storm vs. Heron. ## Storm vs. Heron, Part 1: Reusing a Storm topology for Heron DevFeed: [Storm vs. Heron, Part 1: Reusing a Storm topology for Heron]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2016-10-15T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [stream-processing](), [streaming-data-processing](), [Hadoop](), [hdfs](), [Maven](), [Deployment]() Tags: [apache](), [big-data](), [configuration](), [data](), [deployment](), [github](), [hadoop](), [hdfs](), [heron](), [java](), [maven](), [processing](), [spark](), [storm](), [topology](), [vs]() ### AI overview This article examines upgrading an Apache Storm topology to Twitter Heron. Using a Twitter analysis pipeline that processes tweets and stores results in HDFS, it reports compatibility problems with the storm-hdfs library, including a missing Storm class, and discusses integrating or rewriting the library as possible workarounds. ### Source excerpt This article is part 1 of an upcoming article series, Storm vs. Heron. ## Real-time Big Data at Target DevFeed: [Real-time Big Data at Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-11-11T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Hadoop](), [stream-processing](), [streaming-data-processing](), [Kafka](), [Latency](), [Low Latency](), [Open Source]() Tags: [analytics](), [apache-kafka](), [batch](), [big-data](), [data](), [hadoop](), [kafka](), [latency](), [low-latency](), [open-source](), [real-time](), [storm](), [stream-processing](), [streaming]() ### AI overview Target's Big Data platform team describes its effort to move data into Hadoop in real time using flexible open source components. The article outlines requirements for resilience, message-format support, usability, and low latency, then evaluates Apache Storm and identifies recovery and batching-related latency issues in testing. ### Source excerpt An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener