# Yahoo A peek under the purple rug! This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More DevFeed: [Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More]() Original publisher: [Read original article]() Author: rosaliebeevm-blog Published: 2019-03-08T17:12:50Z Content type: release Language: en Sources: [Yahoo]() Topics: [streaming-data-processing](), [stream-processing](), [Query (disambiguation)](), [Streaming](), [Apache Spark]() Tags: [announcements](), [big-data](), [code](), [java](), [javascript](), [open-source](), [releases](), [spark](), [stream](), [stream-processing](), [streaming](), [updates](), [yahoo-engineering]() ### AI overview This update describes new windowing support in Bullet, an open-source query system for data flowing through streaming systems. It explains time- and record-based windows, including tumbling, sliding, and hopping window patterns, for returning intermediate query results. ### Source excerpt yahoodevelopers: By Akshay Sarma, Principal Engineer, Verizon Media & Brian Xiao, Software Engineer, Verizon Media This is the first of an ongoing series of blog posts sharing releases and announcements for Bullet, an open-sourced lightweight, scalable, pluggable, multi-tenant query system. Bullet allows you to query any data flowing through a streaming system without having to store it first through its UI or API. The queries are injected into the running system and have minimal overhead. Running hundreds of queries generally fit into the overhead of just reading the streaming data. Bullet requires running an instance of its backend on your data. This backend runs on common stream processing frameworks (Storm and Spark Streaming currently supported). The data on which Bullet sits determines what it is used for. For example, our team runs an instance of Bullet on user engagement data (~1M events/sec) to let developers find their own events to validate their code that produces this data. We also use this instance to interactively explore data, throw up quick dashboards to monitor live releases, count unique users, debug issues, and more. Since open sourcing Bullet in 2017, we've been hard at work adding many new features! We'll highlight some of these here and continue sharing update posts for future releases. Windowing Bullet used to operate in a request-response fashion - you would submit a query and wait for the query to meet its termination conditions (usually duration) before receiving results. For short-lived queries, say, a few seconds, this was fine. But as we started fielding more interactive and iterative queries, waiting even a minute for results became too cumbersome. Enter windowing! Bullet now supports time and record-based windowing. With time windowing, you can break up your query into chunks of time over its duration and retrieve results for each chunk. For example, you can calculate the average of a field, and stream back results every second: In th ## Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More DevFeed: [Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-27T17:01:17Z Content type: release Language: en Sources: [Yahoo]() Topics: [Databases](), [Time Series](), [Hadoop](), [Monitoring](), [Data analysis](), [Open Source](), [Temporal data]() Tags: [data-analysis](), [databases](), [hadoop](), [monitoring](), [open-source](), [release](), [salesforce](), [time-series](), [workflows](), [yahoo-engineering]() ### AI overview OpenTSDB 2.4.0 is released with rollup and pre-aggregation storage for time series data, along with histograms and sketches. The release focuses on retaining lower-resolution data for longer periods, reducing query workloads, and supporting more accurate percentile analysis across multiple series. ### Source excerpt yahoodevelopers: By Chris Larsen, Architect OpenTSDB is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release! A few of the exciting new features include: Rollup and Pre-Aggregation Storage As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years' high fidelity data. It's more efficient to store lower resolution "rollups" for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries. Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit http://opentsdb.net/docs/build/html/user_guide/rollups.html. Histograms and Sketches When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 9 ## Vespa Product Updates, December 2018 - ONNX Import and Map Attribute Grouping DevFeed: [Vespa Product Updates, December 2018 - ONNX Import and Map Attribute Grouping]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-13T22:18:41Z Content type: release Language: en Sources: [Yahoo]() Topics: [onnx](), [AI Models](), [API](), [ci](), [Continuous integration]() Tags: [cost](), [format](), [import](), [maps](), [onnx](), [streaming](), [tensorflow](), [tensors](), [yahoo-engineering]() ### AI overview This Vespa product update covers improved Streaming Search performance, ONNX model importing and conversion to Tensors for ranking, more precise transaction log pruning during restarts, and grouping on map attribute fields. ### Source excerpt yahoodevelopers: Today we're kicking off a blog post series of need-to-know updates on Vespa, summarizing the features and fixes detailed in Github issues. We welcome your contributions and feedback about any new features or improvements you'd like to see. For December, we're excited to share the following product news: Streaming Search Performance Improvement Streaming Search is a solution for applications where each query only searches a small, statically determined subset of the corpus. In this case, Vespa searches without building reverse indexes, reducing storage cost and making writes more efficient. With the latest changes, the document type is used to further limit data scanning, resulting in lower latencies and higher throughput. Read more here. ONNX Integration ONNX is an open ecosystem for interchangeable AI models. Vespa now supports importing models in the ONNX format and transforming the models into Tensors for use in ranking. This adds to the TensorFlow import included earlier this year and allows Vespa to support many training tools. While Vespa's strength is real-time model evaluation over large datasets, to get started using single data points, try the stateless model evaluation API. Explore this integration more in Ranking with ONNX models. Precise Transaction Log Pruning Vespa is built for large applications running continuous integration and deployment. This means nodes restart often for software upgrades, and node restart time matters. A common pattern is serving while restarting hosts one by one. Vespa has optimized transaction log pruning with prepareRestart, due to flushing as much as possible before stopping, which is quicker than replaying the same data after restarting. This feature is on by default. Learn more in live upgrade and prepareRestart. Grouping on Maps Grouping is used to implement faceting. Vespa has added support to group using map attribute fields, creating a group for values whose keys match the specified key, or field valu ## Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix DevFeed: [Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-06T19:32:04Z Content type: release Language: en Sources: [Yahoo]() Topics: [phoenix](), [Transactions](), [Hadoop](), [SQL](), [Deployment](), [Cloud]() Tags: [big-data](), [cloud](), [deployment](), [hadoop](), [low-latency](), [open-source](), [phoenix](), [real-time](), [speed](), [sql](), [transactions](), [yahoo-engineering]() ### AI overview This release article introduces Omid 1.0.0, an open source transaction processing platform for Big Data, and announces its selection as the transaction management provider for Apache Phoenix. It describes the Omid Low Latency protocol, which reduces short-transaction latency by 5 times under light load and by 10 to 100 times under heavy load. ### Source excerpt yahoodevelopers: By Ohad Shacham, Yonatan Gottesman, Edward Bortnikov Scalable Systems Research, Verizon/Oath Omid, an open source transaction processing platform for Big Data, was born as a research project at Yahoo (now part of Verizon), and became an Apache Incubator project in 2015. Omid complements Apache HBase, a distributed key-value store in Apache Hadoop suite, with a capability to clip multiple operations into logically indivisible (atomic) units named transactions. This programming model has been extremely popular since the dawn of SQL databases, and has more recently become indispensable in the NoSQL world. For example, it is the centerpiece for dynamic content indexing of search and media products at Verizon, powering a web-scale content management platform since 2015. Today, we are excited to share a new chapter in Omid's history. Thanks to its scalability, reliability, and speed, Omid has been selected as transaction management provider for Apache Phoenix, a real-time converged OLTP and analytics platform for Hadoop. Phoenix provides a standard SQL interface to HBase key-value storage, which is much simpler and in many cases more performant than the native HBase API. With Phoenix, big data and machine learning developers get the best of all worlds: increased productivity coupled with high scalability. Phoenix is designed to scale to 10,000 query processing nodes in one instance and is expected to process hundreds of thousands or even millions of transactions per second (tps). It is widely used in the industry, including by Alibaba, Bloomberg, PubMatic, Salesforce, Sogou and many others. We have just released a new and significantly improved version of Omid (1.0.0), the first major release since its original launch. We have extended the system with multiple functional and performance features to power a modern SQL database technology, ready for deployment on both private and public cloud platforms. A few of the significant innovations include: Protocol ## Hadoop Contributors Meetup at Oath DevFeed: [Hadoop Contributors Meetup at Oath]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-11-08T19:07:25Z Content type: news Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Development](), [Containers](), [Latency](), [Open Source]() Tags: [big-data](), [containers](), [development](), [docker](), [hadoop](), [low-latency](), [open-source](), [yahoo-engineering]() ### AI overview Oath hosted a day-long Hadoop Contributors Meetup at its Sunnyvale campus, bringing together more than 80 Hadoop users, contributors, committers, and PMC members. Talks and breakout sessions covered challenges and solutions across the Hadoop ecosystem, including HDFS, Tez, containers, low-latency processing, deep learning, scalability, security, and cloud deployment. ### Source excerpt yahoodevelopers: By Scott Bush, Director, Hadoop Software Engineering, Oath On Tuesday, September 25, we hosted a special day-long Hadoop Contributors Meetup at our Sunnyvale, California campus. Much of the early Hadoop development work started at Yahoo, now part of Oath, and has continued over the past decade. Our campus was the perfect setting for this meetup, as we continue to make Hadoop a priority. More than 80 Hadoop users, contributors, committers, and PMC members gathered to hear talks on key issues facing the Hadoop user community. Speakers from Ampool, Cloudera, Hortonworks, Microsoft, Oath, and Twitter detailed some of the challenges and solutions pertinent to their parts of the Hadoop ecosystem. The talks were followed by a number of parallel, birds of a feather breakout sessions to discuss HDFS, Tez, containers and low latency processing. The day ended with a reception and consensus that the event went well and should be repeated in the near future. Presentation recordings (YouTube playlist) and slides (links included in the video description) are available here: Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda Tan, Hortonworks Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara, Botong Huang "HDFS Scalability and Security", Daryn Sharp, Senior Engineer, Oath The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool Moving the Oath Grid to Docker, Eric Badger, Software Developer Engineer, Oath Vespa: Open Source Big Data Serving Engine, Jon Bratseth, Distinguished Architect, Oath Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shane Kumpf, Hortonworks How Twitter Hadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu Thank you to all the presenters and the attendees both in person and remote! P.S. We're hiring! Learn more about career opportunities at Oath. ## Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup DevFeed: [Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-17T17:02:27Z Content type: article Language: en Sources: [Yahoo]() Topics: [big-data](), [Open Source](), [recommendation systems]() Tags: [github](), [meetup](), [open-source](), [recommender-systems](), [yahoo-engineering]() ### AI overview A report on a Vespa presentation at the SF Big Analytics Meetup. It describes Vespa as an open-source big data processing and serving engine and gives Zedge as an example of its use for search, recommendation systems, and mobile content discovery. ### Source excerpt yahoodevelopers: By Jon Bratseth, Distinguished Architect, Oath I had the wonderful opportunity to present Vespa at the SF Big Analytics Meetup on September 26th, hosted by Amplitude. Several members of the Vespa team (Kim, Frode and Kristian) also attended. We all enjoyed meeting with members of the Big Analytics community to discuss how Vespa could be helpful for their companies. Thank you to Chester Chen, T.J. Bay, and Jin Hao Wan for planning the meetup, and here's our presentation, in case you missed it (slides are also available here): Largely developed by Yahoo engineers, Vespa is our big data processing and serving engine, available as open source on GitHub. It's in use by many products, such as Yahoo News, Yahoo Sports, Yahoo Finance and Oath Ads Platforms. Vespa use is growing even more rapidly; since it is open source under a permissive Apache license, Vespa can power other external third-party apps as well. A great example is Zedge, which uses Vespa for search and recommender systems to support content discovery for personalization of mobile phones (Android, iOS, and Web). Zedge uses Vespa in production to serve millions of monthly active users. Visit https://vespa.ai/ to learn more and download the code. We encourage code contributions and welcome opportunities to collaborate. ## Open-Sourcing Panoptes, Oath's distributed network telemetry collector DevFeed: [Open-Sourcing Panoptes, Oath's distributed network telemetry collector]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-04T20:59:00Z Content type: release Language: en Sources: [Yahoo]() Topics: [telemetry](), [Network](), [Kafka](), [Open Source](), [Development](), [Python](), [Scalability](), [Redis]() Tags: [distributed-system](), [kafka](), [network](), [open-source](), [python](), [redis](), [scalability](), [telemetry](), [yahoo-engineering]() ### AI overview The Oath network automation team announces the open-sourcing of Panoptes, a distributed system for collecting, enriching, and distributing network telemetry. The article describes its pluggable polling and discovery architecture, Kafka-based metrics bus, Python implementation, and Celery scheduling with Redis. ### Source excerpt yahoodevelopers: By Ian Flint, Network Automation Architect and Varun Varma, Senior Principal Engineer The Oath network automation team is proud to announce that we are open-sourcing Panoptes, a distributed system for collecting, enriching and distributing network telemetry. We developed Panoptes to address several issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems. Panoptes is a pluggable, distributed, high-performance data collection system which supports multiple polling formats, including SNMP and vendor-specific APIs. It is also extensible to support emerging streaming telemetry standards including gNMI. Architecture The following block diagram shows the major components of Panoptes: Panoptes is written primarily in Python, and leverages multiple open-source technologies to provide the most value for the least development effort. At the center of Panoptes is a metrics bus implemented on Kafka. All data plane transactions flow across this bus; discovery publishes devices to the bus, polling publishes metrics to the bus, and numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems. For example, in preparing for open-source, we identified a need for a generally available time series datastore. We developed, tested and released a plugin to push metrics into InfluxDB in under a week. This flexibility allows Panoptes to evolve with industry standards. Check scheduling is accomplished using Celery, a horizontally scalable, open-source scheduler utilizing a Redis data store. Celery's scalable nature combined with Panoptes' distributed nature yields excellent scalability. Across Oath, Panoptes currently runs hundreds of thousands of checks per second, and the infrastructure has been tested ## Apache Pulsar graduates to Top-Level Project DevFeed: [Apache Pulsar graduates to Top-Level Project]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-09-25T16:31:00Z Content type: release Language: en Sources: [Yahoo]() Topics: [Apache Pulsar](), [Messaging](), [Streaming](), [Open Source](), [Cloud](), [systems]() Tags: [apache](), [data](), [distributed](), [encryption](), [hybrid-cloud](), [load-balancer](), [messaging](), [open-source](), [streaming](), [yahoo-engineering]() ### AI overview Apache Pulsar has graduated from the Apache Incubator to become a Top-Level Project. The article describes its use in Oath's streaming data pipelines and hybrid cloud strategy, along with ongoing work on messaging, load balancing, encryption, memory pressure, and connectors. ### Source excerpt yahoodevelopers: By Joe Francis, Director, Storage & Messaging We're excited to share that The Apache Software Foundation announced today that Apache Pulsar has graduated from the incubator to a Top-Level Project. Apache Pulsar is an open-source distributed pub-sub messaging system, created by Yahoo in June 2015 and submitted to the Apache Incubator in June 2017. Apache Pulsar is integral to the streaming data pipelines supporting Oath's core products including Yahoo Mail, Yahoo Finance, Yahoo Sports and Oath Ad Platforms. It handles hundreds of billions of data events each day and is an integral part of our hybrid cloud strategy. It enables us to stream data between our public and private clouds and allows data pipelines to connect across the clouds. Oath continues to support Apache Pulsar, with contributions including best-effort messaging, load balancer and end-to-end encryption. With growing data needs handled by Apache Pulsar at Oath, we're focused on reducing memory pressure in brokers and bookkeepers, and creating additional connectors to other large-scale systems. Apache Pulsar's future is bright and we're thrilled to be part of this great project and community. P.S. We're hiring! Learn more here. ## Introducing HaloDB, a fast, embedded key-value storage engine written in Java DevFeed: [Introducing HaloDB, a fast, embedded key-value storage engine written in Java]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-09-19T23:48:42Z Content type: release Language: en Sources: [Yahoo]() Topics: [Databases](), [Java](), [benchmarking](), [Open Source](), [Software]() Tags: [architecture](), [benchmarking](), [data](), [database-performance](), [embedded](), [go](), [java](), [open-source](), [performance](), [yahoo-engineering]() ### AI overview Yahoo developers introduces HaloDB, an embedded key-value storage engine written in Java. Built for an IO-bound workload with high read and write throughput, HaloDB uses append-only log files and an in-memory index. The article reports a 50% improvement in write capacity while maintaining submillisecond read latency at the 99th percentile, and announces the project's availability under the Apache license. ### Source excerpt yahoodevelopers: By Arjun Mannaly, Senior Software Engineer At Oath, multiple ad platforms use a high throughput, low latency distributed key-value database that runs in data centers all over the world. The database stores billions of records and handles millions of read and write requests per second at millisecond latencies. The data we have in this database must be persistent, and the working set is larger than what we can fit in memory. Therefore, a key component of the database performance is a fast storage engine. Our current solution had served us well, but it was primarily designed for a read-heavy workload and its write throughput started to be a bottleneck as write traffic increased. There were other additional concerns as well; it took hours to repair a corrupted DB, or iterate over and delete records. The storage engine also didn't expose enough operational metrics. The primary concern though was the write performance, which based on our projections, would have been a major obstacle for scaling the database. With these concerns in mind, we began searching for an alternative solution. We searched for a key-value storage engine capable of dealing with IO-bound workloads, with submillisecond read latencies under high read and write throughput. After concluding our research and benchmarking alternatives, we didn't find a solution that worked for our workload, thus we were inspired to build HaloDB. Now, we're glad to announce that it's also open source and available to use under the terms of the Apache license. HaloDB has given our production boxes a 50% improvement in write capacity while consistently maintaining a submillisecond read latency at the 99th percentile. Architecture HaloDB primarily consists of append-only log files on disk and an index of keys in memory. All writes are sequential writes which go to an append-only log file and the file is rolled-over once it reaches a configurable size. Older versions of records are removed to make space by a bac ## Introducing Oak: an Open Source Scalable Key-Value Map for Big Data Analytics DevFeed: [Introducing Oak: an Open Source Scalable Key-Value Map for Big Data Analytics]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-09-13T16:04:58Z Content type: article Language: en Sources: [Yahoo]() Topics: [Open Source](), [Data structures](), [big-data](), [Scalability](), [data](), [real-time](), [Java](), [Programming](), [systems]() Tags: [big-data](), [data](), [data-structures](), [java](), [open-source](), [programming](), [real-time](), [scalability](), [yahoo-engineering]() ### AI overview The article introduces Oak, an open-source Java project from Oath that implements OakMap, a concurrent ordered key-value map. It presents Oak as a tool for scalable real-time analytics workloads and describes its relationship to concurrent data structures and Druid. ### Source excerpt yahoodevelopers: By Dmitry Basin, Edward Bortnikov, Anastasia Braginsky, Eshcar Hillel, Idit Keidar, Hagar Meir, Gali Sheffi Real-time analytics applications are on the rise. Modern decision support and machine intelligence engines strive to continuously ingest large volumes of data while providing up-to-date insights with minimum delay. For example, in Flurry Analytics, an Oath service which provides mobile developers with rich tools to explore user behavior in real time, it only takes seconds to reflect the events that happened on mobile devices in its numerous dashboards. The scalability demand is immense - as of late 2017, the Flurry SDK was installed on 2.6B devices and monitored 1M+ mobile apps. Mobile data hits the Flurry backend at a huge rate, updates statistics across hundreds of dimensions, and becomes queryable immediately. Flurry harnesses the open-source distributed interactive analytics engine named Druid to ingest data and serve queries at this massive rate. In order to minimize delays before data becomes available for analysis, technologies like Druid should avoid maintaining separate systems for data ingestion and query serving, and instead strive to do both within the same system. Doing so is nontrivial since one cannot compromise on overall correctness when multiple conflicting operations execute in parallel on modern multi-core CPUs. A promising approach is using concurrent data structure (CDS) algorithms which adapt traditional data structures to multiprocessor hardware. CDS implementations are thread-safe - that is, developers can use them exactly as sequential code while maintaining strong theoretical correctness guarantees. In recent years, CDS algorithms enabled dramatic application performance scaling and became popular programming tools. For example, Java programmers can use the ConcurrentNavigableMap JDK implementations for the concurrent ordered key-value map abstraction that is instrumental in systems like Druid. Today, we are excited ## Innovating on Authentication Standards DevFeed: [Innovating on Authentication Standards]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-06-25T13:59:44Z Content type: article Language: en Sources: [Yahoo]() Topics: [Authentication](), [OAuth 2.0](), [Mobile]() Tags: [aol](), [authentication](), [identity](), [mobile](), [mobile-apps](), [oauth2](), [standards](), [yahoo](), [yahoo-engineering]() ### AI overview Yahoo and AOL describe how they used existing identity standards to address migrating identities in mobile apps from a legacy provider to the Oath identity platform. The article focuses on using OAuth 2.0 Token Exchange so apps can move between identity providers without requiring users to re-enter credentials. ### Source excerpt yahoodevelopers: By George Fletcher and Lovlesh Chhabra When Yahoo and AOL came together a year ago as a part of the new Verizon subsidiary Oath, we took on the challenge of unifying their identity platforms based on current identity standards. Identity standards have been a critical part of the Internet ecosystem over the last 20+ years. From single-sign-on and identity federation with SAML; to the newer identity protocols including OpenID Connect, OAuth2, JOSE, and SCIM (to name a few); to the explorations of "self-sovereign identity" based on distributed ledger technologies; standards have played a key role in providing a secure identity layer for the Internet. As we navigated this journey, we ran across a number of different use cases where there was either no standard or no best practice available for our varied and complicated needs. Instead of creating entirely new standards to solve our problems, we found it more productive to use existing standards in new ways. One such use case arose when we realized that we needed to migrate the identity stored in mobile apps from the legacy identity provider to the new Oath identity platform. For most browser (mobile or desktop) use cases, this doesn't present a huge problem; some DNS magic and HTTP redirects and the user will sign in at the correct endpoint. Also it's expected for users accessing services via their browser to have to sign in now and then. However, for mobile applications it's a completely different story. The normal user pattern for mobile apps is for the user to sign in (via OpenID Connect or OAuth2) and for the app to then be issued long-lived tokens (well, the refresh token is long lived) and the user never has to sign in again on the device (entering a password on the device is NOT a good experience for the user). So the issue is, how do we allow the mobile app to move from one identity provider to another without the user having to re-enter their credentials? The solution came from researching what ## Accessing Yahoo and AOL Mail Features Through IMAP DevFeed: [Accessing Yahoo and AOL Mail Features Through IMAP]() Original publisher: [Read original article]() Author: marcelatoath Published: 2018-05-18T16:19:38Z Content type: tutorial Language: en Sources: [Yahoo]() Topics: [Protocol (disambiguation)](), [servers](), [client](), [openssl]() Tags: [aol](), [developers](), [openssl](), [protocol](), [server](), [yahoo](), [yahoo-engineering]() ### AI overview This tutorial explains how developers can access Yahoo and AOL Mail metadata such as DECOS and THREADID through IMAP. It demonstrates an IMAP command sequence using openssl to connect, authenticate, list folders, select the inbox, and search for message UIDs. ### Source excerpt USE IMAP TO ACCESS SOME UNIQUE FEATURES By Libby Lin, Principal Product Manager Well, we actually won't show you how we create the magic in our big OATH consumer mail factory. But nevertheless we wanted to share how interested developers could leverage some of our unique features we offer for our Yahoo and AOL Mail customers. To drive experiences like our travel and shopping smart views or message threading, we tag qualified mails with something we call DECOS and THREADID. While we will not indulge in explaining how exactly we use them internally, we wanted to share how they can be used and accessed through IMAP. So let's just look at a sample IMAP command chain. We'll just assume that you are familiar with the IMAP protocol at this point and you know how to properly talk to an IMAP server. So here's how you would retrieve DECO and THREADIDs for specific messages: 1. CONNECT openssl s_client -crlf -connect imap.mail.yahoo.com:993 2. LOGIN a login username password a OK LOGIN completed 3. LIST FOLDERS a list "" "*" * LIST (\Junk \HasNoChildren) "/" "Bulk Mail" * LIST (\Archive \HasNoChildren) "/" "Archive" * LIST (\Drafts \HasNoChildren) "/" "Draft" * LIST (\HasNoChildren) "/" "Inbox" * LIST (\HasNoChildren) "/" "Notes" * LIST (\Sent \HasNoChildren) "/" "Sent" * LIST (\Trash \HasChildren) "/" "Trash" * LIST (\HasNoChildren) "/" "Trash/l2" * LIST (\HasChildren) "/" "test level 1" * LIST (\HasNoChildren) "/" "test level 1/nestedfolder" * LIST (\HasNoChildren) "/" "test level 1/test level 2" * LIST (\HasNoChildren) "/" "&T2BZfXso-" * LIST (\HasNoChildren) "/" "&gQKAqk7WWr12hA-" a OK LIST completed 4.SELECT FOLDER a select inbox * 94 EXISTS * 0 RECENT * OK [UIDVALIDITY 1453335194] UIDs valid * OK [UIDNEXT 40213] Predicted next UID * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk) * OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)] Permanent flags * OK [HIGHESTMODSEQ 205] a OK [READ-WRITE] SELECT completed; ## Achieving Major Stability and Performance Improvements in Yahoo Mail with a Novel Redux Architecture DevFeed: [Achieving Major Stability and Performance Improvements in Yahoo Mail with a Novel Redux Architecture]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-04-18T14:04:59Z Content type: article Language: en Sources: [Yahoo]() Topics: [Redux](), [React](), [API](), [JavaScript]() Tags: [api](), [developers](), [engineering](), [improvements](), [javascript](), [open-source](), [performance](), [react](), [redux](), [yahoo-engineering](), [yahoo-mail]() ### AI overview Yahoo Mail describes changes to its Redux architecture that improved stability, page-load performance, frame-rendering time, and API-call efficiency. The article explains how action syncers separate API interactions from action creators and components, enabling techniques such as batching and deduplication. ### Source excerpt yahoodevelopers: By Mohit Goenka, Gnanavel Shanmugam, and Lance Welsh At Yahoo Mail, we're constantly striving to upgrade our product experience. We do this not only by adding new features based on our members' feedback, but also by providing the best technical solutions to power the most engaging experiences. As such, we've recently introduced a number of novel and unique revisions to the way in which we use Redux that have resulted in significant stability and performance improvements. Developers may find our methods useful in achieving similar results in their apps. Improvements to product metrics Last year Yahoo Mail implemented a brand new architecture using Redux. Since then, we have transformed the overall architecture to reduce latencies in various operations, reduce JavaScript exceptions, and better synchronized states. As a result, the product is much faster and more stable. Stability improvements: when checking for new emails - 20% when reading emails - 30% when sending emails - 20% Performance improvements: 10% improvement in page load performance 40% improvement in frame rendering time We have also reduced API calls by approximately 20%. How we use Redux in Yahoo Mail Redux architecture is reliant on one large store that represents the application state. In a Redux cycle, action creators dispatch actions to change the state of the store. React Components then respond to those state changes. We've made some modifications on top of this architecture that are atypical in the React-Redux community. For instance, when fetching data over the network, the traditional methodology is to use Thunk middleware. Yahoo Mail fetches data over the network from our API. Thunks would create an unnecessary and undesirable dependency between the action creators and our API. If and when the API changes, the action creators must then also change. To keep these concerns separate we dispatch the action payload from the action creator to store them in the Redux state for later ## Yahoo and AOL Mail Begin Serving Images Through Secure Proxy Servers DevFeed: [Yahoo and AOL Mail Begin Serving Images Through Secure Proxy Servers]() Original publisher: [Read original article]() Author: marcelatoath Published: 2018-03-20T14:54:05Z Content type: release Language: en Sources: [Yahoo]() Topics: [Security](), [Server](), [App]() Tags: [android](), [aol](), [applications](), [browser](), [ios](), [secure-images](), [security](), [servers](), [yahoo-engineering]() ### AI overview Yahoo and AOL mail products are beginning to serve email images through secure proxy servers to improve speed and security and reduce phishing and scam risks. The change applies to desktop and mobile applications on iOS and Android, while most image-based open tracking remains supported. ### Source excerpt oath-postmaster: By Marcel Becker The mail team at OATH is busy integrating Yahoo and AOL technology to deliver an even better experience across all our consumer mail products. While privacy and security are top priority for us, we also want to improve the experience and remove unnecessary clutter across all of our products. Starting this week we will be serving images in mails via our own secure proxy servers. This will not only increase speed and security in our own mail products and reduce the risk of phishing and other scams, but it will also mean that our users don't have to fiddle around with those "enable images" settings. Messages and inline images will now just show up as originally intended. We are aware that commercial mail senders are relying on images (so-called pixels) to track delivery and open rates. Our proxy solution will continue to support most of these cases and ensure that true mail opens are recorded. For senders serving dynamic content based on the recipient's location (leveraging standard IP-based browser and app capabilities) we recommend falling back on other tools and technologies which do not rely on IP-based targeting. All of our consumer mail applications (Yahoo and AOL) will benefit from this change. This includes our desktop products as well as our mobile applications across iOS and Android. If you have any feedback or want to discuss those changes with us personally, just send us a note to mail-questions@oath.com. ## Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez DevFeed: [Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-02-05T14:43:07Z Content type: article Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Open Source](), [data](), [Apache Spark]() Tags: [2017](), [apache](), [blog](), [bug](), [data](), [distributed](), [hadoop](), [hdfs](), [oath](), [open-source](), [spark](), [tez](), [yahoo](), [yahoo-engineering]() ### AI overview Kuhu Shukla describes her experience contributing to Apache projects at Yahoo and migrating Yahoo jobs from Apache MapReduce to Apache Tez. The article also discusses Oath's Apache-based grid infrastructure and her early work with YARN, HDFS, and Hadoop. ### Source excerpt yahoodevelopers: Kuhu Shukla (bottom center) and team at the 2017 DataWorks Summit By Kuhu Shukla This post first appeared here on the Apache Software Foundation blog as part of ASF's "Success at Apache" monthly blog series. As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1. Yahoo - which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year - has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then-new DAG based execution engine. Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath's business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community. On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched ## How to Make Your Web App More Reliable and Performant Using webpack: a Yahoo Mail Case Study DevFeed: [How to Make Your Web App More Reliable and Performant Using webpack: a Yahoo Mail Case Study]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-12-13T18:27:36Z Content type: tutorial Language: en Sources: [Yahoo]() Topics: [Webpack](), [web applications](), [JavaScript](), [Development]() Tags: [case-study](), [css](), [dependency](), [developers](), [how-to](), [javascript](), [oath](), [web-app](), [webpack](), [yahoo](), [yahoo-engineering](), [yahoo-mail]() ### AI overview This Yahoo Mail case study explains how the team adopted webpack as its JavaScript module bundler to improve development cycles, reliability, and performance at scale. It describes webpack's dependency-graph and bundling model, plugin selection and development, Grunt integration, and bundle optimization before production deployment. ### Source excerpt yahoodevelopers: By Murali Krishna Bachhu, Anurag Damle, and Utkarsh Shrivastava As engineers on the Yahoo Mail team at Oath, we pride ourselves on the things that matter most to developers: faster development cycles, more reliability, and better performance. Users don't necessarily see these elements, but they certainly feel the difference they make when significant improvements are made. Recently, we were able to upgrade all three of these areas at scale by adopting webpack® as Yahoo Mail's underlying module bundler, and you can do the same for your web application. What is webpack? webpack is an open source module bundler for modern JavaScript applications. When webpack processes your application, it recursively builds a dependency graph that includes every module your application needs. Then it packages all of those modules into a small number of bundles, often only one, to be loaded by the browser. webpack became our choice module bundler not only because it supports on-demand loading, multiple bundle generation, and has a relatively low runtime overhead, but also because it is better suited for web platforms and NodeJS apps and has great community support. Comparison of webpack to other open source bundlers How did we integrate webpack? Like any developer does when integrating a new module bundler, we started integrating webpack into Yahoo Mail by looking at its basic config file. We explored available default webpack plugins as well as third-party webpack plugins and then picked the plugins most suitable for our application. If we didn't find a plugin that suited a specific need, we wrote the webpack plugin ourselves (e.g., We wrote a plugin to execute Atomic CSS scripts in the latest Yahoo Mail experience in order to decrease our overall CSS payload**). During the development process for Yahoo Mail, we needed a way to make sure webpack would continuously run in the background. To make this happen, we decided to use the task runner Grunt. Not only does Grunt ## Yahoo Mail's New Tech Stack, Built for Performance and Reliability DevFeed: [Yahoo Mail's New Tech Stack, Built for Performance and Reliability]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-06-27T15:51:49Z Content type: release Language: en Sources: [Yahoo]() Topics: [React](), [Node.js](), [Redux](), [Server-side rendering](), [Caching](), [Front end](), [Web Development](), [Routing (disambiguation)]() Tags: [cache](), [engineering](), [node-js](), [open-source](), [react](), [redux](), [routing](), [server-side-rendering](), [yahoo](), [yahoo-engineering](), [yahoo-mail]() ### AI overview Yahoo describes a rewritten desktop Yahoo Mail experience built with a front-end stack including React, Redux, Node.js, and open-source technologies. The architecture uses proximity-based routing, server-side rendering, isomorphic code, code bundling, caching, modular CSS, and open-source experimentation tools to improve loading, reliability, and payload size. ### Source excerpt By Suhas Sadanandan, Director of Engineering When it comes to performance and reliability, there is perhaps no application where this matters more than with email. Today, we announced a new Yahoo Mail experience for desktop based on a completely rewritten tech stack that embodies these fundamental considerations and more. We built the new Yahoo Mail experience using a best-in-class front-end tech stack with open source technologies including React, Redux, Node.js, react-intl (open-sourced by Yahoo), and others. A high-level architectural diagram of our stack is below. New Yahoo Mail Tech Stack In building our new tech stack, we made use of the most modern tools available in the industry to come up with the best experience for our users by optimizing the following fundamentals: Performance A key feature of the new Yahoo Mail architecture is blazing-fast initial loading (aka, launch). We introduced new network routing which sends users to their nearest geo-located email servers (proximity-based routing). This has resulted in a significant reduction in time to first byte and should be immediately noticeable to our international users in particular. We now do server-side rendering to allow our users to see their mail sooner. This change will be immediately noticeable to our low-bandwidth users. Our application is isomorphic, meaning that the same code runs on the server (using Node.js) and the client. Prior versions of Yahoo Mail had programming logic duplicated on the server and the client because we used PHP on the server and JavaScript on the client. Using efficient bundling strategies (JavaScript code is separated into application, vendor, and lazy loaded bundles) and pushing only the changed bundles during production pushes, we keep the cache hit ratio high. By using react-atomic-css, our homegrown solution for writing modular and scoped CSS in React, we get much better CSS reuse. In prior versions of Yahoo Mail, the need to run various experiments in parallel resu ## Yahoo Mail's Continuous Delivery Pipeline for Frequent, Stable Releases DevFeed: [Yahoo Mail's Continuous Delivery Pipeline for Frequent, Stable Releases]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-06-27T15:50:35Z Content type: article Language: en Sources: [Yahoo]() Topics: [CI/CD](), [Testing](), [code productivity](), [Front end](), [browsers]() Tags: [browsers](), [ci-cd](), [continuous-delivery](), [continuous-integration](), [developer](), [pipelines](), [testing](), [yahoo](), [yahoo-engineering](), [yahoo-mail]() ### AI overview An engineering article describes Yahoo Mail's continuous delivery pipeline, designed to support frequent product updates while maintaining release stability. It covers extensive unit and integration testing, staged rollout through internal, alpha, beta, and canary environments, and strategies for handling intermittent frontend test failures. ### Source excerpt By Mohit Goenka, Senior Engineering Manager Building the technology powering the best consumer email inbox in the world is no easy task. When you start on such a journey, it is important to consider how to deliver such an experience to the users. After all, any consumer feature we build can only make a difference after it is delivered to everyone via the tech pipeline. As we began building out the new version of Yahoo Mail, we wanted to ensure that our internal developer productivity would not be hindered by how our pipelines work. Keeping this in mind, we identified the following principles as most important while designing the delivery pipeline for the new Yahoo Mail experience: Product updates are pushed at regular intervals Releases are stable Builds are not blocked by irrational test failures Developers are notified of code pushes Hotfixes Rollbacks Heartbeat pushes Product updates are pushed at regular intervals We ensure that our engineers can push any code changes to all Mail users everyday, with the ability to push multiple times a day, if necessary or desired. This is possible because of the time we spent building a solid testing infrastructure, which continues to evolve as we scale to new users and add new features to the product. Every one of our builds runs 10,000+ unit tests and 5,000+ integration tests on various combinations of operating systems and browsers. It is important to push product updates regularly as it allows all our users to get the best Mail experience possible. Releases are stable Every code release starts with the company's internal audience first, where all our employees get to try out the latest changes before they go out to production. This begins with our alpha and beta environments that our Mail engineers use by default. Our build then goes out to the canary environment, which is a small subset of production users, before making it to all users. This gives us the ability to analyze quality metrics on internal and canary servers b ## Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data DevFeed: [Open Sourcing Bullet, Yahoo's Forward-Looking Query Engine for Streaming Data]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-06-15T15:55:29Z Content type: release Language: en Sources: [Yahoo]() Topics: [Streaming](), [data](), [real-time](), [GitHub](), [Persistence]() Tags: [big-data](), [data](), [github](), [open-source](), [persistence](), [real-time](), [storm](), [streaming](), [yahoo](), [yahoo-engineering]() ### AI overview Yahoo announces the open sourcing of Bullet, a general-purpose query engine for streaming data. Bullet queries data as it flows through the system, supports filtering, record retrieval, aggregations, and distributions, and does not use a persistence layer. ### Source excerpt By Michael Natkovich, Akshai Sarma, Nathan Speidel, Marcus Svedman, and Cat Utah Big Data is no longer just Apache server logs. Nowadays, the data may be user engagement data, performance metrics, IoT (Internet of Things) data, or something else completely atypical. Regardless of the size of the data, or the type of querying patterns on it (exploratory, ad-hoc, periodic, long-term, etc.), everyone wants queries to be as fast as possible and cheap to run in terms of resources. Data can be broadly split into two kinds: the streaming (generally real-time) kind or the batched-up-over-a-time-interval (e.g., hourly or daily) kind. The batch version is typically easier to query since it is stored somewhere like a data warehouse that has nice SQL-like interfaces or an easy to use UI provided by tools such as Tableau, Looker, or Superset. Running arbitrary queries on streaming data quickly and cheaply though, is generally much harder... until now. Today, we are pleased to share our newly open sourced, forward-looking general purpose query engine, called Bullet, with the community on GitHub. With Bullet, you can: Powerful and nested filtering Fetching raw data records Aggregating data using Group Bys (Sum, Count, Average, etc.), Count Distincts, Top Ks Getting distributions of fields like Percentiles or Frequency histograms One of the key differences between how Bullet queries data and the standard querying paradigm is that Bullet does not store any data. In most other systems where you have a persistence layer (including in-memory storage), you are doing a look-back when you query the layer. Instead, Bullet operates on data flowing through the system after the query is started - it's a look-forward system that doesn't need persistence. On a real-time data stream, this means that Bullet is querying data after the query is submitted. This also means that Bullet does not query any data that has already passed through the stream. The fact that Bullet does not rely on a persistence ## Open Sourcing Daytona: A Framework For Automated and Application-agnostic Performance Analysis DevFeed: [Open Sourcing Daytona: A Framework For Automated and Application-agnostic Performance Analysis]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-05-23T16:00:06Z Content type: release Language: en Sources: [Yahoo]() Topics: [Testing](), [benchmarking](), [Framework](), [Open Source](), [Tool](), [ui](), [Command-line interface](), [CI/CD](), [Cloud](), [on-prem]() Tags: [architecture](), [ci-cd](), [cloud](), [cloud-services](), [command-line](), [data](), [framework](), [metrics](), [on-prem](), [open-source](), [performance](), [performance-analysis](), [performance-testing](), [systems-engineering](), [tool](), [ui](), [yahoo](), [yahoo-engineering]() ### AI overview Yahoo presents Daytona as an open-source, application-agnostic framework for automated performance testing and analysis. It provides repeatable test execution, standardized reporting, profiling, a unified interface, command-line script support, scheduling, CI/CD integration, and deployment on-premises or in the cloud. ### Source excerpt By Sapan Panigrahi and Deepesh Mittal Today, we are pleased to offer Daytona, an open-source framework for automated performance testing and analysis, to the community. Daytona is an application-agnostic framework to conduct integrated performance testing and analysis with repeatable test execution, standardized reporting, and built-in profiling support. Daytona gives you the capability to build a customized test harness in a single, unified framework to test and analyze the performance of any application. You'll get easy repeatability, consistent reporting, and the ability to capture trends. Daytona's UI accepts a performance testing script that can run on a command line. This includes websites, databases, networks, or any workload you need to test and tune for performance. You can submit tests to the scheduler queue from the Daytona UI or from your CI/CD tool. You can deploy Daytona as a hosted service in your on-prem environment or on the public cloud of your choice. In fact, you can even host test harnesses for multiple applications with a single centralized service so that developers, architects, and systems engineers from different parts of your organization can work together on a unified view and manage your performance analysis on a continuous basis. Daytona's differentiation lies in its ability to aggregate and present essential aspects of application, system, and hardware performance metrics with a simple and unified user interface. This helps you maintain your focus on performance analysis without changing context across various sources and formats of data. The overall goal of performance analysis is to find ways of maximizing application throughput with minimum hardware resource and the best user experience. Metrics and insights from Daytona help achieve this objective. Prior to Daytona, we created multiple, heterogenous performance tools to meet the specific needs of various applications. This meant that we often stored test results inconsistently, maki