# Oath Published articles for Oath. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Hadoop Contributors Meetup at Oath DevFeed: [Hadoop Contributors Meetup at Oath]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-11-08T19:07:25Z Content type: news Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Developer Events](), [hdfs](), [software-development](), [Tez](), [container]() Tags: [california](), [career](), [cloudera](), [containers](), [contributors](), [hadoop](), [hdfs](), [hiring](), [low-latency](), [meetup](), [microsoft](), [oath](), [recordings](), [slides](), [software-engineering](), [tez](), [twitter](), [yahoo](), [yahoo-engineering](), [youtube]() ### AI overview Oath hosted a day-long Hadoop Contributors Meetup in Sunnyvale, bringing together more than 80 users, contributors, committers, and PMC members. Speakers from Ampool, Cloudera, Hortonworks, Microsoft, Oath, and Twitter presented challenges and solutions across the Hadoop ecosystem. Breakout sessions covered HDFS, Tez, containers, and low-latency processing. The event's presentation recordings are available in a YouTube playlist, with links to slides in each video description. ### Source excerpt yahoodevelopers: By Scott Bush, Director, Hadoop Software Engineering, Oath On Tuesday, September 25, we hosted a special day-long Hadoop Contributors Meetup at our Sunnyvale, California campus. Much of the early Hadoop development work started at Yahoo, now part of Oath, and has continued over the past decade. Our campus was the perfect setting for this meetup, as we continue to make Hadoop a priority. More than 80 Hadoop users, contributors, committers, and PMC members gathered to hear talks on key issues facing the Hadoop user community. Speakers from Ampool, Cloudera, Hortonworks, Microsoft, Oath, and Twitter detailed some of the challenges and solutions pertinent to their parts of the Hadoop ecosystem. The talks were followed by a number of parallel, birds of a feather breakout sessions to discuss HDFS, Tez, containers and low latency processing. The day ended with a reception and consensus that the event went well and should be repeated in the near future. Presentation recordings (YouTube playlist) and slides (links included in the video description) are available here: Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda Tan, Hortonworks Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara, Botong Huang "HDFS Scalability and Security", Daryn Sharp, Senior Engineer, Oath The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool Moving the Oath Grid to Docker, Eric Badger, Software Developer Engineer, Oath Vespa: Open Source Big Data Serving Engine, Jon Bratseth, Distinguished Architect, Oath Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shane Kumpf, Hortonworks How Twitter Hadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu Thank you to all the presenters and the attendees both in person and remote! P.S. We're hiring! Learn more about career opportunities at Oath. ## Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup DevFeed: [Sharing Vespa (Open Source Big Data Serving Engine) at the SF Big Analytics Meetup]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-17T17:02:27Z Content type: article Language: en Sources: [Yahoo]() Topics: [big-data](), [Open Source](), [open source communities]() Tags: [ads](), [amplitude](), [analytics](), [apache](), [big-data](), [content](), [contributions](), [data-processing](), [discovery](), [engine](), [github](), [license](), [meetup](), [mobile-phones](), [news](), [oath](), [open-source](), [personalization](), [production](), [source](), [yahoo-engineering]() ### AI overview Vespa is an open-source big data processing and serving engine developed largely by Yahoo engineers. The text says it powers products including Yahoo News, Yahoo Sports, Yahoo Finance, and Oath Ads Platforms. Zedge uses it in production for search and recommender systems that support personalized content discovery across Android, iOS, and Web, serving millions of monthly active users. Vespa is available on GitHub under a permissive Apache license. ### Source excerpt yahoodevelopers: By Jon Bratseth, Distinguished Architect, Oath I had the wonderful opportunity to present Vespa at the SF Big Analytics Meetup on September 26th, hosted by Amplitude. Several members of the Vespa team (Kim, Frode and Kristian) also attended. We all enjoyed meeting with members of the Big Analytics community to discuss how Vespa could be helpful for their companies. Thank you to Chester Chen, T.J. Bay, and Jin Hao Wan for planning the meetup, and here's our presentation, in case you missed it (slides are also available here): Largely developed by Yahoo engineers, Vespa is our big data processing and serving engine, available as open source on GitHub. It's in use by many products, such as Yahoo News, Yahoo Sports, Yahoo Finance and Oath Ads Platforms. Vespa use is growing even more rapidly; since it is open source under a permissive Apache license, Vespa can power other external third-party apps as well. A great example is Zedge, which uses Vespa for search and recommender systems to support content discovery for personalization of mobile phones (Android, iOS, and Web). Zedge uses Vespa in production to serve millions of monthly active users. Visit https://vespa.ai/ to learn more and download the code. We encourage code contributions and welcome opportunities to collaborate. ## Open-Sourcing Panoptes, Oath's distributed network telemetry collector DevFeed: [Open-Sourcing Panoptes, Oath's distributed network telemetry collector]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-04T20:59:00Z Content type: release Language: en Sources: [Yahoo]() Topics: [Network](), [Apache Kafka](), [software-development](), [Data pipelines](), [Prometheus](), [Grafana]() Tags: [collector](), [data-collection](), [data-enrichment](), [discovery](), [distributed](), [distributed-system](), [high-performance](), [industry-standards](), [influxdb](), [kafka](), [network](), [network-automation](), [network-telemetry](), [oath](), [open](), [open-source](), [python](), [redis](), [scalability](), [telemetry](), [yahoo-engineering]() ### AI overview Panoptes is a distributed network telemetry system designed to consolidate polling and normalize and enrich metrics. Written primarily in Python, it uses Kafka as a metrics bus: discovery publishes devices, polling publishes metrics, and multiple consumers can process or forward the data, including to legacy datastores. It supports SNMP and vendor APIs and is extensible to streaming standards such as gNMI. Celery with Redis schedules checks; discovery can use a supplied CSV workflow or custom CMDB adapters. The initial release includes an InfluxDB integration, which can be paired with Grafana. Oath reports running hundreds of thousands of checks per second and testing beyond one million. Its deployment uses data-center instances to proxy checks to edge sites, with metrics forwarded centrally. The authors say the open-source package can be deployed in Linux environments within hours. ### Source excerpt yahoodevelopers: By Ian Flint, Network Automation Architect and Varun Varma, Senior Principal Engineer The Oath network automation team is proud to announce that we are open-sourcing Panoptes, a distributed system for collecting, enriching and distributing network telemetry. We developed Panoptes to address several issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems. Panoptes is a pluggable, distributed, high-performance data collection system which supports multiple polling formats, including SNMP and vendor-specific APIs. It is also extensible to support emerging streaming telemetry standards including gNMI. Architecture The following block diagram shows the major components of Panoptes: Panoptes is written primarily in Python, and leverages multiple open-source technologies to provide the most value for the least development effort. At the center of Panoptes is a metrics bus implemented on Kafka. All data plane transactions flow across this bus; discovery publishes devices to the bus, polling publishes metrics to the bus, and numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems. For example, in preparing for open-source, we identified a need for a generally available time series datastore. We developed, tested and released a plugin to push metrics into InfluxDB in under a week. This flexibility allows Panoptes to evolve with industry standards. Check scheduling is accomplished using Celery, a horizontally scalable, open-source scheduler utilizing a Redis data store. Celery's scalable nature combined with Panoptes' distributed nature yields excellent scalability. Across Oath, Panoptes currently runs hundreds of thousands of checks per second, and the infrastructure has been tested ## Success at Apache: A Newbie's Narrative DevFeed: [Success at Apache: A Newbie's Narrative]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-02-05T14:43:07Z Content type: article Language: en Sources: [Yahoo]() Topics: [Tez](), [Hadoop](), [Apache Spark]() Tags: [2015](), [2017](), [apache](), [apache-software-foundation](), [asf](), [bug-fixes](), [cluster](), [code](), [community](), [contribution](), [contributor](), [dag](), [data](), [data-pipeline](), [engine](), [execution](), [frameworks](), [grid](), [hadoop](), [hdfs](), [hive](), [oath](), [open-source](), [tez](), [yahoo](), [yahoo-engineering]() ### AI overview Kuhu Shukla recounts joining Yahoo in 2015 and helping migrate its jobs from Apache MapReduce to Tez, a DAG-based engine. She learned the Hadoop ecosystem through project materials, mailing lists, a local cluster, and beginner contributions, then worked with Tez contributors through sync calls and community JIRAs to define and implement progress reporting. At Oath scale, Tez's inherited MapReduce shuffle service became a performance bottleneck, prompting a community effort to build a Tez-specific handler. After six months, the feature reached sandbox clusters; subsequent rollout and issue fixing helped complete the Tez migration and deploy the handler to research clusters. The account emphasizes shared ownership, cross-project collaboration with Hive and Pig, and the practical challenges of debugging and design. It reports roughly 100 million Tez DAGs and 50 billion tasks across nearly 38,000 nodes as of the article's timeframe. ### Source excerpt yahoodevelopers: Kuhu Shukla (bottom center) and team at the 2017 DataWorks Summit By Kuhu Shukla This post first appeared here on the Apache Software Foundation blog as part of ASF's "Success at Apache" monthly blog series. As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1. Yahoo - which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year - has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then-new DAG based execution engine. Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath's business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community. On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched ## How to Make Your Web App More Reliable and Performant Using webpack: a Yahoo Mail Case Study DevFeed: [How to Make Your Web App More Reliable and Performant Using webpack: a Yahoo Mail Case Study]() Original publisher: [Read original article]() Author: mikesefanov Published: 2017-12-13T18:27:36Z Content type: tutorial Language: en Sources: [Yahoo]() Topics: [Webpack](), [Web app](), [software-development](), [Front end](), [Page Speed Optimization]() Tags: [browser](), [bundlers](), [config](), [css](), [dependency](), [dependency-graph](), [developers](), [javascript](), [loading](), [module](), [modules](), [nodejs](), [oath](), [open-source](), [overhead](), [plugin](), [plugins](), [web-app](), [webpack](), [yahoo](), [yahoo-engineering](), [yahoo-mail]() ### AI overview Yahoo Mail's webpack migration reduced development rebuild waits from 30-60 seconds to 5 seconds, moved JavaScript bundle generation from runtime to build time to reduce bundle-fetch failures, and cut payload by about 75 KB gzipped. The team used Grunt to keep webpack running and pass environment-specific settings, including source maps, HMR, and uglification; tuned UglifyJS options saved about 10 KB. They selected existing plugins where suitable and built a custom plugin and loader for Atomic CSS, including only styles in the dependency chain. Reported outcomes also include a 50% reduction in server-side render time and a 10% improvement in launch performance measured by above-the-fold render time. These are Yahoo Mail's reported results, not guarantees for other apps. ### Source excerpt yahoodevelopers: By Murali Krishna Bachhu, Anurag Damle, and Utkarsh Shrivastava As engineers on the Yahoo Mail team at Oath, we pride ourselves on the things that matter most to developers: faster development cycles, more reliability, and better performance. Users don't necessarily see these elements, but they certainly feel the difference they make when significant improvements are made. Recently, we were able to upgrade all three of these areas at scale by adopting webpack® as Yahoo Mail's underlying module bundler, and you can do the same for your web application. What is webpack? webpack is an open source module bundler for modern JavaScript applications. When webpack processes your application, it recursively builds a dependency graph that includes every module your application needs. Then it packages all of those modules into a small number of bundles, often only one, to be loaded by the browser. webpack became our choice module bundler not only because it supports on-demand loading, multiple bundle generation, and has a relatively low runtime overhead, but also because it is better suited for web platforms and NodeJS apps and has great community support. Comparison of webpack to other open source bundlers How did we integrate webpack? Like any developer does when integrating a new module bundler, we started integrating webpack into Yahoo Mail by looking at its basic config file. We explored available default webpack plugins as well as third-party webpack plugins and then picked the plugins most suitable for our application. If we didn't find a plugin that suited a specific need, we wrote the webpack plugin ourselves (e.g., We wrote a plugin to execute Atomic CSS scripts in the latest Yahoo Mail experience in order to decrease our overall CSS payload**). During the development process for Yahoo Mail, we needed a way to make sure webpack would continuously run in the background. To make this happen, we decided to use the task runner Grunt. Not only does Grunt