# Open-Sourcing Panoptes, Oath's distributed network telemetry collector

DevFeed: [Open-Sourcing Panoptes, Oath's distributed network telemetry collector](<https://devfeed.tech/articles/open-sourcing-panoptes-oath-s-distributed-network-telemetry-collector-20491.md>)

Original publisher: [Read original article](<https://yahooeng.tumblr.com/post/178738044351>)

Author: amberwilsonla-blog

Published: 2018-10-04T20:59:00Z

Content type: release

Language: en

Sources: [Yahoo](<https://devfeed.tech/sources/yahoo.md>)

Topics: [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Network](<https://devfeed.tech/topics/network.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Development](<https://devfeed.tech/topics/development.md>), [Python](<https://devfeed.tech/topics/python.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Redis](<https://devfeed.tech/topics/redis.md>)

Tags: [distributed-system](<https://devfeed.tech/tags/distributed-system.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [network](<https://devfeed.tech/tags/network.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [redis](<https://devfeed.tech/tags/redis.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [yahoo-engineering](<https://devfeed.tech/tags/yahoo-engineering.md>)

## AI overview

The Oath network automation team announces the open-sourcing of Panoptes, a distributed system for collecting, enriching, and distributing network telemetry. The article describes its pluggable polling and discovery architecture, Kafka-based metrics bus, Python implementation, and Celery scheduling with Redis.

## Source excerpt

yahoodevelopers: By Ian Flint, Network Automation Architect and Varun Varma, Senior Principal Engineer The Oath network automation team is proud to announce that we are open-sourcing Panoptes, a distributed system for collecting, enriching and distributing network telemetry. We developed Panoptes to address several issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems. Panoptes is a pluggable, distributed, high-performance data collection system which supports multiple polling formats, including SNMP and vendor-specific APIs. It is also extensible to support emerging streaming telemetry standards including gNMI. Architecture The following block diagram shows the major components of Panoptes: Panoptes is written primarily in Python, and leverages multiple open-source technologies to provide the most value for the least development effort. At the center of Panoptes is a metrics bus implemented on Kafka. All data plane transactions flow across this bus; discovery publishes devices to the bus, polling publishes metrics to the bus, and numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems. For example, in preparing for open-source, we identified a need for a generally available time series datastore. We developed, tested and released a plugin to push metrics into InfluxDB in under a week. This flexibility allows Panoptes to evolve with industry standards. Check scheduling is accomplished using Celery, a horizontally scalable, open-source scheduler utilizing a Redis data store. Celery's scalable nature combined with Panoptes' distributed nature yields excellent scalability. Across Oath, Panoptes currently runs hundreds of thousands of checks per second, and the infrastructure has been tested