# Fishing For Correlations

DevFeed: [Fishing For Correlations](<https://devfeed.tech/articles/fishing-for-correlations-19823.md>)

Original publisher: [Read original article](<https://tech.gc.com/application-performance-monitoring/>)

Author: GameChanger

Published: 2019-02-12T19:51:49Z

Content type: article

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [Application Performance Management (APM)](<https://devfeed.tech/topics/apm.md>), [Instrumentation](<https://devfeed.tech/topics/instrumentation.md>), [Koa](<https://devfeed.tech/topics/koa.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Node.js](<https://devfeed.tech/topics/node-js.md>), [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [async/await](<https://devfeed.tech/topics/async-await.md>)

Tags: [back-end](<https://devfeed.tech/tags/back-end.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [logging](<https://devfeed.tech/tags/logging.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance-monitoring](<https://devfeed.tech/tags/performance-monitoring.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>)

## AI overview

The article describes GameChanger's effort to improve observability during performance degradation in a new back-end system. It discusses instrumenting a Node.js and Koa application, using PostgreSQL, Datadog metrics and APM, and Loggly for centralized logging. The team can monitor system and request metrics but still faces challenges correlating logs, timelines, and behavioral trends.

## Source excerpt

Have you ever participated in a firefight where the root cause seems unclear, then suddenly the symptoms self-resolve? Sometimes inadvertent action may cause it, other times it appears miraculous. If you find yourself in this situation, or excusing recovery with "it was a one-off blip," your system likely lacks sufficient observability. With development and operations increasingly converging, application instrumentation continues to concern many teams building new platforms and maintaining legacy ones. The general rule of thumb when introducing observability into a system is to instrument everything. However, this philosophy creates a user experience challenge. How do we deliver the salient system correlations to the people who can best leverage them? The platform team at GameChanger recently had the opportunity to tackle this question when we encountered performance degradation while scaling a new back-end system. We are not yet finished, but we have progressed meaningfully. Diving into the black box Modern Node.js, using async/await with Koa, powers GameChanger's nascent Team Manager product. PostgreSQL serves as the primary datastore and PgBouncer manages connection pooling to the database. We already use Datadog to monitor many of our legacy systems. Starting with Datadog for a new system made sense for a number of reasons: Vendor familiarity: engineers already understand the operational model and where to look for diagnostics Breadth of functionality: support for a wide variety of metrics, application performance monitoring (APM) and centralized logging, and mature monitor types, ensure our telemetry can scale alongside the codebase Low adoption cost: integrating new services with our existing Datadog configuration is trivial After building an MVP, our baseline instrumentation included typical system level metrics, such as load average, CPU utilization, and used memory, as well as custom metrics for all requests like duration and status code. Such metrics enabl