# Tracing High Volume Services

DevFeed: [Tracing High Volume Services](<https://devfeed.tech/articles/tracing-high-volume-services-26520.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2017/09/26/tracing-services.html>)

Author: Anton Kropp was a Software Engineer

Published: 2017-09-26T12:11:36Z

Content type: article

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [tracing](<https://devfeed.tech/topics/tracing.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [Finagle](<https://devfeed.tech/topics/finagle.md>), [HTTP](<https://devfeed.tech/topics/http.md>)

Tags: [debugging](<https://devfeed.tech/tags/debugging.md>), [devops](<https://devfeed.tech/tags/devops.md>), [finagle](<https://devfeed.tech/tags/finagle.md>), [http](<https://devfeed.tech/tags/http.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logging-and-metrics](<https://devfeed.tech/tags/logging-and-metrics.md>), [okhttp](<https://devfeed.tech/tags/okhttp.md>), [scala](<https://devfeed.tech/tags/scala.md>), [services](<https://devfeed.tech/tags/services.md>), [soa](<https://devfeed.tech/tags/soa.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

## AI overview

The article explains how to trace requests across a high-volume service ecosystem by propagating a trace identifier through service calls and including it in logs. It describes Curalate's use of Finagle and custom OkHttp interceptors to place the identifier in HTTP headers, then reuse it in logging contexts and structured JSON logs.

## Source excerpt

We like to think that building a service ecosystem is like stacking building blocks. You start with a function in your code. That function is hosted in a class. That class in a service. That service is hosted in a cluster. That cluster in a region. That region in a data center, etc. At each level there's a myriad of challenges. From the start, developers tend to use things like logging and metrics to debug their systems, but a certain class of problems crops up when you need to debug across services. From a debugging perspective, you'd like to have a higher projection of the view of the system: a linearized view of what requests are doing. I.e. You want to be able to see that service A called service B and service C called service D at the granularity of single requests. Cross Service Logging The simplest solution to this is to require that every call from service to service comes with some sort of trace identifier. Incoming requests into the system, either from public API's or client side requests, or even from async daemon invoked timers/schedules/etc generates a trace. This trace then gets propagated through the entire system. If you use this trace in all your log statements you can now correlate cross service calls. How is this accomplished at Curalate? For the most part we use Finagle based services and the Twitter ecosystem has done a good job of providing the concept of a thread local TraceId and automatically propagating it to all other twitter-* components (yet another reason we like Finatra!). All of our service clients automatically pull this thread local trace id out and populate a known HTTP header field that services then pick up and re-assume. For Finagle based clients this is auto-magick'd for you. For other clients that we use, like OkHttp, we had to add custom interceptors that pulled the trace from the thread local and set it on the request. Here is an example of the header being sent automatically as part of Zipkin based headers (which we re-use