Distributed Tracing: The What, Why and How

As we move further into a containerized, cloud-native landscape, what were once monolith applications are now being broken into a variety of smaller services. On one hand, this is a fantastic change, allowing engineers to quickly develop, deploy, and iterate applications. However, this has also resulted in a multitude of moving parts—all the more emphasizing the importance of the three pillars of observability: logging, metrics, and distributed tracing. Within this post, we will take a quick look at distributed tracing and go through a quick example of how you might instrument a simple golang application and leverage a tool such as Zipkin to get traces.

So exactly what is tracing? At its core, tracing refers to propagating metadata through different request calls, threads, and processes, and ultimately, constructing a directed acyclic graph (DAG) based upon this metadata. Note that by DAG, we simply mean that the “branches” of the graph (connecting the nodes) are directional, and no path in the graph can lead back to a node. Leveraging tracing, users may be able to diagnose the cause of potential latency within an application or even note slow paths between different services. But tracing has long suffered from a problem—namely, how can one reconcile instrumenting tracing when a company’s stack may consist of a multitude of third-party software, OSs, and custom applications, all in different languages? OpenTracing, a standardized tracing API, is the solution. The project provides standardization of instrumentation APIs for span (i.e. timed operation) management and inter-process propagation. As a result, users can easily switch out tracing libraries or centralized tracing systems (such as Zipkin, LightStep, etc.) with minimal configuration and headache.

So how would you potentially leverage tracing for your own application? Let’s break it down into three steps:

(A) Select the OpenTracing instrumentation library for your language of choice. This is what you will use to start and close spans within certain contexts.
(B) Select the tracing backend you would like to use. This is the centralized platform (often with a UI) to which your traces will go. This may be an open source solution, such as Zipkin, or a Software-as-a- Service product such as LightStep.
(C) Determine which tracing driver library you would like to use.

The amazing thing to note is that by using the standardized OpenTracing library, you can easily switch out which backend system and tracing library you would use.

So let’s consider a very simple golang server and client. Step one would be to select the appropriate OpenTracing library for your application. In this case, we can leverage the Go OpenTracing library. One can then instrument the server and client. Below is an example of this from the OpenTracing docs themselves:

Func makeSomeRequest (ctx context.Context) … { 
	If span := opentracing.SpanFromContext(ctx) ; span != nil { 
		httpClient := &http.Clinet{}
		httpReq, _ := http.NewRequest(“Get” , “http://myservice/”, nil) 

		// Transmit the span’s TraceContext as HTTP headers on our 
		// outbound requests. 
		opentracing.GlobalTracer().Inject( 
			span.Context(), 
			opentracing.HTTPHeaders, 
			opentracing.HTTPHeadersCarrier(httpReq.Header)) 

		resp, err := httpClient.Do(httpReq)
		…
	 }
	 …
} 

Fig. 1: How one might instrument the client

http.HandleFunc(“/”, func(w http,ResponseWriter, req *http.Request) {
	Var serverSpan opentracing.Span 
	appSpecificOperationName :=...
	wireContext, err := opentracing.GlobalTracer().Extract( 
		opentracing.HTTPHeaders, 
		opentracing.HTTPHeadersCarrier(req.Header))
	If err !=nil { 
		// Optionally record something about err here
	} 

	// Create the span referring to the RPC client if available. 
	// If wireContext == nil, a root span will be created. 
	serverSpan = opentracing.StartSpan( 
		appSpecificOperationName, 
		ext.RPCServerOption(wireContext))

	defer derverSpan.Finish()

	ctx := opentracing.ContextWithSpan(context.Background(), serverSpan) 
	…
} 

Fig. 2: How one might instrument the server

Given that we’ve decided upon using Zipkin, the global tracer should be initialized leveraging the Zipkin tracing driver library. This might be done like so:

Func main() { 
	// create collector. 
	collector, err := zipkin.NewHTTPCollector(zipkinHTTPEndpoint) 
	if err != nil { 
		fmt.Print(“unable to create ZipKin HTTP Collector: %+v/n”, err)
		os.Exit(-1)
} 

// create recorder. 
Recorder := zipkin.NewRecorder(collector, debug hostPort, ServiceName) 

// create tracer. 
Tracer, err :=zipkin.NewTracer(
	recorder, 
	zipkin.ClinetServerSameSpan(sameSpan), 
	zipkin.TraceID128Bit(traceID1288it), 
) 
If err !+ nil { 
	fmt.Printf(“unable to create Zipkin tracer: %+v\n”, err) 
	os.Exit(-1)
} 

// explicitly set our tracer to be the default tracer. 
opentracing.InitGlobalTracer(tracer)

Fig. 3. Initializing the global tracer to work with the Zipkin backend

Note that Zipkin can be containerized and deployed in something such as a Kubernetes cluster, or the binary itself can be run on a host. Zipkin is typically run with a storage backend as well, which may be in-memory storage, MySQL, Cassandra, etc.; a StorageComponent primitive is provided to configure storage with one of these options. Zipkin itself can receive spans via either HTTP request or another intermediary collector.

Now that you’ve received a basic tutorial on how to instrument a Go application and using a tracing backend such as Zipkin, what might your next steps be? For those interested in learning more, I highly suggest first running the Zipkin docker-compose file locally to get an idea of and understand the software itself. Additionally, the zipkin-go-opentracing library actually has a full-scale example of instrumenting a CLI and two services that provide a more in-depth view of properly instrumenting an application, and using Zipkin as a tracing backend. Happy learning!