blog に戻る

2020年06月16日 Theo Despoudis

Tracing Tools Compared: Jaeger vs. OpenTracing

With the advent of microservices, technologies like Docker, Kubernetes and services like Cloud Computing, have showcased the broader need for observability. Collecting valuable information about the communication endpoints and how they propagate through the discrete components of the application stack is the key to understanding when, why and what happens in case of failure.

The previous process is called Distributed Tracing. To grasp what the fuss is all about, we will explore two excellent tools in the tracing scene: Jaeger and OpenTracing. We will see an example usage of both of them and compare their differences. We will also learn how to leverage both in the same program as you will find that Jaeger offers native support for the OpenTracing Standard.

Let’s get started...

Jaeger

Jaeger was open sourced by Uber Technologies and donated to CNCF, so it lives and breathes within scalable distributed systems. It consists of several components as depicted in the diagram below.

There is the jaeger-client, which runs as a sidecar process next to the application container, and samples the runtime operations. They essentially perform all the instrumentation logic, and they adhere to the OpenTracing specification.

There is the jaeger-agent, which sits next to the network interface of the container or the host, and performs efficient and transparent batch processing of the payload as it’s collected from the various jaeger-clients. The data are sent to the jaeger-collector process.

Next, there is the jaeger-collector, which implements the persistence logic from the various jaeger-agents, and it supports several storage backends.

Lastly is jaeger-query, which is responsible for extracting and providing the tracing information to the UI layer.

Source: jaegertracing.com

As you can see on the above architecture, Jaeger has abstracted all the individual components of the observability pipeline so that it’s easier to scale.

Jaeger is implemented in Go, so it’s fairly portable and memory efficient. It offers several deployment options, such as via a Kubernetes Operator – an all-in-one binary – as a Windows Service, or via Docker.

Let’s see an example where we run all the Jaeger components in a sample application using Docker.

First we need to run the backend and UI sections using the all-in-one docker image:

$ docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \

This docker image contains the agent, the collector and the query components that handle all the tracing persistence and retrieval logic.

Now if we visit the following link: http://localhost:16686 we will see the main dashboard page:

Next we need an application that has enabled instrumentation via the jaeger-client that will feed the UI with tracing information. Luckily for us, there is an example docker image ready to use. We only have to deploy it:

$ docker run --rm -it \
  --link jaeger \
  -p8080-8083:8080-8083 \
  -e JAEGER_AGENT_HOST="jaeger" \
  jaegertracing/example-hotrod:latest \
  all

Then navigate to http://localhost:8080:

Notice that if you refresh the browser, you get a new client id; and clicking on each button will trigger a different button consecutively, which will queue all the requests. Then when they are all completed, you will get some information about the latency:

Now switch to the Jaeger UI and try to search for some transactions:

For every observable component or application, there is a complete trail of requests and response information. We can delve into details, for example, when we click on a Trace:

Each Trace Line will expand to a list of Spans, which are the individual units of work done to capture the tracing information.

We can even delve into each Span and see the detail and logs for each operation. For example, here is the breakdown of the MYSQL SELECT statement:

Overall, Jaeger is a valuable tracing tool when rolled out properly in the production infrastructure. For more information you can visit the official site: https://www.jaegertracing.io/

OpenTracing

If Jaeger implements the OpenTracing Specification, then what is the difference? We ask that because by looking at the example code in the official website, we can see it uses Jaeger. However, there lies the key difference.

OpenTracing is an open distributed tracing vendor neutral standard (not official) for applications. Jaeger is one implementation of it. But there are more tracer implementations like Elastic APM, Datadog and more. Their key differences are their level of support and desired features. Jaeger is free, but you get only community support. Datadog is commercial, so you get Enterprise support.

This level of interoperability allows developers and SREs to switch implementation when something better exists to satisfy the needs of the business.

The official documentation should be the first point of reference when someone is interested in learning more about Distributed Tracing. Most of the examples there use Jaeger as a client, as it’s part of the same CNCF family.

In fact, the default github repo of OpenTracing contains only reference implementations – they map to Noops so the Spans will not capture any information if you want to use compatible tracers like Jaeger.

Let’s see an example using the node client. You will need Node v10+ for the following example.

In a new folder, create a new npm project and install opentracing-js:

$ npm init -y
$ npm install opentracing --save

Then create a new file index.js and add the following code:

const MockTracer = require('opentracing').MockTracer;
const tracer = new MockTracer();

const parent = tracer.startSpan('parent_span');
parent.setTag('custom', 'tag value');
parent.setTag('id', '1000');

setTimeout(() => {
  console.log('Starting child span.');
  const child = tracer.startSpan('child_span', { childOf: parent });
  child.setTag('a', '200');
  child.setTag('b', '50');
  child.log({state: 'waiting'});

  console.log('Waiting...');
  setTimeout(() => {
    console.log('Finishing child and parent.');
    child.log({state: 'done'});
    child.finish();
    parent.finish();

    // Print some information about the two spans. Note the `report` method
    // is specific to the MockTracer implementation and is not part of the
    // OpenTracing API.
    console.log('\nSpans:');
    const report = tracer.report();
    for (const span of report.spans) {
      const tags = span.tags();
      const tagKeys = Object.keys(tags);

      console.log(`    ${span.operationName()} - ${span.durationMs()}ms`);
      for (const key of tagKeys) {
        const value = tags[key];
        console.log(`        tag '${key}':'${value}'`);
      }
    }
  }, 500);
}, 1000);

The example above creates a Mock Tracer and records some spans in the console. Now run the code and you will see the following log:

$ node index.js
Starting child span.
Waiting...
Finishing child and parent.

Spans:
    parent_span - 1513ms
        tag 'custom':'tag value'
        tag 'id':'1000'
    child_span - 505ms
        tag 'a':'200'
        tag 'b':'50'

As you can see from the example, the Mock implementation is suitable for understanding how tracing works and what needs to be added in the code to instrument the application. Once you are ready, you can replace the MockTracer with Jaeger and continue as before.

Next Steps

Both Jaeger and Open Tracing are widely used and are part of the CNCF suite. That means they will have continuous support and development for the time being.

Those technologies together unlock all the observability benefits such as easier root-cause analysis, distributed monitoring, and useful insights for performance and optimization. They have great interoperability allowing more seamless transition or interchange in case there is a change in requirements. This flexibility allows SREs and DevOps teams to be on top of potent incidents and disruptions.

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.

Navigate Kubernetes with Sumo Logic.

Build, run, and secure modern applications and cloud infrastructures.

Start free trial
Theo Despoudis

Theo Despoudis

Senior Software Engineer

Theo Despoudis is a Senior Software Engineer, a consultant and an experienced mentor. He has a keen interest in Open Source Architectures, Cloud Computing, best practices and functional programming. He occasionally blogs on several publishing platforms and enjoys creating projects from inspiration. He can be contacted via http://www.techway.io/.

More posts by Theo Despoudis.

これを読んだ人も楽しんでいます