MLModelScope can be used to inject observers to capture and measure library calls, such as cuDNN and CUBLAS, and report their execution time to the tracer. MLModelScope can integrate with system and hardware profilers to capture GPU profiles, system traces, and hardware performance counters. The metrics are stored along with the trace information, can span multiple machines, and are viewable within the same time line as the evaluation trace. All performance counters and profilers can be enabled and disabled per user request, during \carml compilation, or globally through the configuration file. To capture CUDA events, MLModelScope uses the CUPTI library — the same library used by nvprof. Users can control which CUDA runtime functions to capture, along with whether to capture low level details such as GPU occupancy and branch divergence.

Futhremore, MLModelScope is configured to capture hardware performance counters (such as power draw, L1 cache misses, and RAM utilization) using PAPI, Intel’s power counters, and NVML. Users can add extra system metrics as meta information to tracing events.

The list of currently builtin observers includes the ability to measure memory usage, trigger Linux perf, notify MacOS’s Instruments tool, etc… Adding an observer for traces is pretty simple and requires little coding. The following is a dummy observer that does nothing:

package observer

import (
	opentracing ""

var (
	Dummy otobserver.Observer = dummy{}

type dummy struct{}

// OnStartSpan creates a new Dummy for the span
func (o dummy) OnStartSpan(sp opentracing.Span, operationName string, options opentracing.StartSpanOptions) (otobserver.SpanObserver, bool) {
	return NewSpanDummy(sp, options)

// SpanDummy collects perfevent metrics
type SpanDummy struct {
	sp opentracing.Span

// NewSpanDummy creates a new SpanDummy that can emit perfevent
// metrics
func NewSpanDummy(s opentracing.Span, opts opentracing.StartSpanOptions) (*SpanDummy, bool) {
	so := &SpanDummy{
		sp: s,
	return so, true

func (so *SpanDummy) OnSetOperationName(operationName string) {

func (so *SpanDummy) OnSetTag(key string, value interface{}) {

func (so *SpanDummy) OnFinish(options opentracing.FinishOptions) {