What is observability?

Simply put: observability is how well you can understand the inner workings and behaviour of your application given a situation. There’s many ways you can increase observability, and it can encompass a broad range of actions:

• Logging messages (for example, model provider inputs/outputs)
• Using println! to debug an object
• Collecting the metrics of your application to compare in the future
• Using a platform like Grafana to collect and analyse logs, metrics and traces

Rig primarily uses tracing for traces and spans.

How observable is Rig?

Rig aims to be fully compatible with OpenTelemetry GenAI Semantic Conventions. This allows you to use a wide range of backends that are compatible with the aforementioned conventioned, such as:

• Langfuse
• Arize Phoenix
• and more!

More specifically, we support instrumenting completions (whether streamed or not) with your own spans as well as providing our own by default.

We also have full support for providing tracing spans for Agents, whether you are using regular multi-turn prompting or streamed. Due to limitations in tracing being unable to change span names, the default span names have been set to the operation name. However, you can change this in your OTel collector. See Setting up your OTel collector config for more information.

Currently, content capturing is enabled by default - so whatever observability backend you are using will be able to see all message contents.

Please reach out to us if you would like any feature additions or changes when it comes to observability as it is still relatively experimental. You can do so by joining our discord.

Setting up your OTel collector config

When setting up your OpenTelemetry config, you may find the following example helpful as a reference:

receivers:
  otlp:
    protocols:
      http:
        # this is the default endpoint
        endpoint: 0.0.0.0:4318
 
processors:
  transform:
    trace_statements:
      - context: span
        statements:
          # Rename span if it's "invoke_agent" and has an agent attribute
          - set(name, attributes["gen_ai.agent.name"]) where name == "invoke_agent" and attributes["gen_ai.agent.name"] != nil
 
exporters:
  otlphttp/langfuse:
    endpoint: "https://cloud.langfuse.com/api/public/otel"
    headers:
      Authorization: "Basic ${AUTH_STRING}"
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform]
      exporters: [otlphttp/langfuse]

You can see we do the following:

• Set up an endpoint to collect traces from a HTTP endpoint at port 4318
• Transform span names so that any invoke_agent spans (where an agent multi-turn prompt starts) instead change to the name of the Agent being ran
• Exports transformed traces/spans to Langfuse

I don’t want spans!

If you just want to see logs instead of entire spans, you can do so by writing your own subscriber layer (assuming you’re using tracing_subscriber).

Below is an example of a layer that only outputs the message itself without any of the span fields or metadata.

#[derive(Clone)]
struct MessageOnlyLayer;
 
impl<S> Layer<S> for MessageOnlyLayer
where
    S: Subscriber + for<'a> LookupSpan<'a>,
{
    fn on_event(&self, event: &tracing::Event<'_>, _ctx: Context<'_, S>) {
        use tracing::field::{Field, Visit};
 
        struct MessageVisitor {
            message: Option<String>,
        }
 
        impl Visit for MessageVisitor {
            fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
                if field.name() == "message" {
                    self.message = Some(format!("{:?}", value));
                }
            }
        }
 
        let mut visitor = MessageVisitor { message: None };
        event.record(&mut visitor);
 
        if let Some(msg) = visitor.message {
            let msg = msg.trim_matches('"');
            let metadata = event.metadata();
 
            let colored_level = match metadata.level() {
                &tracing::Level::TRACE => "\x1b[35mTRACE\x1b[0m", // Purple
                &tracing::Level::DEBUG => "\x1b[34mDEBUG\x1b[0m", // Blue
                &tracing::Level::INFO => "\x1b[32m INFO\x1b[0m",  // Green
                &tracing::Level::WARN => "\x1b[33m WARN\x1b[0m",  // Yellow
                &tracing::Level::ERROR => "\x1b[31mERROR\x1b[0m", // Red
            };
            let _ = writeln!(std::io::stdout(), "{colored_level} {msg}");
        }
    }
}

To use, you would ideally place it after an EnvFilter like so:

tracing_subscriber::registry()
    .with(EnvFilter::new("info"))
    .with(MessageOnlyLayer)
    .init();

Troubleshooting

• If your tool runs too quickly (less than 1ms), your spans may not run in order (the resulting completion might appear before the tool is used, but they will have the same timestamp on your observability backend). This should not often be a problem in production however, as production workloads tend to require enough compute to solve this problem by itself.