Orleans observability

Metrics

Metrics are numerical measurements reported over time. You most often use them to monitor an application's health and generate alerts. For more information, see Metrics in .NET. Orleans uses the System.Diagnostics.Metrics APIs to collect metrics. These metrics are exposed to the OpenTelemetry project, which exports them to various monitoring systems.

To monitor your app without making any code changes, use the dotnet counters .NET diagnostic tool. To monitor Orleans ActivitySource counters for a specific <ProcessName>, use the dotnet counters monitor command as shown:

dotnet counters monitor -n <ProcessName> --counters Microsoft.Orleans

Imagine you're running the Orleans GPS Tracker sample app and monitoring it in a separate terminal with the dotnet counters monitor command. The following output is typical:

Press p to pause, r to resume, q to quit.
    Status: Running
[Microsoft.Orleans]
    orleans-app-requests-latency-bucket (Count / 1 sec)                    0
        duration=10000ms                                                   0
        duration=1000ms                                                    0
        duration=100ms                                                     0
        duration=10ms                                                      0
        duration=15000ms                                                   0
        duration=1500ms                                                    0
        duration=1ms                                                   2,530
        duration=2000ms                                                    0
        duration=200ms                                                     0
        duration=2ms                                                       0
        duration=400ms                                                     0
        duration=4ms                                                       0
        duration=5000ms                                                    0
        duration=50ms                                                      0
        duration=6ms                                                       0
        duration=800ms                                                     0
        duration=8ms                                                       0
        duration=9223372036854775807ms                                     0
    orleans-app-requests-latency-count (Count / 1 sec)                 2,530
    orleans-app-requests-latency-sum (Count / 1 sec)                       0
    orleans-catalog-activation-working-set                                36
    orleans-catalog-activations                                           38
    orleans-consistent-ring-range-percentage-average                     100
    orleans-consistent-ring-range-percentage-local                       100
    orleans-consistent-ring-size                                           1
    orleans-directory-cache-size                                          27
    orleans-directory-partition-size                                      26
    orleans-directory-ring-local-portion-average-percentage              100
    orleans-directory-ring-local-portion-distance                          0
    orleans-directory-ring-local-portion-percentage                        0
    orleans-directory-ring-size                                        1,295
    orleans-gateway-received (Count / 1 sec)                           1,291
    orleans-gateway-sent (Count / 1 sec)                               2,582
    orleans-messaging-processing-activation-data                           0
    orleans-messaging-processing-dispatcher-forwarded (Count / 1           0
    orleans-messaging-processing-dispatcher-processed (Count / 1       2,543
        Direction=Request,Status=Ok                                    2,582
    orleans-messaging-processing-dispatcher-received (Count / 1        1,271
        Context=Grain,Direction=Request                                1,291
        Context=None,Direction=Request                                 1,291
    orleans-messaging-processing-ima-enqueued (Count / 1 sec)          5,113

For more information, see Investigate performance counters (dotnet-counters).

Orleans meters

Orleans uses the System.Diagnostics.Metrics APIs to collect metrics. Orleans categorizes each meter into domain-centric concerns, such as networking, messaging, gateway, etc. The following subsections describe the meters Orleans uses.

Networking

The following table shows a collection of networking meters used to monitor the Orleans networking layer.

Messaging

The following table shows a collection of messaging meters used to monitor the Orleans messaging layer.

Gateway

The following table shows a collection of gateway meters used to monitor the Orleans gateway layer.

Runtime

The following table shows a collection of runtime meters used to monitor the Orleans runtime layer.

Catalog

The following table shows a collection of catalog meters used to monitor the Orleans catalog layer.

Directory

The following table shows a collection of directory meters used to monitor the Orleans directory layer.

Consistent ring

The following table shows a collection of consistent ring meters used to monitor the Orleans consistent ring layer.

Watchdog

The following table shows a collection of watchdog meters used to monitor the Orleans watchdog layer.

Client

The following table shows a collection of client meters used to monitor the Orleans client layer.

Miscellaneous

The following table shows a collection of miscellaneous meters used to monitor various layers.

App requests

The following table shows a collection of app request meters used to monitor the Orleans app request layer.

Reminders

The following table shows a collection of reminder meters used to monitor the Orleans reminder layer.

Storage

The following table shows a collection of storage meters used to monitor the Orleans storage layer.

Streams

The following table shows a collection of stream meters used to monitor the Orleans stream layer.

Transactions

The following table shows a collection of transaction meters used to monitor the Orleans transaction layer.

Prometheus

Various third-party metrics providers are available for use with Orleans. One popular example is Prometheus, which you can use to collect metrics from your app with OpenTelemetry.

To use OpenTelemetry and Prometheus with Orleans, call the following IServiceCollection extension method:

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .AddPrometheusExporter()
            .AddMeter("Microsoft.Orleans");
    });

The AddPrometheusExporter method ensures the PrometheusExporter is added to the builder. Orleans uses a Meter named "Microsoft.Orleans" to create Counter<T> instances for many Orleans-specific metrics. Use the AddMeter method to specify the name of the meter to subscribe to, in this case, "Microsoft.Orleans".

After configuring the exporter and building your app, call MapPrometheusScrapingEndpoint on the IEndpointRouteBuilder (the app instance) to expose the metrics to Prometheus. For example:

WebApplication app = builder.Build();

app.MapPrometheusScrapingEndpoint();
app.Run();

Distributed tracing

Distributed tracing is a set of tools and practices for monitoring and troubleshooting distributed applications. It's a key component of observability and a critical tool for understanding your app's behavior. Orleans supports distributed tracing with OpenTelemetry.

Regardless of the distributed tracing exporter you choose, call:

• AddActivityPropagation(ISiloBuilder): which enables distributed tracing for the silo.
• AddActivityPropagation(IClientBuilder): which enables distributed tracing for the client.

Referring back to the Orleans GPS Tracker sample app, you can use the Zipkin distributed tracing system to monitor the app by updating Program.cs. To use OpenTelemetry and Zipkin with Orleans, call the following IServiceCollection extension method:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        // Set a service name
        tracing.SetResourceBuilder(
            ResourceBuilder.CreateDefault()
                .AddService(serviceName: "GPSTracker", serviceVersion: "1.0"));

        tracing.AddSource("Microsoft.Orleans.Runtime");
        tracing.AddSource("Microsoft.Orleans.Application");

        tracing.AddZipkinExporter(zipkin =>
        {
            zipkin.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        });
    });

The Zipkin trace is shown in the Jaeger UI (an alternative to Zipkin that uses the same data format):

For more information, see Distributed tracing.

Orleans outputs its runtime statistics and metrics through the ITelemetryConsumer interface. Your application can register one or more telemetry consumers for its silos and clients to receive statistics and metrics the Orleans runtime periodically publishes. These can be consumers for popular telemetry analytics solutions or custom ones for any other destination and purpose. Three telemetry consumers are currently included in the Orleans codebase.

They're released as separate NuGet packages:

• Microsoft.Orleans.OrleansTelemetryConsumers.AI: For publishing to Azure Application Insights.

• Microsoft.Orleans.OrleansTelemetryConsumers.Counters: For publishing to Windows performance counters. The Orleans runtime continually updates many of them. The CounterControl.exe tool, included in the Microsoft.Orleans.CounterControl NuGet package, helps register necessary performance counter categories. It must run with elevated privileges. Monitor the performance counters using any standard monitoring tools.

• Microsoft.Orleans.OrleansTelemetryConsumers.NewRelic: For publishing to New Relic.

To configure your silo and client to use telemetry consumers, the silo configuration code looks like this:

var siloHostBuilder = new HostBuilder()
    .UseOrleans(c =>
    {
        c.AddApplicationInsightsTelemetryConsumer("INSTRUMENTATION_KEY");
    });

The client configuration code looks like this:

var clientBuilder = new ClientBuilder();
clientBuilder.AddApplicationInsightsTelemetryConsumer("INSTRUMENTATION_KEY");

To use a custom-defined TelemetryConfiguration (which might have TelemetryProcessors, TelemetrySinks, etc.), the silo configuration code looks like this:

var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
var siloHostBuilder = new HostBuilder()
    .UseOrleans(c =>
    {
        c.AddApplicationInsightsTelemetryConsumer(telemetryConfiguration);
    });

Client configuration code look like this:

var clientBuilder = new ClientBuilder();
var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
clientBuilder.AddApplicationInsightsTelemetryConsumer(telemetryConfiguration);