Dartvel monitoring: metrics, traces, crashes and alerts

Every generated backend serves metrics and a health check, and every app records its own crashes, with nothing to install.

curl -H "Authorization: Bearer $DARTVEL_METRICS_TOKEN" https://your-host/metrics
curl https://your-host/health
dartvel metrics

It is private by default. It answers the same host and private networks, and anyone sending the server's DARTVEL_METRICS_TOKEN as a bearer token. Everyone else gets a 404.

// Registered on first use, and served at GET /metrics as Prometheus text.
DV.ObservabilityAndLogging.metrics
    .counter('refunds_total', <String, String>{'currency': 'usd'})
    .increment();

DV.ObservabilityAndLogging.metrics.gauge('queue_depth').set(12);

DV.ObservabilityAndLogging.metrics
    .histogram('checkout_seconds')
    .observe(0.42);

// Each check answers for itself when GET /health is asked, under a
// deadline. Degraded is not down: a cold cache is slow, and reporting it
// as an outage pages somebody for nothing.
DV.ObservabilityAndLogging.health.register('payments', () async {
  final bool reachable = await gatewayReachable();
  return reachable
      ? DVHealthResult.up()
      : DVHealthResult.down('the gateway did not answer');
});

DV.log writes one record: a message, a tag and the fields as data. It takes the same arguments in the app and on the server, and there is a method per level, like DV.log.warn.

Each record also goes to the platform's own log: logcat on Android, the unified log on iOS and macOS, journald or stderr on Linux, the console in a browser, and JSON lines on stdout from a server.

Sensitive data model fields, tokens, passwords in URLs and secrets your app has read are redacted before any of them sees the record.

// One record: a message, a tag to filter on, and the fields as data. The
// same call on a phone, in a browser and on the server; there is no second
// logger to configure per package.
void recordRefund(String orderId, int cents) {
  DV.log(
    'refunded an order',
    tag: 'billing',
    context: <String, Object?>{'order': orderId, 'cents': cents},
  );
  // A method per level, each taking the same shape.
  DV.log.warn('refund took a second attempt', tag: 'billing');
}

// The same two names on the server: DV.log for a line, and
// DV.ObservabilityAndLogging for everything the server serves about itself.
DV.log('Refund accepted', context: <String, Object?>{'orderId': 'o-42'});

DV.log(
  'Gateway slow',
  level: DVLogLevel.warn,
  // A code is what an alert rule matches on. Wording changes; this does not.
  code: 'PAY-SLOW',
  context: <String, Object?>{'gateway': 'stripe', 'ms': 2400},
);

On a phone or desktop, records also go to a log file that never grows past 2 MB, by default, and drops files older than 7 days. Set the size and age under dartvel.logging.file, or turn it off with file: false.

To collect device logs on your own backend, set dartvel.logging.ship.enabled to true. Devices then send warnings and errors with a random install id, the release and the platform, and nothing that names a person.

The backend writes those records into its own log stream. On a deployment Dartvel Cloud hosts, it can also pass them on to the hosting log service.

// What the device kept, oldest first, one JSON object per line -- for a
// "send us your logs" button. share() opens the platform share sheet with
// the newest records that fit.
Future<void> sendLogs() => DV.log.share();
Future<String> logsAsText() => DV.log.export();

Sampling is decided from the trace id, so every service in a trace makes the same choice without talking to the others.

// Timed as one named step, so a slow checkout points at the work inside
// it instead of at the request as a whole.
Future<T> priced<T>(Future<T> Function() work) =>
    DV.ObservabilityAndLogging.trace<T>('pricing', work);

// The request already has a span, joined to the traceparent it arrived
// with. This is for the work inside it worth seeing on its own.
final DVSpan span =
    DV.ObservabilityAndLogging.tracer.startSpan('reprice-basket');
span.setAttribute('basket.id', 'b-9');
try {
  await Future<void>.delayed(const Duration(milliseconds: 5));
} finally {
  // In a finally: a thrown error would otherwise leave the span open, and
  // the trace then shows a request that never ended.
  span.end();
}

The generated client installs crash reporting at startup. A crash is written as it happens and sent on the next launch, once.

With sink: dartvel under dartvel.crashes, your own backend receives reports. Server processes record a 500, a failing schedule and a dead-lettered job.

On Android a Java or Kotlin crash is recorded too, by a handler installed before the Flutter engine starts. It writes into the store Dart sends from on the next launch, then lets Android end the process as it would have.

A release build is obfuscated. dartvel build android collects that build's R8 mapping and Flutter debug symbols into build/symbols, and dartvel symbols upload (or dartvel.crashes.symbols.upload) sends them to your backend with the DARTVEL_SYMBOLS_TOKEN secret. Each report is then read with its own release's symbols and no other's, and grouped by the original names so a bug keeps its group across releases.

No native signal handler on Android, and no handlers at all on iOS or Windows, so a crash below Java and Dart is not caught.

Symbols are collected and read for Android only: no iOS dSYMs and no web source maps. Native frames are stored but not read.

DVAlerting fires a rule after it has held for a set time and delivers through DV.Notifications or PagerDuty, once per episode.

An alert opens an incident with a timeline, and a status snapshot shows each component's health without internal detail.

// A promise with a budget attached, and a rule that fires on the budget
// instead of on a single bad minute.
const DVServiceLevel checkout = DVServiceLevel(
  name: 'checkout',
  objective: DVObjective.successRate(0.995, over: Duration(days: 30)),
  applies: DVAppliesTo.backendFunction('placeOrder'),
);

final DVAlertRule backlog = DVAlertRule(
  name: 'mail-backlog',
  signal: const DVSignalRef.queueDepth('mail'),
  condition: const DVAlertWhen.above(100),
  // A rule that fires on the first sample pages somebody for a spike that
  // cleared itself.
  forDuration: const Duration(minutes: 5),
  notify: const <DVAlertTarget>[DVAlertTarget.team('ops')],
);

Declare consent categories under dartvel.analytics. Events in a denied category are dropped at the call, never sent later.

GETTING STARTED

APP

DATA

BACKEND

OPERATIONS

SHIPPING

REFERENCE

Monitoring

Read metrics and health from any backend

One log stream, on the device and on the server

Keep logs on the device and send them when asked

Follow one request across services

Crash reports are on from the first build

Alert on error budgets and open incidents

Product analytics that respect consent

Status