On-call analytics and postmortems

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/analytics"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/6fdf613e-7f77-4c16-aa7d-e65e9b95583a/social-card.svg?v=3e7ae33eec8b" alt="On-call analytics and postmortems" width="600" height="315"></a>

GET /v1/projects/{prj}/oncall/analytics/{metric}?start=&end= where metric is mtta, mttr, noise or burden. The window is [start, end), at most 365 days. Each query runs read-only in Postgres with a 10 second statement timeout (a timeout answers as a query timeout). Add ?drill=true (and part= for noise) to list the exact rows a figure averaged, as {data, next_cursor} with ?limit=&cursor=. The summary and the drill-down read the same row set, so the count and mean always match.

Definitions

  • MTTA = mean(acked_at − created) over pages with an ack. Pages created in the window.

  • MTTR = mean(resolved_at − declared_at) over resolved incidents. Open incidents are excluded; incidents resolved in the window; merged-away incidents are excluded.

  • Noise = alerts per incident, the % of alerts auto-resolved within 5 min, and the % of pages never acked.

    • part=alerts_per_incident: for every incident declared in the window, the number of alerts linked to it; the figure is the mean.

    • part=auto_resolved: for every alert started in the window, whether it resolved within 5 minutes of starting with no ack; the figure is flagged / alerts.

    • part=unacked_pages: for every page created in the window, whether it has no acked_at; the figure is flagged / pages.

  • Burden = pages per user per week, off-hours pages (22:00–07:00 in the user's tz, plus weekends) and sleep interruptions (00:00–06:00).

    • A page counts once per notified responder, at the first notification; owner notices (exhausted policy, removed member) are not pages. Pages are those created in the window.

    • The user's tz is oncall_user_settings.tz (the on-call settings), UTC when unset. Weekends are Saturday and Sunday in that tz.

    • Pages per week = pages / (window length in weeks, at least one day).

Visibility

Incidents marked private count only for people who can see them (admins, the declarer, assigned roles). Alerts linked to a private incident the caller cannot see are left out of the alert figures, so two people can see different figures over the same window.

Postmortems

  • Templates: POST/GET /v1/projects/{prj}/postmortem-templates, each a list of sections {key, title}. Keys summary, impact, timeline, root_cause and action_items are filled from the incident; any other key is a heading left to write.

  • POST /v1/projects/{prj}/incidents/{inc}/postmortem drafts one per incident from the template (default: all five): summary, impact (services, duration, customer impact), pinned timeline entries, the root cause given in the request and the incident's action_item tasks. There is no LLM drafting (Phase 5).

  • GET and PUT the same path read and edit it. Status goes draft, in_review, published; publishing is final.

  • GET …/postmortem.md exports the document as text/markdown. Private notes are never in the export. A postmortem is visible only to people who can see its incident (404 otherwise). Everything the draft copies from the incident (titles, notes, task titles) is backslash-escaped so it cannot inject links, images, headings or HTML.

Pay cost report

GET /v1/projects/{prj}/oncall/schedules/{sch}/cost?from=&to= (CSV with Accept: text/csv) prices the schedule's shifts with its pay rule (PUT .../pay-rule): each quarter hour, in the schedule's zone, is paid rate times the highest multiplier that applies (night, weekend, holiday); multipliers never stack. Shadows are not paid and overlapping shifts of one user count once. It reads only; there is no payroll export.

Business services and owner paging

An alert on a service pages, in order: the service's own policy, the policy of a business service that lists it, else the owner team's receivers. The router stamps page_policy and page_reason on the alert and writes the resolution to the alert timeline when it fires. Incidents list their impacted_business_services.

Domain events

incident.declared|acknowledged|resolved, alert.firing|resolved, oncall.shift_started, oncall.override_created and issue.created|regressed can be subscribed to; bodies are {type, version: 1, at, project, data}. Managing subscriptions needs ManageConnectors.

Updated