Monitoring
vibe-live exposes Prometheus metrics at /internal/metrics on its
internal listener (VIBE_LIVE_INTERNAL_LISTEN_ADDR, :8081 by
default). That listener must never be reachable from the internet, so
scrape it from inside your network or cluster.
When you cannot reach the internal listener (a hosted instance, or a
Prometheus elsewhere), the public port serves the same metrics at
/api/v1/admin/metrics, for an admin only:
-
Signed in as an admin, make an app password under Settings โ App passwords with Read this instance's metrics (admin, for Prometheus).
-
Give it to Prometheus as a bearer token:
- job_name: vibe-live scheme: https metrics_path: /api/v1/admin/metrics authorization: credentials: vlap_... static_configs: - targets: [your.instance.example]
A Grafana dashboard for an instance is served at
/api/v1/dashboards/admin.json and linked under Settings โ Instance โ
Monitoring: import it, then pick your Prometheus and the job that
scrapes the instance. Its panels: version and uptime, accounts and live
channels, the media engine's state, requests by status and response
times (the slowest routes too), failed sign-ins and rate-limited
requests, the disk and recordings, and deferred tasks. Streamers have one
of their own (/api/v1/dashboards/streamer.json).
That password reads metrics and nothing else. It works while its owner is still an admin: a local admin, or someone whose latest sign-in through the identity provider carried the admin group within the last 30 days. App passwords cannot ask the identity provider, so a sign-in without the group ends it at once, and an admin who stops signing in loses it after 30 days. Each replica answers with its own counters, as on the internal listener.
The metrics count things and never name anyone: no user, channel or address appears in a label. Each replica reports its own counters since it started, so sum them across replicas for the instance.
What it reports
| Series | What it tells you |
|---|---|
vibe_live_channels_live, vibe_live_users, vibe_live_uptime_seconds, vibe_live_build{version} |
the basics: how busy, how big, how long up, which version |
vibe_live_tasks_pending, vibe_live_tasks_failed |
deferred work (deliveries, deletions); failed ones need a look |
vibe_live_http_request_duration_seconds{route,code} |
how long each API route takes to answer, by status class. A histogram, labelled by the route's pattern (GET /api/v1/channels/{id}), never its path. WebSockets are left out. |
vibe_live_rate_limited_total{limiter} |
requests each rate limiter refused: app_password_auth, login_account, second_factor, lookup, mention, force_stop, inbox, federation, chat |
vibe_live_rate_limited_by_instance_total{limiter,instance} |
the same refusals on lookup, inbox and federation, by the instance the requests said they came from: a peer's host, or other (the admin dashboard's Instance menu filters on it) |
vibe_live_login_failures_total{reason} |
failed password sign-ins: unknown_account, bad_password, locked, second_factor. A brute force shows here first; the addresses are in the log. |
vibe_live_recordings_bytes, vibe_live_disk_used_bytes, vibe_live_disk_total_bytes |
what past broadcasts take, and the disk they are on (only where something records) |
vibe_live_recordings_limit_bytes or vibe_live_recordings_limit_disk_percent |
the limit you set in Settings, Recording, in whichever form you set it |
vibe_live_engine_status{status} |
the media engine's health, 1 on its current status: answering, not_answering, in_process (the built-in engine) or unknown |
The engine and the media workflows add series of their own:
vibe_live_ome_* for OvenMediaEngine, vibe_live_builtin_* for the
built-in engine, and the ingest, rendition and thumbnail counters.
Alerts
deploy/prometheus/vibe-live.rules.yml, in the repository, holds alerts on these series. Load it with rule_files: in
prometheus.yml, or put its groups in a PrometheusRule if you run
the Prometheus Operator.
| Alert | Fires when |
|---|---|
VibeLiveDiskAlmostFull |
the recordings disk is over 90% full for 10 minutes |
VibeLiveRecordingsNearTheirLimit |
recordings take over 90% of the size you allow |
VibeLiveSlowAnswers |
one answer in twenty takes over a second, for 5 minutes |
VibeLiveServerErrors |
over 5% of answers are server errors, for 5 minutes |
VibeLiveSignInFailuresSpike |
over ten failed sign-ins a minute, for 5 minutes |
VibeLiveEngineNotAnswering |
the media engine stops answering for 2 minutes |
The thresholds are starting points: tune them to your instance's normal.
No Prometheus? The app checks the same conditions itself every minute and posts them to a webhook the admin sets in Settings, Instance, Alerts. Use one or the other; both would announce each alert twice.