Agent index (llms.txt) — fetch this first, then the page it names.

Monitoring

Cluster health, from your machine. Three parts work together:

1. +orama node report runs on each node, collects everything local and outputs JSON.
2. +The cluster gateway's telemetry gathers every node's report, adds each gateway's request metrics (traffic), derives alerts, and serves the result to operators at /v1/operator/telemetry. A public projection is at /status.
3. +orama status runs on your computer, reads that telemetry and shows it: a live view, one table, or one aspect at a time.

Quick start

orama status                          # live view in a terminal, one table when piped
orama status --once                   # one table and exit, even in a terminal
orama status cluster                  # verdict, components, a row per node, top alerts
orama status alerts                   # alerts with what to do about each
orama status report                   # full JSON (pipe to jq, or feed to an LLM)
orama status cluster --ssh            # gateways are down: read the nodes directly

You need to be signed in as an operator of the cluster (orama auth login). Every command takes --env <env>; without it the active environment is used (orama network use).

Architecture

One-shot subcommands do a GET of the snapshot; the live view keeps one server-sent-events stream open.

By default the monitor makes no SSH connection. It reads a ClusterSnapshot (every node's report, how old each report is, and the derived alerts) from the gateway of the environment, authenticated with the credentials orama auth login stored for that gateway. Only the cluster's operators may read it.

--ssh is the break-glass path for when no gateway answers: it SSHes into every node, runs sudo orama node report --json there, and derives the alerts locally. It is never chosen automatically — when the API fails, the monitor stops with an error that says why and suggests --ssh, so a gateway outage is reported as one instead of being hidden behind a slower path. Traffic is counted by the gateways, so it is empty over --ssh.

The telemetry API

Both take Authorization: Bearer <token>. How the monitor answers failures:

A missing credential or an ended session is an auth error (3); failing to reach the gateway to renew a session, or any renewal answer that is not a refusal (a 5xx, a rate limit), is an outage (5) and leaves the session intact — the CLI ends a session only when the gateway refuses it (401/403), so a live view running through a rolling upgrade keeps its login. The client caches the bearer for 30s and renews it one caller at a time (a renewal rotates the refresh token, so two at once would end the session), bounds the connect (10s) and the wait for response headers (45s: the stream sends them with its first snapshot), and follows no redirect; neither does node resolution nor the session renewal behind orama auth login credentials.

TLS trusts the same CAs as every other API command: ORAMA_CA_CERT_PATH, and the environment's ca_file (orama network add <name> <url> --ca-file <pem>).

Live stream behaviour

The live view keeps one stream open. When the gateway ends a stream that delivered snapshots (it does so on a schedule), the monitor reopens it after 1s without changing what the view shows. Any other drop — the connection broke, or it went silent for longer than max(30s, 3 × interval) with no keepalive — shows ↻ reconnecting in Ns (attempt n): <why> and reconnects with backoff: 1s, doubling to at most 30s, each wait spread ±20% so monitors do not reconnect in lockstep, starting over after any connection that delivered a snapshot. A refused credential (401/403) is not retried, nor is a missing endpoint (404) before the first snapshot; after one, a 404 is retried (a reconnect may reach a gateway not yet upgraded). The view then shows ✗ stopped: <why>. With --ssh, a collection that cannot succeed on retry (an unreadable --config, an environment with no nodes) stops the view the same way.

Old data is never shown as current. Every view shows the snapshot's age ("updated 4s ago"; in the live view, measured from when the snapshot arrived on this machine, so a gateway clock that differs from yours does not skew it); in the live view it turns into STALE: updated … ago once the snapshot is older than three intervals (plus the SSH collection time with --ssh), or when the source has stopped. A manual refresh (r) that returns an older snapshot than the stream's is dropped. An event: error from the gateway is shown beside the connection state and keeps the last snapshot.

Cluster telemetry on the gateway

What /v1/operator/telemetry and /v1/status are built from. Each node keeps its own health report fresh, and the cluster gateways assemble the cluster view from those reports over the WireGuard mesh.

 every node                                          any node
┌───────────────────────────────┐     mesh (WG)    ┌────────────────────────────┐
│ orama-namespace-gateway@index │ ◀──────────────▶ │ cluster gateway            │
│  every 10s: orama-privhelper  │ /v1/internal/    │  aggregator: own report +  │
│   node-report (as root)       │  telemetry       │  every peer's, 5s cache    │
│  + this gateway's traffic     │ (coordination    │  → alerts, components      │
│  = the node's latest report   │  MAC + WG source)│  → /v1/operator/telemetry  │
└───────────────────────────────┘                  │  → /v1/status, /status     │
                                                   └────────────────────────────┘

Cost at scale. A node pays one report collection per 10s whatever the number of viewers; each collection is one short-lived root helper process (one start/stop pair in the journal every 10s, the helper's audit trail). A viewer pays nothing beyond the cached snapshot: the public page and the operator stream are answered from the 5s cache, so load grows with the number of nodes and not with the audience. Each gateway that is being watched fetches every peer's full report once per 5s, so if every node is serving status traffic at once the mesh carries N×N reports per 5s — about 25 MB/s at 100 nodes. That is acceptable at today's size; the next step, before a fleet of hundreds, is a compact per-node component summary for the public page (or one aggregator per region) rather than full reports. The 15s assembly bound with 16-way concurrency covers roughly 60 unresponsive peers before later ones are cut off and shown unreachable.

What an operator sees vs the public. /v1/operator/telemetry (operator grant and the operator list) is every node's full report — addresses, ports, firewall state, versions. /v1/status is the public projection (cluster.PublicStatus): service states and uptime, node counts, chain progress and network traffic, and no address, peer id, hostname or error text. The public verdict comes from service states alone; operator alerts (a firewall rule, a certificate expiring) do not change it.

Public status page

https://<base-domain>/status in a browser is the status page; /v1/status is its JSON. On the bare base domain the gateway serves its own pages — /status, /status/assets/, /health — instead of looking for a deployment there (none may use the apex: a custom domain equal to the base domain is refused); a subdomain's /status is its deployment's. The page is static and embedded in the gateway (pkg/gateway/statuspage), loads only its own script and stylesheet under a CSP that allows nothing else, and refreshes every 10 seconds. It shows:

• +the overall verdict and when it was measured,
• +nodes healthy, network requests per second, error rate and p95 latency,
• +each service with its state and a 90-day uptime bar (one bar per UTC day: ≥ 99.9% green, ≥ 95% amber, below that red, no data grey),
• +the Orama L1: chain id, height, average block time, last block age, mempool, and each validator's share of voting power (consensus addresses are public on-chain data), taken from the node at the median height so that one node's view — say an RPC answered by something else on its loopback — cannot set what the public sees.

A namespace gateway answers /v1/status with status and server only; the cluster view is the cluster gateway's. The public view is built once per 5s however many requests arrive, outside any lock, and a failed build ("Status unavailable", state unknown) is cached for the same 5s, so a slow registry costs one attempt per window. /status, /v1/status, the page's assets and /v1/internal/telemetry answer while the gateway is still starting.

orama status

Usage

orama status [subcommand] [--env <environment>] [flags]

Without a subcommand it shows everything about your nodes in one place: each node's cluster health and chain, the verdict with what to do, and your account on the chain (earnings, spendable balance, bond), the one orama setup registered your nodes under or the one --operator names. In a terminal that is the live view; piped, or with --once, it prints one table; with --json it prints a document whose healthy is true only when the verdict is operational, every node is healthy and every node's chain answers and has caught up.

Flags

Subcommands

Every table starts with the verdict line; every subcommand takes --json. There is no live subcommand: the live view is orama status itself.

The verdict line

✓ All systems operational · 3/3 nodes · updated 2s ago
✗ Degraded: Database (RQLite) · 2 critical, 1 warning · 3/3 nodes · updated 4s ago
✗ Outage: API Gateway · 3 critical · 3/3 nodes · updated 3s ago

It is cluster.Summarize over cluster.Components: the worst component state (operational, degraded, outage), made at least degraded by any critical alert, with the components that are degraded or down named. "3/3 nodes" counts nodes whose report came back. Green, yellow and red mark the three states.

Color is used only on a terminal: output to a pipe or file, or any output with NO_COLOR set, carries no escape codes. Columns are padded by visible width, so colored cells line up. Every string that arrives from the cluster (hosts, alert messages, a gateway's error answers) and every error the monitor reports (which can quote a certificate's hostname, a resolver or a wallet) has its control characters replaced before it is shown, so a node or gateway cannot drive the operator's terminal with escape sequences. The telemetry client never follows a redirect, so the operator's bearer token is only ever sent to the gateway it was issued for.

Alert hints

alerts, cluster and the live view print a next step under each critical and warning alert. They are real commands and runbook sections, by subsystem. A host is only put into a command when it is an IP address:

Examples

# Cluster overview as JSON
orama status cluster --env devnet --json

# Single-node deep dive
orama status node --env devnet --node 51.195.109.238

# What the gateways are serving
orama status traffic --env devnet

# Chain height, sync and validators
orama status chain --env devnet

# Full report for one node
orama status report --env devnet --node 51.195.109.238

# Break-glass, with a custom node list
orama status cluster --env devnet --ssh --config /path/to/nodes.conf

# Live view over SSH, refreshing every 30s
orama status --env devnet --ssh --interval 30s

Live view

orama status in a terminal opens a full-screen view. The top two lines are always the environment with the connection state (● live, ◌ connecting…, ↻ reconnecting…, ✗ stopped) and the verdict line.

Tabs: 1 Overview · 2 Operator · 3 Nodes · 4 Services · 5 Traffic · 6 Chain · 7 Mesh · 8 DNS · 9 Namespaces · 0 Alerts

Keys:

Report format

orama status report writes one JSON document for scripts, the lifecycle harness and LLMs. Fields are only ever added, never renamed or removed:

{
  "meta": {
    "environment": "testnet",
    "collected_at": "2026-02-16T12:00:00Z",
    "duration_seconds": 3.2,
    "node_count": 3,
    "healthy_count": 3,
    "failed_count": 0
  },
  "summary": {
    "rqlite_leader": "51.195.109.238",
    "rqlite_quorum": "ok",
    "wg_mesh_status": "ok",
    "service_health": "ok",
    "critical_alerts": 0,
    "warning_alerts": 1,
    "verdict": { "state": "operational", "headline": "All systems operational · 1 warning to look at", "critical": 0, "warning": 1, "info": 0, "nodes_healthy": 3, "nodes_total": 3 }
  },
  "components": [
    { "id": "gateway", "name": "API Gateway", "state": "operational", "healthy": 3, "total": 3, "summary": "All 3 nodes healthy" }
  ],
  "alerts": [...],
  "nodes": [
    { "host": "51.195.109.238", "role": "nameserver-ns1", "status": "ok", "report": { ... } }
  ]
}

summary.rqlite_leader is the leader's public host, or "none". A node's status is ok, degraded (a critical alert names it) or unreachable (error says why; no report). report_age_sec is how old the node's report was when the gateway assembled the snapshot (0 over --ssh).

orama node report — VPS-Side Collector

Runs locally on a VPS node. Collects all system and service data in parallel and outputs a single JSON blob. Requires root privileges. A process running as orama gets the same report through the privileged helper, orama-privhelper call node-report (Go: privhelper.NodeReport), which only orama-node.service and orama-namespace-gateway@index.service may use.

Usage

# On a VPS node
sudo orama node report --json

What It Collects

Bounds

The collectors read services on loopback, where any local process — a tenant deployment included — can bind a stopped service's port. Every response body the report reads (httpGet, the gateway's /v1/health, the Kubo RPC, the IPFS Cluster REST API) is capped at 2 MiB (maxLocalResponseBytes); a larger body is an error naming the URL, never a truncated read. Bodies that are only drained are discarded up to the same cap. Every external command, the namespace systemctl is-active checks included, runs with a 4-second timeout (runCmd).

Inside the privileged helper, node-report takes a non-blocking exclusive flock on /run/orama-privhelper-node-report.lock: while one collection runs, another request fails with "a node report collection is already running" instead of queueing. A collection that has not finished within 50 seconds — below the gateway's 60-second client timeout — is answered with a failure; the helper instance exits after responding, which ends the collection and releases the lock.

Performance

All 17 collectors run in parallel with goroutines. Typical collection time is < 1 second per node. HTTP timeouts are 3 seconds, command timeouts are 4 seconds. The chain's slashing and staking queries have their own 8-second budget after the CometBFT RPC and do not change responsive when they fail.

Output Schema

{
  "timestamp": "2026-02-16T12:00:00Z",
  "hostname": "ns1",
  "public_ip": "51.195.109.238",
  "wireguard_ip": "10.0.0.1",
  "version": "",
  "collect_ms": 526,
  "system": { "cpu_count": 4, "load_avg_1": 0.1, "mem_total_mb": 7937, ... },
  "services": { "services": [...], "failed_units": [] },
  "rqlite": { "responsive": true, "raft_state": "Leader", "term": 42, ... },
  "olric": { "service_active": true, "memberlist_up": true, ... },
  "ipfs": { "daemon_active": true, "swarm_peers": 2, ... },
  "vault": { "service_active": true, "responsive": true, "status": "healthy", ... },
  "gateway": { "responsive": true, "http_status": 200, ... },
  "wireguard": { "interface_up": true, "wg_ip": "10.0.0.1", "peers": [...], ... },
  "dns": { "coredns_active": true, "caddy_active": true, "base_tls_days_left": 88, ... },
  "tor": { "client_active": true, "socks_listening": true, "bootstrapped": true, "bootstrap_pct": 100, "legacy_anyone": false },
  "network": { "internet_reachable": true, "ufw_active": true, ... },
  "processes": { "zombie_count": 0, "orphan_count": 0, "panic_count": 0, ... },
  "namespaces": [],
  "deployments": { "total_count": 0, "running_count": 0, "failed_count": 0, "static_count": 0 },
  "serverless": { "function_count": 0, "engine_status": "healthy" },
  "chain": { "service_active": true, "responsive": true, "chain_id": "orama-stagenet-1", "latest_height": 1234, "block_age_sec": 1.8, "avg_block_time_sec": 5.0, "is_validator": true, ... }
}

Alert Detection

Alerts are derived from cross-node analysis of all collected reports (cluster.DeriveAlerts): by the cluster gateway for the telemetry API, and by the monitor itself with --ssh. Each alert has a severity level and identifies the affected subsystem and node.

Alert Severities

Cross-Node Checks

These checks compare data across all nodes:

• +RQLite Leader: Exactly one leader exists (no split brain)
• +Leader Agreement: All nodes agree on the same leader address
• +Raft Term Consistency: Term values within 1 of each other
• +Follower Contact: Every follower has heard from the leader within 2 seconds (rqlite.StalenessMaxLastContact, the bound after which a namespace gateway stops serving none-reads from it); a follower that has never heard from one is flagged too. Each node is judged from its own /status read (store.raft.last_contact). The applied indexes of the nodes are deliberately not compared: each report is collected at a slightly different moment, so under steady writes their spread counts the writes made between the reads, not lag (a healthy 5-node stagenet read 101 apart). A node's own apply backlog is the per-node "RQLite commit-applied gap" warning (commit_index minus applied_index over 100, both from one read). orama maint inspect follows the same rule: rqlite.last_contact warns on a stale follower and there is no cross-node applied-index check
• +WireGuard Peer Symmetry: Each node has N-1 peers
• +Clock Skew: Node clocks within 5 seconds of each other (critical beyond 60s). The offset is measured when each report is served — the peer sends its clock (X-Orama-Clock-Ms) and the collector compares it with the request's midpoint — not read from report timestamps, which differ by up to the collection interval on synchronised clocks. A peer that sends no clock (0.122.109) is left unmeasured and its report still counts
• +Binary Version: All nodes running the same version. orama node report used to emit an empty version, so every node read as "unknown" and the alert could never fire; the version is compiled into the binary now, so it carries a real value.
• +Chain height: A responsive node that is not catching up and is more than 20 blocks behind the median responsive height. One node's height cannot set the median.

The lifecycle harness

e2e/lifecycle consumes orama status report --json as its only view of the cluster, so the report's schema is a test contract. It decodes each node's report into the real report.NodeReport and the alerts into cluster.Alert; only the meta/summary envelope is declared in the harness. (It used to be a hand copy whose field names had drifted — rqlite.leader, allowed_ip, restart_loop — so those fields read as zero and the predicates using them could not work; bug 2701.) Its predicates assert:

• +Converged(n) — exactly n nodes, every report at most 90s old (report_age_sec: the age at which the gateway itself stops counting a report, three missed 10s collections plus the 60s collection timeout, so a loaded node whose collection takes a while still reads converged), quorum ok, a leader (not "none"), WG mesh ok, zero critical alerts, and per node: responsive rqlite in Leader/Follower, gateway 200, wg0 up with N-1 peers, no service with restart_loop_risk, no failed Orama unit (orama-*, wg-quick@wg0, caddy, coredns: the inspector's rule; a failed unit the host image ships is a warning alert, not an unconverged cluster).
• +LeaderAgreement() — every responsive node names the same leader (rqlite.leader_addr). Split brain is failed on by name, because both halves look healthy from inside.
• +Forgotten(wgIP) — no surviving node lists the address, in the node list (wireguard_ip) or as a WireGuard peer (allowed_ips). It does not read raft membership, and through the API the node list is the cluster's node registry as the gateway sees it. A node evicted from raft but left in the mesh is the failure that survives a restart.
• +Serving() — gateways respond and nameservers run CoreDNS, with raft ignored entirely. A cluster mid-election must still serve.

Adding a field is safe; renaming one that these read fails make test: e2e/lifecycle decodes a marshalled report.NodeReport, and cmd/orama/internal/monitor/display/report_contract_test.go runs the real orama status report output through the harness's ParseReport and predicates.

The rolling-upgrade gate

A rolling upgrade (orama upgrade, orama maint rollout) uses the same signals, in pkg/nodehealth, as its gate between nodes. A node passes when all of these hold:

Anything short of all four stops the rollout, leaving the remaining voters untouched. The same package backs orama maint node install's post-install verification, orama node start, and the post-upgrade step of orama node upgrade, so "ready" means one thing across the CLI.

Per-Node Checks

• +RQLite: Responsive, ready, strong read
• +WireGuard: Interface up, handshake freshness
• +System: Memory, disk, load, OOM kills, swap
• +Services: Systemd state, restart loops
• +DNS: CoreDNS/Caddy up, TLS cert expiry, SOA resolution
• +IPFS: Daemon/cluster down, no swarm peers, swarm key missing, bootstrap list not empty, repo above 90%/95%, and a pin/add, pin/update or repo/gc active for more than 30 minutes — the repo GC unit's TimeoutStartSec. Such a request holds or waits for Kubo's pin lock, so orama-namespace-ipfs-gc@index cannot run and times out having freed nothing (a timed-out GC run itself shows as a failed systemd unit)
• +Tor: SOCKS port bound while active, bootstrap below 100% (an unknown bootstrap is not an alert), Anyone-network leftovers
• +Processes: Zombies, orphans, panics in logs
• +Namespaces: Gateway and RQLite per namespace. A circuit breaker toward a namespace gateway that is open or half-open is a warning, one per target node, naming up to five namespaces, how many of the breakers are open and how many half-open (3 open, 1 half-open), and the consecutive failures and last error of the breaker whose last failure is the oldest: requests for those namespaces through the reporting node are refused (or probed) while it lasts, the others on that node are served. The same for an app: a breaker toward the nodes that run a deployment is a deployment warning per target node, naming namespace/deployment. The gateway log has each transition (circuit breaker opened, half-open, closed)
• +Network: UFW, internet reachability, TCP retransmission
• +Chain: Unit not active, RPC not answering, height lag, no peers against a multi-member validator set, a block older than 60 seconds, jailed, tombstoned, missed-block ratio against min_signed_per_window (warning at half the downtime miss fraction, critical at the fraction). A node that is catching up is not alerted for lag or for a stale block. A signing query that fails while jail and the ratio are both unknown is a warning, not a claim that the validator is fine. A full node that is absent from the validator set is not an error.
• +Update: a newer release was found and not installed (info); a release was refused or an install was rolled back (warning). See Upgrades.
• +Global: Installed orama-global-ipfs, orama-global-provider, orama-global-indexer, orama-global-tor-relay, and orama-global-tor-dirauth units that are not active. An active chain indexer whose status does not answer (global.indexer.api) or that reports it is stuck on a block (global.indexer.stuck, with the block and the reason). Public Kubo repo over StorageMax. Provider hot-key balance 0, proof misses, and disk over the declared maximum, and a relay or directory authority (global.relay.consensus, global.dirauth.consensus) that says it is outside the relay set, only when monitor.json carries those fields. The provider writes its file every step, so its three alerts fire; the relay and the authority write their own every five minutes, so the relay-set alert fires when the node's consensus does not list it and stays quiet while it has no valid consensus.

Request metrics

Every gateway (the cluster gateway index and each namespace gateway) keeps live request metrics in memory — nothing is written to a database, and a restarted gateway starts from zero. They are kept by pkg/telemetry/traffic.Recorder, fed from the gateway's logging middleware, and read with Gateway.TrafficSnapshot() as a report.TrafficReport.

• +What is counted: every request the gateway answers, with its status, response bytes and duration. requests, errors_4xx, errors_5xx, bytes_per_sec, rps, and error_rate (5xx ÷ requests, 0 with no requests). total_requests counts every request since the gateway started. A WebSocket upgrade counts as a request but adds no latency sample, since its duration is the life of the connection.
• +Window: the last 60 seconds, as 60 one-second buckets. Rates divide by the time the window actually covers: since the first request when the gateway has served for less than a minute (never less than 1s); window_sec says which. After more than 60s without traffic the window is empty.
• +Percentiles: p50_ms, p95_ms, p99_ms come from a fixed latency histogram with log-spaced buckets from 1ms to 30s, interpolated linearly inside the bucket the percentile falls in. Anything slower than 30s reports as 30000ms.
• +Namespaces: each request is attributed to the namespace auth or domain routing resolved for it (for the cluster gateway, the target of an ns-<name>.<domain> or serverless proxy, or a deployment's namespace); otherwise to the gateway's own client_namespace. Names are validated ([A-Za-z0-9][A-Za-z0-9_-]*, at most 64 characters) or counted as (invalid). Each second tracks at most 256 distinct namespaces and folds the rest into (other), so memory stays bounded whatever hosts clients send. A snapshot lists the 20 busiest namespaces with their requests, rps, 5xx and p95. Paths and client IPs are never labels.
• +Excluded: the gateway's own health and telemetry plumbing, so monitoring does not measure itself: /health, /v1/health, /v1/internal/ping, and everything under /v1/internal/telemetry and /v1/operator/telemetry.
• +Slow requests: a request that takes a second or more is also logged at warning level as slow request, with the time each step took: routing_ms (from the request's arrival until it is routed to a namespace), auth_ms (validating the credential for a namespace proxy), targets_ms (looking the namespace's gateways up), upstream_ms (until the namespace gateway's response headers) and rest_ms (everything after the last step, including streaming the response body, so a large download is slow there by design). A step the request did not take is absent. WebSocket upgrades are not logged: their duration is the connection's. Read it with orama node logs orama-namespace-gateway@index.

Monitor vs Inspector

Both tools check cluster health, but they serve different purposes:

Use orama status for day-to-day health checks and the live view. Use orama maint inspect (Inspector) for deep diagnostics when something is already known to be broken.

In-cluster failure detection (the ring monitor)

Separate from the SSH tooling above, every node runs a ring-based failure detector inside its index gateway process. It is what drives automatic recovery, so its thresholds decide how fast a dead node is noticed and how easily a healthy one is wrongly evicted.

The probe port is configuration, not a literal. It is the index gateway port on the peer, because the ring monitors nodes rather than namespaces and only the index gateway serves /v1/internal/ping. health.NewMonitor returns an error if it is unset: the port was hardcoded once, survived the move of the index internals into the 10100–10109 block untouched, and every probe on a healthy fleet then failed — which the ring turns into a cluster-wide false eviction about seven minutes after the gateways start.

A recent heartbeat outranks the probe. Before issuing HTTP, the monitor checks whether the peer updated dns_nodes.last_seen within the last 65 seconds (two heartbeat ticks). That row is a raft write, so a fresh value proves the peer was alive and had quorum — evidence that does not depend on the HTTP path. A stale heartbeat proves nothing either way and falls through to the probe rather than counting as a miss.

Observations land in node_health_events; recovery is triggered only by the lowest-id observer once quorum agrees, so a confirmed death produces one recovery action rather than one per observer.

Namespace DNS self-management

Separately from the ring monitor, each node probes the namespaces it hosts every 30s and keeps its own DNS records in step. rqlite and Olric are probed by dialling their ports on this node's WireGuard address; the gateway is asked GET /v1/health on that same address, because it binds and answers long before it has a usable schema and a TCP dial cannot tell the difference — a gateway that could not serve a single request used to stay in the round-robin. A tenant gateway does not listen on loopback, so a probe of 127.0.0.1 reports every namespace down and withdraws a node that is serving.

The gateway probe reads readiness only. starting (waiting for its schema) and blocked (schema below what the binary requires) count against the namespace; degraded and unhealthy do not, because those are subsystem health — rqlite and Olric have their own probes above, and withdrawing a node because its IPFS daemon blipped would turn one unavailable subsystem into an unavailable node.

A gateway that is starting therefore adds its convergence time to the re-advertise lag after a restart (bug-286).

Two safety rules:

• +Never the last record. A withdrawal is refused when this node's record is the only active one for that name. The count is evaluated inside the UPDATE, so simultaneous withdrawals on different nodes cannot empty the round-robin between them.
• +A peer's verdict outranks a stale one, not a live one. Records this process withdrew are restored as soon as its own probe recovers. A record a peer disabled (the ring monitor's suspect path) is only reclaimed after it has sat untouched for 10 minutes, so a monitor that still considers this node suspect keeps it out.

Watch it with:

sudo orama node logs node --since -1h | grep -E 'namespace DNS round-robin'

Configuration

The API source uses the environment's gateway URL (orama network list) and the credentials orama auth login stored for it — the same ones orama nodes uses to list your nodes. With --ssh, nodes are resolved the same way the inspector does — network API first, nodes.conf as the fallback — unless --config names a file. See Inspector for the file format.

Prerequisites

• +API (default): a signed-in operator session for the environment's gateway (orama network use <env>, then orama auth login), and gateways that serve /v1/operator/telemetry.
• +--ssh: nodes must have the orama CLI installed (via orama maint node install, or updated via orama maint push / orama maint rollout), since the monitor runs sudo orama node report --json over SSH; the binary must be at /usr/local/bin/orama on each node. SSH keys come from RootWallet.

Rolling Upgrade Monitoring Protocol

After every node upgrade or restart, you must verify health before proceeding to the next node. This prevents cascading failures across the cluster.

1. +Verify node health — Run orama status report --env <env> --node <ip> and confirm all services are active, RQLite is in Leader or Follower state, and WireGuard peers are connected.
2. +Verify cluster health — Run orama status report --env <env> and confirm the leader is stable, quorum is intact, the WireGuard mesh is fully connected, and all node indices are in sync.
3. +Only proceed to the next node after both checks pass. If a node shows issues, investigate before continuing. Do not skip and hope it resolves itself.

Common Issues

For detailed troubleshooting, see the Troubleshooting guide.