Agent index (llms.txt) — fetch this first, then the page it names.

Inspector

The inspector is a cluster health check tool that SSHs into every node, collects subsystem data in parallel, runs deterministic checks, and optionally sends failures to an AI model for root-cause analysis.

Pipeline

Collect (parallel SSH) → Check (deterministic Go) → Report (table/JSON) → Analyze (optional AI)

1. +Collect — SSH into every node in parallel, run diagnostic commands, parse results into structured data.
2. +Check — Run pure Go check functions against the collected data. Each check produces a pass/fail/warn/skip result with a severity level.
3. +Report — Print results as a table (default) or JSON. Failures sort first, grouped by subsystem.
4. +Analyze — If --ai is enabled and there are failures or warnings, send them to an LLM via OpenRouter for root-cause analysis.

Quick Start

# Inspect all subsystems on devnet
orama maint inspect --env devnet

# Inspect only RQLite
orama maint inspect --env devnet --subsystem rqlite

# JSON output
orama maint inspect --env devnet --format json

# With AI analysis
orama maint inspect --env devnet --ai

Usage

orama maint inspect [flags]

Subsystem Names

rqlite, olric, ipfs, dns, wireguard (alias: wg), system, network, namespace, tor, webrtc, global

Multiple subsystems can be combined: --subsystem rqlite,olric,dns

Subsystems

Collection Failures

How a node is collected. The sessions of one node share one SSH connection (ControlMaster, sockets in a private /tmp/orama-inspect-* directory removed at the end), and its collectors run side by side, at most four at a time (WireGuard first, because the network probe pings its peers). Setting up an SSH session is most of what a collector costs: on a CPU-starved VPS it took 1-4s against 0.4s over an open connection, so the 14 sessions of one node ran for 49s one after another, longer than --timeout, and every subsystem of the node was reported not collected. Measured on that node, the same collection takes 9s. A collector that panics is recorded as that subsystem's failure, and Ctrl-C or SIGTERM cancels the run and closes the connections before the command exits with an error. Service states are read with one systemctl is-active call, and the Olric journal once.

Olric log checks. olric.log_failed_members (critical) fails when memberlist marked a member failed (Marking <member> as failed) in the last hour: that is its verdict that the member is gone. olric.log_suspicions (warning) counts suspicions (Suspect <member> has failed, no acks received, Refuting a suspect message): the first step of the protocol, refuted when the member answers; a member that keeps being suspected answers late (a starved or partitioned node), which is worth a look and not a failed cluster.

The inspector never reads a failed collection as zero values. Before collecting, it opens one SSH session (true) to each node. If that session fails (a connection reset from an overloaded node, a timeout, a refused key), the node is unreachable: nothing is collected from it and it produces one critical result, node.reachable (subsystem node), could not collect from <node>: <error>; every check on this node was skipped. No wireguard, network, system or dns check runs against it, so there is no "wg0 down", "no default route" or "2/3 nameservers active" from a node the inspector simply could not read.

Each subsystem is collected over its own SSH session. When one of them returns no output, or output cut short of the fields the collector reads, only that subsystem is affected: its data is left unset, its checks on that node are skipped, and one critical <subsystem>.collected result (for example dns.collected) reports why. A namespace collection failure is reported as both namespace.collected and webrtc.collected. The other subsystems on the node are checked normally, and cross-node checks count only the nodes whose data was collected. A command that ran and reported a service inactive is still a normal failure of that service's check.

Every collection failure is also listed under "Collection Errors" in the results summary and handed to the AI analysis.

Severity Levels

Check Statuses

Output Formats

Table (default)

Inspecting 14 devnet nodes...

## RQLITE
----------------------------------------------------------------------
  OK [CRITICAL] RQLite responding (ubuntu@10.0.0.1)
    responsive=true version=v8.36.16
  FAIL [CRITICAL] Cluster has exactly one leader
    leaders=0 (NO LEADER)
  ...

======================================================================
Summary: 800 passed, 12 failed, 31 warnings, 0 skipped (4.2s)

Failures sort first, then warnings, then passes. Within each group, higher severity checks appear first.

JSON (--format json)

{
  "summary": {
    "passed": 800,
    "failed": 12,
    "warned": 31,
    "skipped": 0,
    "total": 843,
    "duration_seconds": 4.2
  },
  "checks": [
    {
      "id": "rqlite.responsive",
      "name": "RQLite responding",
      "subsystem": "rqlite",
      "severity": 3,
      "status": "pass",
      "message": "responsive=true version=v8.36.16",
      "node": "ubuntu@10.0.0.1"
    }
  ]
}

AI Analysis

When --ai is enabled, failures and warnings are sent to an LLM via OpenRouter for root-cause analysis.

# Use default model (kimi-k2.5)
orama maint inspect --env devnet --ai

# Use a different model
orama maint inspect --env devnet --ai --model openai/gpt-4o

# Pass API key directly
orama maint inspect --env devnet --ai --api-key sk-or-...

The API key can be set via:

1. +--api-key flag
2. +OPENROUTER_API_KEY environment variable
3. +.env file in the current directory

The AI receives the full check results plus cluster metadata and returns a structured analysis with likely root causes and suggested fixes.

Exit Codes

Configuration

By default the inspector resolves nodes the same way every other command does: it asks the network API for the nodes your wallet owns, falling back to nodes.conf when the API is unreachable. It used to read scripts/nodes.conf relative to the working directory, so an installed binary only worked from inside the source tree.

--config overrides that with a pipe-delimited file. nodes.conf is gitignored — copy core/scripts/nodes.conf.example to core/scripts/nodes.conf, or put it at ~/.orama/nodes.conf where it does not depend on the working directory.

Format

# environment|user@host|role
devnet|ubuntu@1.2.3.4|node
devnet|ubuntu@5.6.7.8|nameserver-ns1

SSH keys are resolved from the RootWallet agent over its Unix socket, not by shelling out to rw. See Troubleshooting.

Blank lines and lines starting with # are ignored.

Node Roles

• +node — Regular cluster node. Runs RQLite, Olric, IPFS, WireGuard, namespaces.
• +nameserver-* — DNS nameserver. Runs CoreDNS + Caddy in addition to base services. System checks verify nameserver-specific services.

Examples

# Full cluster inspection
orama maint inspect --env devnet

# Check only networking
orama maint inspect --env devnet --subsystem wireguard,network

# Quick RQLite health check
orama maint inspect --env devnet --subsystem rqlite

# Verbose mode (shows collection progress)
orama maint inspect --env devnet --verbose

# JSON for scripting / piping
orama maint inspect --env devnet --format json | jq '.checks[] | select(.status == "fail")'

# AI-assisted debugging
orama maint inspect --env devnet --ai --model anthropic/claude-sonnet-4

# Custom config file
orama maint inspect --config /path/to/nodes.conf --env testnet