Agent index (llms.txt) — fetch this first, then the page it names.

Troubleshooting

Named failures, what causes them, and how to fix them. Start with the tools, then find your symptom. Always use the orama CLI for diagnostics and service management, never raw systemctl: the CLI handles dependency ordering and safety checks.

Start here

orama status                                   # which node is unhealthy?
orama status alerts --env <env>               # what the cluster thinks is wrong, with a next step for each
orama status node --env <env> --node <ip>   # everything one node reported
orama maint inspect --subsystem rqlite               # deep checks for one subsystem (see Inspector)
sudo orama node doctor                         # on the node: services, ports, connectivity, Raft state
sudo orama node logs <service> -n 100 -f       # logs; service aliases: caddy, cluster, coredns, gateway, ipfs, ipfs-cluster, node, olric, rqlite, turn
sudo orama node logs orama-namespace-olric@<ns> -n 30    # a tenant service by full unit name

General rules:

• +Use sudo orama node restart, not raw systemctl. The CLI handles dependency order, quorum checks and health verification.
• +Namespace data lives at /opt/orama/.orama/data/namespaces/<name>/ on a node.
• +A node reporting degraded is serving. Check which components have not converged before you reach for a recovery command: sudo orama node logs node -n 100 | grep "Boot component".
• +Check WireGuard with wg show wg0 and look for recent handshakes and transfer bytes. Check a gateway's health from the node with curl http://<wg-ip>:<port>/v1/health (tenant gateways bind the WireGuard address, not loopback).
• +Node IPs: orama nodes. SSH keys come from RootWallet, never from a file.
• +OramaOS nodes have no SSH. Use the gateway API (/v1/node/status, /v1/node/logs).
• +If a command in a heredoc SSHes to several nodes and only the first runs, add -n to each ssh: it was reading the rest of your script.

RootWallet agent: locked, waiting or unreachable

orama node setup, push, build, auth approve and anything that signs talk to the RootWallet agent on ~/.rootwallet/agent.sock (override with RW_AGENT_SOCK). The agent answers with a code and the CLI turns it into an instruction:

A first vault run (SSH keys in orama node setup, for example) against a locked wallet can take four minutes (two for approval, two for unlock); a wallet operation such as signing in orama maint build refuses at once instead. If a command seems to hang, look at the desktop app: there is probably a prompt on it. rootwallet agent not reachable: … (is the desktop app running?) means the socket is absent: the desktop app is closed. rootwallet agent is locked — unlock it in the desktop app first means unlock it and run the command again. orama auth login does not need the agent: on a machine without a wallet it prints a code, and orama auth approve <code> on one that has approves it.

Binary signature verification fails

orama maint push, orama maint node install or orama node upgrade refuses the archive and names the cause. The full table is in Build and sign. The ones people hit: built with --unsigned; signed by an account not in /etc/orama/archive-signers; a genesis install without --operator-wallet; a --signers rotation that leaves out its own signer; an archive for the wrong architecture; a join refused with 409 because the cluster trusts other signers than you expected (use --join-via).

Service will not start

sudo orama node status
sudo orama node doctor
sudo orama node logs <service> -n 100
sudo orama node restart              # dependency order and quorum safety

Dependency order matters: WireGuard, RQLite, then Olric, then the gateway. Starting a gateway before Olric is ready causes a permanent initialization failure because the Olric client does not retry; the CLI orders them for you. sudo orama node restart --force bypasses the quorum safety check.

Namespace gateway: "Olric unavailable"

Symptom. https://ns-<name>.<base>/v1/health shows "olric": {"status": "unavailable"}.

Cause. Olric's memberlist gossip between the namespace's nodes is broken. Olric uses UDP pings; if they fail the cluster cannot bootstrap.

First, is it drift the reconciler already fixed? When a namespace member is replaced, the tenant reconciler rewrites configs/olric-<node>.yaml peers and configs/gateway-<node>.yaml olric_servers within 60 seconds. If a config still names a node that is gone, the reconciler is not converging; that is the bug, not the config.

1. +WireGuard packet loss. ping -c 10 -W 2 <peer-wg-ip> from each node. Loss over WireGuard but not over the public IP means a corrupted peer session. The 60-second peer sync should have re-applied it. Check what the node believes before touching it: wg show wg0 dump, sudo grep -A3 '\[Peer\]' /etc/wireguard/wg0.conf, and sudo orama node logs node --since -10min | grep 'WireGuard peer sync completed' (the line reports added, updated, removed, persisted; persisted=false means the mesh is right now but will regress on the next wg-quick up). A hand reset (wg set wg0 peer <pubkey> remove, then re-add with endpoint, allowed-ips <wg-ip>/32 and persistent-keepalive 25, on both sides) is break-glass; if you need it, that is worth reporting.
2. +Olric bound to 0.0.0.0. cat …/configs/olric-*.yaml. A bindAddr of 0.0.0.0 makes dual-stack hosts bind IPv6 and breaks gossip. Fixed in code for new namespaces.
3. +Constant "Failed UDP ping" in orama node logs orama-namespace-olric@<name> -n 30. If UDP fails but TCP succeeds, it is the WireGuard problem in 1.

Namespace gateway: missing config fields

global_rqlite_dsn missing, olric_timeout: 0s, or olric_servers listing only localhost: gateways spawned before a fix. A remote spawn request only says whether the gateway needs a registry; the receiving node fills in its own address. Re-spawning or restarting the gateway fixes it. A node with no registry address refuses the spawn.

Namespace not restoring after a restart (missing cluster-state.json)

After orama node restart the namespace services do not come back and ls /opt/orama/.orama/data/namespaces/<name>/cluster-state.json shows no file. The disk pass runs once at boot. The tenant reconciler then converges from the database every 60 seconds, so a node with no state file recovers as soon as its RQLite has a leader. The state file is not trusted for Raft membership (it is refreshed by a best-effort push, so the node most likely to hold a stale copy is the one that was down). A namespace RQLite holding Raft state restarts on its own configuration, writing peers.json only when live membership differs from what the node recorded. One without Raft state, but with a membership record, joins the others and never bootstraps. After three consecutive sweeps with its RQLite not active, a cluster is marked degraded (rqlite is not running on node <id>), still served.

Namespace services not restarting after an upgrade

orama node stop disables the tenant template services. sudo orama node restart re-enables them before restarting. A tenant service still down after a minute is a real failure: read its unit's logs.

RQLite returns 401 Unauthorized

This release starts rqlited with -auth. Unauthenticated calls are 401 by design. The credentials are orama and /opt/orama/.orama/secrets/rqlite-password. Every client sends them; gateway and SFU config embed them in rqlite_dsn. A 401 during a mixed-version upgrade means an old binary with no credentials is talking to a new rqlited: finish the rolling upgrade, followers first, leader last.

rqlited binds only the node's WireGuard address (discovery.http_adv_address in node.yaml, for example 10.0.0.1:10100); nothing listens on loopback, so a curl localhost:10100 is refused. To query it by hand:

RQ="http://$(sudo sed -n 's/^ *http_adv_address: *"\(.*\)"/\1/p' /opt/orama/.orama/configs/node.yaml)"
rqcurl() { printf 'user = "orama:%s"\n' "$(sudo cat /opt/orama/.orama/secrets/rqlite-password)" | curl -K - "$@"; }
rqcurl -sS "$RQ/status" | jq -r '.store.raft.state, .store.raft.num_peers'

The credentials go to curl on stdin so they never appear in ps.

"served by this node … has no rqlite.env"

orama node upgrade stops with namespace(s) <ns> are served by this node … but … has no rqlite.env for them. The env file that says where the instance listens is missing from /var/lib/orama-unit-env/<ns>/rqlite.env. If orama-node is still moving the old layout its log names the path it stopped on (Legacy layout: …); resolve that and it retries. If the namespace no longer runs here, its directory is a leftover from an unfinished teardown: remove the namespace through the normal deprovision path, and do not delete cluster-state.json by hand while it is still assigned here.

Raft quorum issues

orama status report --env <env>              # quorum, leader, indices
orama maint inspect --subsystem rqlite
sudo orama node logs rqlite -n 100

RQLite needs a majority of its voters. If quorum is lost the database stops accepting writes. To recover:

1. +Identify which nodes are down: orama status report --env <env> shows every node's status.
2. +Bring them back: start the offline nodes with orama node start.
3. +Wait for a leader. Once a majority is available Raft elects one by itself.
4. +Verify: orama status cluster --env <env> shows the leader and the log indices in sync.

A node stuck in Candidate, or "leader not found" everywhere, with the whole cluster leaderless and quorum permanently lost (for example a majority of the nodes destroyed): orama maint node recover-raft. It is destructive. It keeps one node's data and rebuilds every other node's raft log and database from it, with no backup. Take a backup first if the survivor might be wrong, and find the node with the highest applied index first (orama maint inspect --env <env> --subsystem rqlite). A single node reporting degraded is not a reason to recover.

orama maint node recover-raft --env <env>                      # keep the node with the highest applied index (it prints what each node reported)
orama maint node recover-raft --env <env> --leader <node-ip>   # keep a node of your choice
orama maint node recover-raft --env <env> --leader-raft-addr 10.0.0.1:10101 --force   # quorum lost and rqlite answers nowhere

Index RQLite refuses to start: "holds raft state but no raft-node-id"

The node predates recorded Raft ids and was upgraded without this release's CLI. Find the node's id in the configuration on a live member (orama node report, or /nodes on a voter: the entry whose addr is this node's WireGuard IP), write it as the orama user to /opt/orama/.orama/data/rqlite/raft-node-id, and write the address it lists to raft-adv-addr beside it. orama-node retries on its own. Upgrade the remaining nodes with orama node upgrade --env <env> --yes from this release's CLI.

Index RQLite: "no other member to join so that the leader re-registers it"

After an upgrade that moved the index Raft port, the node's address changed and RQLite only moves a member to a new address when it joins the leader again; this node has nobody recorded to join. On a cluster of one, reform it: orama maint node recover-raft --env <env> --leader-raft-addr <wg-ip>:10101. On a larger cluster set database.rqlite_join_address in node.yaml to a live member's Raft address and let orama-node retry.

Namespace RQLite has no leader

A namespace's RQLite lost quorum (its other voters are gone). If the namespace has a single live host, as the orama user on that host: stop the namespace's gateway and RQLite units (the CLI has no per-tenant stop, so the exact commands are in Replace a node), back up raft/peers.info, write raft/peers.json as [{"id":"<this node's raft advertise address>","address":"<same>","non_voter":false}], start the RQLite unit and wait until /status shows Leader. Then fix the namespace's Olric config (memberlist.peers: []) and the gateway's olric_servers to name only the live node, restart those, and check /v1/health. Re-provision the namespace to three nodes afterwards so it is not single-node for ever. orama node remove --dry-run shows this risk before you cause it.

Namespace RQLite refuses to bootstrap: "this node has been a member … but holds no raft state"

The node was a member of the namespace and lost its RQLite data, and there is no other member to join. Bootstrapping would create an empty namespace database. Restore the node's data, or, if the namespace's data is gone everywhere and an empty one is what you want, delete the record named in the log (data/namespaces/<ns>/rqlite/cluster-membership.json); the next restore pass bootstraps.

Namespace RQLite crash-loops: "cannot replay WAL files: existing WAL file present"

orama-namespace-rqlite@<ns> restarts every few seconds; raft/peers.json is present, and the data directory holds recovery.db-wal, recovery.db-shm (and usually restore-wal-N.tmp) from an earlier recovery. The data is intact. A node on this release cannot reach the state; one that already has it needs one node at a time (never two voters): stop the unit with the CLI, as orama delete recovery.db* and restore-wal-*.tmp from …/data/namespaces/<ns>/rqlite/<node-id>/, keep raft/peers.json, start the unit. If peers.json lists members that no longer exist, correct it first: it replaces the node's Raft configuration.

A node never catches up: "no WAL data available for snapshot"

The monitor raises a critical "Raft snapshot term N is above the current term M" and one node's applied index stays put. It came from recover-raft run before 2026-10-03, which deleted the leader's raft.db and so restarted the cluster at term 1 while the recovery snapshot kept the old term. Do not restart RQLite or roll an upgrade until it is repaired: a restart restores the stale snapshot and loops. The repair uses RQLite's HTTP API over SSH against overlay addresses only, with credentials on stdin: (1) GET /db/backup from the leader as a safety copy, written mode 0600 and never copied off the node; (2) POST /leader?wait (stepdown) until the term reaches N-1 or higher; (3) take a fresh backup and POST /db/load it straight back, which makes every node's next snapshot a full one; (4) stepdown once more and POST /snapshot on every node; (5) check each node's newest snapshot contains data.db, then POST /reap on every node to remove the stale one. Writes made between the backup and the load are rolled back, so do it when nothing is writing. recover-raft now keeps raft.db, so a recovery cannot leave this state.

WireGuard connectivity

Symptoms: orama status mesh --env <env> shows a peer with a stale or no handshake; Olric or RQLite complain about peers. Diagnose with orama status mesh --env <env>, orama maint inspect --subsystem wg, wg show wg0, and a ping across the overlay. A handshake within the last two minutes is healthy; stale or missing handshakes mean a connectivity problem. To resolve it:

1. +Check the firewall: UDP 51820 must be open in both directions.
2. +Check the provider's firewall. Some VPS providers have an external firewall panel separate from ufw.
3. +Restart through the CLI: orama node restart handles the full dependency chain.
4. +Verify: orama status mesh --env <env> shows every peer reconnected.

See WireGuard.

RQLite replication lag

Symptoms. A follower's Raft log index is lower than the leader's, queries return stale data, replication lag warnings in the logs. Compare the nodes with orama maint inspect --env <env> --subsystem rqlite and orama status cluster --env <env>.

• +Small lag is normal under a high write load; the follower catches up.
• +Large lag: check the network between the follower and the leader with orama status mesh --env <env>.
• +A node that fell too far behind may need to rejoin the cluster with a fresh snapshot.

Olric cluster problems

Symptoms. Cache misses on data that should be cached, "member not found" errors in the gateway logs, Olric restarting often. Inspect with orama maint inspect --env <env> --subsystem olric and sudo orama node logs olric.

• +Olric cannot find its peers: check WireGuard; Olric discovers peers over the mesh.
• +A crash loop: check that RQLite is healthy first, because Olric depends on it.
• +Stale cluster membership: orama node restart restarts in dependency order.

Olric cluster split after enabling encryption

Olric memberlist encryption is all-or-nothing: nodes with encryption cannot talk to nodes without it. All nodes must restart together when enabling it; the cache is lost and rebuilds from the database. This is expected: Olric is a cache.

IPFS Cluster: pins never replicate

ipfs-cluster runs and /v1/health shows ipfs: ok, but content pinned on one node never appears on the others. The cluster secret differs from the rest of the fleet (it is the key to the cluster's libp2p private network, so a node with another value completes no handshake). Compare sudo head -c 8 /opt/orama/.orama/secrets/cluster-secret; echo on every node (never paste the whole value into a ticket). They must be identical. Copy the value from a healthy node and restart orama-node. The node now refuses to start ipfs-cluster when the secret is unreadable or not 64 characters, instead of generating a new one.

IPFS repo garbage collection times out on every run

orama-namespace-ipfs-gc@index fails with a timeout after 30 minutes, the monitor raises "IPFS pin/add active for …", and the repo never shrinks. Kubo runs pin/add and repo gc under one lock; a recursive pin of content no peer holds never finishes and GC waits behind it. The pin ends at ipfs-cluster's pin_timeout (2 minutes without progress). service.json must say "node_multiaddress": "/ip4/127.0.0.1/tcp/10110"; a /unix/… value means the node was not upgraded with this release's CLI. Upgrade the node (orama node upgrade). To see which request holds the lock, and whether its content exists anywhere (never print the command line of diag cmds -v unfiltered: the GC oneshot's options carry the Kubo bearer):

sudo bash -c 'set -a; . /var/lib/orama-unit-env/index/ipfs-gc.env; set +a; HOME=/opt/orama \
  ipfs --api=$IPFS_API --api-auth=$IPFS_API_AUTH diag cmds' | grep -E 'pin/|repo/gc'
# for the CID of a stuck pin/add, on every node, with the same --api/--api-auth:
ipfs block stat --offline <cid>; ipfs refs -r --offline <cid>

A CID whose blocks exist on no node cannot be pinned by anything; unpinning it is the content owner's decision.

IPFS daemon crash-loops: "error loading plugins: invalid character '\x00'"

The Kubo repo config is zero-filled (a crash or reboot before an unsynced rewrite reached disk; fixed in 0.122.113, so every 0.3.0 release has the fix). The node's IPFS identity lives in that file and is lost. orama node upgrade refuses the corrupt config and says so. Stop the node, move the repo directory it names aside, run the upgrade on that node again. A new repo and identity are created, the other nodes re-learn it, and IPFS Cluster re-pins this node's replicas from the others.

Function WASM timeouts, or deploy 504, after a node replacement

Symptom. failed to fetch WASM: wasm fetch from IPFS timed out (about 15 seconds) on invoke after a node was replaced or restarted; other functions still succeed in milliseconds; orama function deploy returns 504 TIMEOUT while /v1/health still shows ipfs: ok.

Cause. Function metadata is in the namespace RQLite; function bytes are IPFS blobs. A new or wiped node starts with an empty Kubo repo, and until every active wasm_cid is pinned locally, cold cat and add can hang. Health only proves the daemon is up. Content now re-replicates on its own (a gateway elected through the ipfs-pin-sweep lock re-issues short pins every 15 minutes), but a fresh node needs a few sweeps.

Fix. Wait for the sweep, or pin the active function CIDs on every nameserver (pin/add with recursive=true against the node's Kubo API). Verify a hot function's cat is fast on the new node and a roughly 1.2 MB add succeeds on all peers. Do not call a cutover complete because the registry Raft group is 3/3. The full procedure (export the active CIDs, pin on every nameserver, verify) is in Replace a node.

A 504 on a slow client's large upload

A storage upload, pin, function deploy or invocation whose body takes more than a minute to arrive was cut off at 60 seconds before 0.3.0. Every long-budget route now gets five minutes to read and write, in the proxy and on the namespace gateway, once the caller's credential is validated. Both gateways must run this build. A body that still does not arrive in five minutes is a client too slow for the route. An anonymous call to a public function keeps the server's 60 and 120 second limits.

A deployment made on 0.122.x is down after upgrading

0.122.x ran each deployment from a unit of its own. The upgrade moves its files to data/deployments/<ns>-<name>/ and removes the old unit; the template unit that replaces it needs a workload token only the gateway can mint, so nothing starts it by itself. Delete the deployment and create it again. --update is not enough.

Certificates

See TLS and certificates and DNS and nameservers.

OramaOS

LUKS unlock fails after a reboot. Not enough peer vault guardians are online to meet the Shamir threshold K = max(2, floor(N/3)). Make sure enough nodes are online and reachable over WireGuard; the agent retries with backoff. For a genesis node before 5 peers exist, orama maint node unlock --genesis --node-ip <wg-ip> --key-file <path> takes a RootWallet-encrypted key file, but nothing produces one yet because key escrow is not built (see "Not yet implemented" in OramaOS). The only unlock that works today is the raw-key curl to port 9998 described there, and that port is a known issue.

orama maint node enroll times out. Port 9999 on the node is not reachable from the gateway, --code is missing or wrong, or --node-ip is not the node's public IPv4. Read the registration code from the node's console (it is not served on port 9999) and open 9999 in your provider's external firewall.

See OramaOS.

Recovering from a failed join

If a node partially joins (it registers in the registry's Raft group, then fails or is wiped), the remaining cluster can lose quorum permanently: RQLite thinks there are N voters but only N-1 are reachable. The symptom is RQLite stuck in Candidate, no leader, every write failing. If the cluster is still healthy, remove the half-joined node with orama node remove --env <env> --node <ip> and join again with a new invite (invites are single use, and a retry is told the invite was already used). If quorum is already lost, use orama maint node recover-raft; on a young cluster with no data worth keeping, a clean reinstall of the affected nodes from the genesis node is the alternative (Wipe a node).

Prevention. Make sure a joining node can complete the whole installation before it joins. The installer does not check for port conflicts up front, so stop anything holding the ports the node needs (for example 8080 and 4001) before you join. See the pre-install checklist in What you need.

General debugging workflow

1. +Run sudo orama node doctor for a quick local diagnostic.
2. +Run orama status report --env <env> for the full cluster picture.
3. +Read the logs of the affected service with sudo orama node logs <service>.
4. +Verify the dependencies are healthy: WireGuard, RQLite, Olric, gateway.
5. +If it looks like the network, check orama status mesh --env <env> and the firewall rules.
6. +For deep diagnostics, run orama maint inspect --env <env>, or one subsystem with --subsystem.
7. +Restart the affected service with orama node restart.
8. +If the problem persists, save a full report with orama maint inspect --env <env> --output ./results and escalate.

To erase a node that is in a bad state and start fresh, use orama node wipe, or take it out of the cluster and erase it with orama remove (Wipe a node).