Agent index (llms.txt) — fetch this first, then the page it names.

Wipe a node

How to take Orama off a server so it can be installed fresh, or so it can leave the cluster for good.

Use the CLI, and pick the right command

SituationCommand
The node is, or was, a member of a running clusterorama remove (from your computer; it uses a survivor)
The node is already retired from the cluster and you only need the machine erasedorama node wipe
You are rebuilding every node of a test clusterorama node wipe --env <env>
The server is already deleted at your providerorama remove --offline
The CLI cannot reach the machine (lost SSH key, never enrolled)The manual steps below

wipe erases the target and says nothing to the cluster. If the node is still a member, the survivors keep counting it toward quorum and keep re-adding its WireGuard peer. That is why remove exists: it checks that no cluster loses quorum, retires the node from Raft, the mesh, every namespace it served and the node registry (and, with --chain-node-id, from the chain), and only then erases it. It refuses a node that signs for the validator set unless you pass --drop-validator.

bash
orama remove --node 203.0.113.12 --dry-run   # the plan, and the quorum arithmetic
orama remove --node 203.0.113.12
orama remove --node 203.0.113.12 --offline   # machine already gone
orama node wipe   --env mycluster --node 203.0.113.12             # erase only
orama node wipe   --env mycluster --node 203.0.113.12 --nuclear   # also remove shared binaries, the Tor package and Orama's system accounts

--yes (--force for wipe) skips the confirmation. Both are destructive. A wipe uses the node's RootWallet SSH key; afterwards that key is deleted from your vault, because a retired node's key opens nothing. If the vault cannot be reached the command says so and fails: run it again once RootWallet is unlocked.

What a wipe removes

CategoryWhat
App data/opt/orama/.orama/ (configs, secrets, logs, IPFS, RQLite, Olric)
Binaries/opt/orama/bin/…
Systemdorama-node, every orama-namespace-* instance, leftover orama-* units, coredns, caddy, orama-deploy-*, the privileged-helper socket and service. Everything named orama-* under /etc/systemd/system goes too: the drop-in directories deployments leave (orama-deploy-build@<namespace>.service.d and its siblings) and the .wants links. A unit that outlives its file (a transient orama-namespace-wireguard@index, listed not-found active exited) is stopped and its failed state reset, then systemd reloads
Privileged helper/usr/local/bin/orama-privhelper and its socket
WireGuard/etc/wireguard/wg0.conf and the wg-quick@wg0 unit
FirewallThe ufw rules Orama added: those tagged orama (the cluster) and orama-global (a global node, route rules included), and the mesh accept rule in iptables. Your own rules and the rules for the ports sshd listens on stay, and ufw stays enabled as it is: the wipe runs over SSH. If sshd -T cannot be read no rule is touched, and the check below reports it
AccountsOnly with --nuclear: the system accounts Orama created below uid 1000 (the orama user, orama-* such as orama-sfu, orama-chain and orama-ipfs-pub, and ntfy) with their groups, after their processes are killed. A plain wipe keeps them, and a reinstall reuses them
Kernel settings/etc/sysctl.d/99-orama-disable-ipv6.conf, on every wipe: the installer turns IPv6 off at every boot with it. The wipe sets net.ipv6.conf.all.disable_ipv6 and net.ipv6.conf.default.disable_ipv6 back to what the install found (each drop-in ends with # prior: <key>=<value> lines recording it), or to 0 (IPv6 enabled, the kernel default) when the install recorded none. Only with --nuclear: /etc/sysctl.d/99-orama-ram-hygiene.conf (suid dumps, core pattern, ptrace scope), with fs.suid_dumpable, kernel.core_pattern and kernel.yama.ptrace_scope set back to the recorded values; a machine installed before the values were recorded keeps its running values until the next boot, which no longer reads the drop-in
Configuration directory/etc/orama itself, not only the files in it
CoreDNS and Caddy/etc/coredns/Corefile, /etc/caddy/, /var/lib/caddy/ (certificates live in the cluster's shared store, not here)
ntfy/etc/ntfy/ and /var/lib/ntfy/; the ntfy binary and account go with --nuclear
Global layer/var/lib/orama-global (the chain with its keys, public storage, the relay), /usr/lib/orama-global (its binaries), /etc/orama-global and /etc/sysctl.d/60-orama-global-netns.conf, on every wipe. Stopping its units removes the orama-global network namespace, its ogl-host veth and its orama_global nft table, and the check below looks for each
TorThe unit, /etc/orama/tor/torrc, /var/lib/orama-tor/ (the tor package stays unless --nuclear), and on a directory authority of a created network its keys in /var/lib/orama-tor-authority (the identity key and its passphrase), on every wipe
Anyone network leftoversOn a node never upgraded past it
Trust and update stateThe archive trust anchor /etc/orama/archive-signers and its rotation mark, so a machine set up again trusts the cluster it next joins or creates, not this one. The auto-update timer is stopped first, then /var/lib/orama-autoupdate (fetched releases and the install intent), /etc/orama/update-notice.json and the adopted release root with its state (release-root.json, release-seen.json, release-staged.json in /etc/orama) are removed
Temporary files/tmp/orama, build directories

rm -rf is unlink, not cryptographic erase. Assume a decommissioned provider disk is readable.

The check after the wipe

The wipe ends by listing what of Orama is still on the machine: the paths above, anything named orama-* under /etc/systemd/system, units still listed by systemd, the wg0 interface and its iptables rule, the ufw rules tagged orama or orama-global that are not SSH rules, a kernel setting that could not be put back (LEFTOVER: sysctl <key> ...), and with --nuclear the binaries, the RAM hygiene drop-in and accounts. Each is printed as LEFTOVER: <what>. If there is any, the script says wipe INCOMPLETE and fails, and so does the command: it never reports success over state it could not remove. Fix the cause, then run the wipe again.

Not removed without --nuclear: the RAM hygiene drop-in /etc/sysctl.d/99-orama-ram-hygiene.conf, the system accounts, binaries in /usr/local/bin (rqlited, ipfs, ipfs-cluster-service, olric-server, coredns, ntfy, orama, orama-node, gateway, identity, sfu, turn, orama-sni-router), Caddy in /usr/bin, and the tor package with its apt source and key.

OramaOS nodes

This applies to Ubuntu-based nodes only. OramaOS has no SSH and no shell. To remove an OramaOS node call POST /v1/node/leave on the gateway API for a graceful departure, or reflash the image at your provider for a factory reset. See OramaOS.

Manual cleanup (when the CLI cannot reach the machine)

Only for a node whose SSH key is lost or that was never enrolled. Over your own SSH session, as root, and in this order:

  1. +Stop and disable orama-autoupdate.timer and its service (an install in progress would restart the services you are stopping), then the privileged helper's socket, then orama-node, then every orama-namespace-* unit, then any leftover orama-*, coredns, caddy, ntfy and wg-quick@wg0 units. Kill leftover orama-node, ipfs, ipfs-cluster-service, olric-server, rqlited, coredns and vault-guardian processes.
  2. +Delete everything named orama-* under /etc/systemd/system (units, timers, .service.d drop-in directories), coredns.service, caddy.service, and systemctl daemon-reload; then systemctl stop and systemctl reset-failed any orama-* unit still listed, and reload again.
  3. +Take WireGuard down (wg-quick@wg0 is a oneshot that stays "active (exited)" until you stop the unit) and delete /etc/wireguard/wg0.conf.
  4. +Delete the ufw rules tagged orama or orama-global, highest number first. Leave your own and the SSH rule.
  5. +Delete /opt/orama, the Orama environment files, /etc/orama, /etc/coredns, /etc/caddy, /var/lib/caddy, /etc/ntfy, /var/lib/ntfy, /var/lib/orama-global, /usr/lib/orama-global, /etc/orama-global, /etc/sysctl.d/60-orama-global-netns.conf, /var/lib/orama-tor, /var/lib/orama-tor-authority, /var/lib/orama-autoupdate.
  6. +Read the # prior: lines at the end of /etc/sysctl.d/99-orama-disable-ipv6.conf and /etc/sysctl.d/99-orama-ram-hygiene.conf, set each key back with sysctl -w (IPv6 to 0 when there is no line), then delete both files.
  7. +Delete the orama user and the other orama-* system accounts and their groups (services must be stopped first or userdel fails).

Then the machine is ready for orama node setup.

Multi-node: every node of a test environment

bash
orama node wipe --env mycluster            # every node in the environment
orama node wipe --env mycluster --nuclear

Bringing a removed machine back

remove revokes the node's key: its row in node_credentials keeps its node_id and gains a revoked_at, so the machine's disk stops being a working credential and cannot re-admit itself. Re-admitting it is a join. Mint an invite and install as you would a new node (Joining nodes). The join clears the revoked row, and the machine records a fresh key on its next start. No manual SQL.

The same applies to a node that lost secrets/node-key.pem while keeping its peer identity: its new key will not match the one on record, the cluster will refuse it and the gateway log will say so. Re-join it; do not repair the row by hand.