orama node

orama maint node

Node commands for maintainers and for the installer.

Install and stage a release on this machine, simulate the auto-update decision,
recover a cluster that lost its raft quorum, migrate older state, apply gateway
schema migrations, and enroll or unlock an OramaOS node.

orama maint node autoupdate [flags]

Report what this cluster should do with a candidate release.

This answers the question for the values you give it and changes nothing. The
agent that asks it of the cluster's real state, and acts, is 'orama node
autoupdate run'.

The default mode is notify: a newer verified release is reported and not
installed. auto means the node may install, and only when the cluster is
healthy, the release is newer, and the maintenance window is open.

A release that fails TUF verification, including a rolled-back snapshot or
an expired timestamp, is refused. So is a downgrade and a release a previous
health-gate failure marked bad.

A validator (--role validator) is never auto: on auto the decision is skip, with
the reason, and the command exits 0; chain upgrades are staged explicitly with
'orama maint global stage-oramad'.

orama maint node autoupdate run

Run this node's auto-update agent once. orama-autoupdate.timer runs it every
15 minutes on every node; running it by hand does the same thing.

The agent reads the cluster's policy (orama maint cluster settings show): auto-update
off, notify or auto; the channel; the maintenance window; the release repository.
It does nothing unless the cluster stored a release repository and this node
adopted a release root (orama node trust add-root).

It fetches the channel's metadata and verifies it against the adopted root:
every role at its threshold, an unexpired timestamp, a snapshot no older than the
newest this node has accepted, and the channel's own keys for the channel's own
paths. What does not verify is refused, reported in 'orama status', and never
installed.

With notify (the default) a newer release is reported in 'orama status' and
nothing is installed. With auto the node installs it only when

  - the cluster is not degraded and a majority of the raft voters are up;
  - the hour is inside the maintenance window, if there is one;
  - no node has failed the release (a failure anywhere marks the release bad for
    every node, until a newer release supersedes it);
  - it is this node's turn in the rollout plan: followers first, the leader
    last, nameservers spaced, one node at a time;
  - it holds the cluster-wide rollout lock.

The install is 'orama maint node stage-archive --release-only', keeping the release it
replaces, then 'orama node upgrade --restart', then the health gate. If the
upgrade or the gate fails, the previous release is put back and the node is
upgraded onto it again; the release is then marked bad for the cluster. A
validator (a machine that runs the chain) is never installed automatically.

If a run is killed in the middle of an install, the next run finishes it first
(the intent is in /var/lib/orama-autoupdate/install-intent.json), whatever the
policy now says. One agent runs at a time on a machine.

orama maint node enroll [flags]

Enroll a freshly booted OramaOS node into the cluster.

The OramaOS node prints a registration code on its console. Provide that code
along with an invite token. The Gateway pushes cluster configuration
(WireGuard, secrets, peer list) to the node, sealed under the code.

The code is not served over the network. A GET on port 9999 used to return it.

Usage:
  orama maint node enroll --node-ip <ip> --code <code> --token <invite-token> --gateway <url>

--gateway must be an https:// URL: the invite token is a credential and is never
sent in the clear.

The node must be reachable over the public internet on port 9999 (enrollment only).
After enrollment, port 9999 is permanently closed and all communication goes over WireGuard.

orama maint node install [flags]

Install and configure an Orama production node on this machine.
For the first node, this creates a new cluster. For subsequent nodes,
use --join and --token to join an existing cluster.

Run it on the node itself with sudo, or from your own machine with --remote to
drive the install over SSH against --vps-ip. Which of the two happened used to
be decided by whether you had used sudo.

The build archive must be extracted at /opt/orama and signed by a wallet in the
node's trust anchor, /etc/orama/archive-signers. A genesis install creates the
anchor from --operator-wallet (required) before it verifies the archive; a
joining node takes it from the cluster in the join response. With --remote,
--archive names the build: it is verified on this machine against
--operator-wallet before it is uploaded.

orama maint node migrate-conf [flags]

One-time migration: reads nodes from nodes.conf for an environment
and registers each with your wallet via the gateway API. After migration,
these nodes will appear in 'orama nodes' output.

Requires: orama auth login (for API authentication)

orama maint node migrate-raft-id [flags]

Give each node a raft identity that survives an address change.

RQLite defaults a node's raft id to its raft advertise address, so identity has
been a function of routing: give the same machine a new overlay address — a
replacement, a WireGuard re-provision, a 10.0.0.x reassignment — and it mints a
new raft id, joins as a SECOND member, and the old entry stays in the
configuration as a voter nothing can reach. Two such events on a five-voter
cluster leave quorum at 3-of-7 with five live voters; one more failure freezes
the registry.

RQLite cannot rename a member in place, so this is a deliberate migration rather
than something an upgrade does silently. Nodes are migrated ONE AT A TIME. For
each: the quorum arithmetic is checked, the old id is removed from the raft
configuration and tombstoned, the node's local raft state is discarded, and it
rejoins under its libp2p peer id and replicates back from the leader. The next
node is not touched until the previous one is back in the configuration.

Safe to re-run: nodes already on a stable id are skipped, so an interrupted run
continues where it stopped.

Examples:
  orama maint node migrate-raft-id --env testnet --dry-run
  orama maint node migrate-raft-id --env testnet
  orama maint node migrate-raft-id --env testnet --node 203.0.113.4

orama maint node recover-raft [flags]

Recover the RQLite Raft cluster from split-brain failure.

One node's data is kept. Every other node's raft log and database are DELETED
and rebuilt from it. Nothing is backed up: there is no copy to restore from
afterwards, and the deleted nodes' data is gone. Take a backup yourself first
if the surviving node might not be the right one.

What happens:
  1. Stop orama-node on every node
  2. Reset the kept node to a single-member cluster, preserving its data,
     raft log and raft term
  3. Start it and confirm it comes back as Leader with its data intact
  4. Delete raft.db, raft/, db.sqlite (+shm/wal) and wsnapshots (rsnapshots) on every other
     node, and record the kept node as the member each one re-joins
     (data/cluster-membership.json)
  5. Start them one at a time; each pulls a full snapshot from the kept node
  6. Verify cluster health

Which node is kept decides which copy of the data survives. Without --leader
the command reads every node's applied index, keeps the furthest ahead, and
prints what each one reported before asking you to confirm. --leader overrides
that.

Use --leader-raft-addr when quorum is already lost and rqlite is not answering
anywhere, so the leader's raft address cannot be read from the cluster.

This is a DESTRUCTIVE operation. Use --force to skip confirmation.

Examples:
  orama maint node recover-raft --env testnet
  orama maint node recover-raft --env testnet --leader 203.0.113.4
  orama maint node recover-raft --env devnet --leader-raft-addr 10.0.0.1:10101 --force

orama maint node schema [flags]

Schema lifecycle commands.

The gateway binary embeds a set of SQL migrations. Each migration is numbered;
the highest number is the schema version the binary requires. After deploying
a new gateway binary, run 'orama maint node schema apply' on every namespace's RQLite
to bring the schema up to date — otherwise function deploys fail at runtime
with cryptic missing-column errors.

orama maint node schema apply [flags]

Apply every embedded migration not yet recorded in schema_migrations.

Each migration runs as one transaction together with its schema_migrations
row, so it is applied and recorded, or not applied at all. A statement whose
effect is already in place (an existing column, table or index, left by an older
engine that applied migrations statement by statement) is skipped. Any other
error aborts the run at that migration, which leaves no trace; re-running is
safe because each migration is independently versioned.

orama maint node schema status

orama maint node stage-archive [flags]

Verify a build archive against this node's trust anchor, /etc/orama/archive-signers,
and only then replace the archive files under /opt/orama with it.

'orama maint push' runs this on every node with the node's installed orama. The
archive is extracted into a private directory, its manifest signature must
recover to a trusted signer and every file must match the signed manifest;
anything else leaves /opt/orama untouched. The replacement is undone if any
step of it fails, and holds the lock install and upgrade take on /opt/orama.

--trust-signers creates the anchor on a node installed before archives were
signed, and only after the archive has verified against those addresses. It
never changes an existing anchor.

--release-metadata and --release-target opt in to the release root adopted at
/etc/orama/release-root.json. Before anything is extracted, the archive file
must be that target in the TUF metadata: the root signs timestamp, snapshot and
targets, the timestamp is unexpired, the snapshot is not older than the one
recorded in /etc/orama/release-seen.json, and the file has the target's length
and hashes. Any failure refuses the archive; the wallet check is not tried
instead. An archive that passes is then verified against the trust anchor as
above: the release root is required in addition to it, not in place of it.

--release-only is the one case where the release root is enough: the archive
is an unsigned release (the CI build), it must not name signers or a release
root, and the node records in /etc/orama/release-staged.json that it was staged
through the release root, which is what lets 'orama node upgrade' install it.
A channel target ('nightly/orama-...') is a target of the top-level targets
metadata, under its channel's path prefix.

orama maint node unlock [flags]

Manually unlock a genesis OramaOS node that cannot reconstruct its LUKS key
via Shamir shares (not enough peers online).

This is only needed for the genesis node before enough peers have joined for
Shamir-based unlock. Once 5+ peers exist, the genesis node transitions to
normal Shamir unlock and this command is no longer needed.

The encrypted genesis key is written where the node was created, and the
OramaOS agent does not serve it, so --key-file is required. The command used to
try fetching it from the node first, on a path the agent has never served, and
spent ten seconds timing out before telling you to pass the flag.

Usage:
  orama maint node unlock --genesis --node-ip <wg-ip> --key-file <path>

The node must be reachable over WireGuard on port 9998.

orama node

Operate Orama nodes, both the one on this machine and the fleet you own.

Local, run on the node itself and needing root (sudo):
  uninstall, upgrade, start, stop, restart, status, logs, doctor, report, invite,
  trust, tls status

Remote, run from your machine and reaching nodes over SSH:
  list, setup, remove, wipe, dns delegation

Installing a node's software, staging an archive, auto-update, recovery and
migration are maintainer commands: see 'orama maint node'.

orama node dns

orama node dns delegation [flags]

Print exactly the records the operator must create at the parent zone (or
registrar) so the internet reaches this cluster's nameservers: one NS record
per nameserver, and the glue A record that gives each nameserver its address.

Nameserver slots (ns1, ns2, …) are claimed by the --nameserver nodes as they
come up, so which address holds which name is only known to the cluster. This
reads it from the cluster over SSH. Only slots whose glue the cluster has
written are listed — the same set the cluster's own zone publishes.

After the records it asks DNS whether the parent zone returns them, and says
which NS or glue record is missing or points at another address. The answer is
stored on the environment (environments.json), replacing the last result for
the domain. A resolver that cannot answer is reported and nothing is
stored. --json adds "delegated" and "findings" to each domain.

Run it again after adding or removing a nameserver, and update the parent
zone to match. See orama.network/docs/operator/nameserver.

orama node doctor

Run a series of diagnostic checks on this node to identify
common issues with services, connectivity, disk space, and more.

orama node invite [flags]

Generate invite tokens that allow new nodes to join the cluster.
Running without a subcommand creates a new token (same as 'invite create').

orama node list [flags]

List all nodes owned by your wallet. Queries the network API
with your stored credentials, falling back to nodes.conf.

Requires: orama auth login (for API-based resolution)

orama node logs <service> [flags]

Stream the journal of one service on this node.

<service> is an alias or a unit name. A tenant service is a systemd template
instance, so name it in full:

  orama node logs orama-namespace-olric@anchat

--since takes a window rather than a line count, which is what a diagnostic
that greps for a periodic line needs:

  orama node logs node --since -30min | grep 'WireGuard peer sync completed'

Aliases: caddy, cluster, coredns, gateway, ipfs, ipfs-cluster, node, olric, rqlite, turn

orama node remove [flags]

The old path of 'orama remove', which is the command to use: this one runs it
(--force is --yes) and prints a notice. The removal is the same: the quorum
arithmetic for every raft cluster the node votes in, the tombstone, the
retirement and the wipe, plus the chain: a node in the validator set is refused
unless --drop-validator, and a node with the global layer needs --chain-node-id
or --no-chain. See 'orama remove --help'.

Examples:
  orama node remove --env testnet --node 203.0.113.4 --dry-run   # Show the plan only
  orama node remove --env testnet --node 203.0.113.4
  orama node remove --env testnet --node 203.0.113.4 --offline   # VPS already deleted
  orama node remove --env testnet --node 203.0.113.4 --force

orama node report [flags]

Collect all system and service data from this node and output
as a single JSON blob. Designed to be called by 'orama status' over SSH.
Requires root privileges for full data collection.

orama node restart [flags]

Restart all Orama services. Stops in dependency order then restarts.
Includes explicit namespace service restart.
The auto-update timer, and a run of the update agent already in progress, are
stopped for the restart, and the timer is started again once the node is serving,
so the update agent never runs against a node that is down. When the restart
fails or the node does not come to serve, the timer stays stopped (starting it
would fire the agent at once against a node that is down) and the output names
the command that starts it. An interrupt, a terminate or a hangup finishes the
restart, then starts the timer if the node serves; a timer that was stopped stays
stopped.
Use --force to bypass quorum safety check.

orama node setup [flags]

Use "orama setup": it does this for every machine you give it, and the rest of joining the
network as well. This command stays for now and installs the cluster node only.

Bootstrap a fresh VPS into a running Orama node in one command.

Creates an SSH key in rootwallet, installs it on the VPS, uploads the binary
archive, and runs the node install. For the first node, use --genesis to
create a new cluster.

Examples:
  # Genesis node (first node, creates new cluster).
  # Store the VPS login first: rw vault add 203.0.113.4 (username root).
  # --password is a switch; it reads that login. --archive is the path
  # "orama maint build" printed.
  orama node setup --ip 203.0.113.4 --password --env devnet \
    --base-domain orama-devnet.network --role nameserver --genesis \
    --archive /tmp/orama-<version>-linux-amd64.tar.gz

  # From a published release: no checkout, no Go or zig. The root is the
  # release signers' key set you decided to trust; the cluster adopts it.
  orama node setup --ip 203.0.113.4 --password --env mycluster \
    --base-domain cluster.example.com --role nameserver --genesis \
    --release 0.3.1 --release-repo https://releases.example.org/tuf --release-root ./root.json

  # Join existing cluster
  orama node setup --ip 203.0.113.8 --password --env devnet \
    --base-domain orama-devnet.network \
    --archive /tmp/orama-<version>-linux-amd64.tar.gz

  # Key-only VPS (no password login): install the RootWallet key once
  # with the key that opens it today. Do not pass --password as well.
  orama node setup --ip 203.0.113.8 --user ubuntu --bootstrap-key ~/.ssh/id_ed25519 \
    --env devnet --base-domain orama-devnet.network \
    --archive /tmp/orama-<version>-linux-amd64.tar.gz

  # Join as nameserver
  orama node setup --ip 203.0.113.12 --password --env devnet \
    --base-domain orama-devnet.network --role nameserver \
    --archive /tmp/orama-<version>-linux-amd64.tar.gz

orama node start

orama node status

Report the systemd units of the Orama node installed on this machine.

For the health of your whole fleet from your own machine, use 'orama status'.

orama node stop [flags]

Stop all Orama services in dependency order and disable auto-start.
Includes namespace services, global services, and supporting services.
Use --force to bypass quorum safety check.

orama node tls

orama node tls status

Report the certificate of every name this node's Caddy manages: the base
domain and its wildcard, and any site of its own.

For each name it says whether Caddy serves an issued certificate, who issued it
and when it expires. When it does not, it says what the certificate authority
last answered to the order, read from Caddy's log: the ACME problem type (for
example rateLimited), the CA's message and, when the CA gave one, the time it
will accept a new order. A rate limit shows here and nowhere a client can see:
the client's only symptom is "tls: internal error".

Reads the journal of orama-namespace-caddy@index, so it needs root. --json
prints the same as a document with a "names" list.

See orama.network/docs/operator/tls-certificates.

orama node trust

orama node trust add-root <root.json> [flags]

Adopt a TUF release root as /etc/orama/release-root.json.

A node trusts its operator's wallet by default (/etc/orama/archive-signers). A
cluster may also trust a release root: a set of keys whose threshold signature
on release metadata makes an archive installable without the operator building
and signing it. The Orama release root is one; a cluster adopts it by choice
and can drop it by deleting the file.

The root is checked before it is written: well-formed, signed by its own keys at
its threshold, not expired. Adopting a root other than the one already adopted
needs --replace. This command changes this node only; 'orama maint build
--release-root' puts the root in a signed archive, and every node that installs
that archive adopts it.

--rotate adopts the root as the next version of the one adopted: it has to be
signed by the adopted root's keys at their threshold and by its own, exactly as a
client following the release repository checks a rotation. A push of a release
uses it, so a root the operator renewed or rotated reaches this node without
--replace and without anyone's word for it.

Examples:
  sudo orama node trust add-root ./root.json
  sudo orama node trust add-root --rotate ./2.root.json

orama node uninstall

orama node upgrade [flags]

Upgrade the Orama node binary and optionally restart services.
Uses rolling restart with quorum safety to ensure zero downtime.

Run on a node, with sudo, this upgrades that node. Run from your machine with
--env it rolls the nodes of an environment one at a time from the build already
staged on them: that remote mode is replaced by 'orama upgrade' (the newest signed
release of your network's channel) and 'orama maint rollout' (a build of your own),
and prints a notice.

orama node wipe [flags]

Remove all Orama data, services and configuration from remote nodes.
Tor is left installed (its config and state are removed); --nuclear purges it.
The wipe ends by listing what of Orama is still on the machine (LEFTOVER lines) and fails if
anything is; the system accounts are removed only with --nuclear.
Every wipe removes the IPv6 sysctl drop-in and puts IPv6 back to its value from before the install
(enabled when the install recorded none); --nuclear also removes the RAM hygiene drop-in and puts back
the values it recorded.

Target-side only: this says nothing to the cluster. If the node is still a
member, use 'orama remove' instead — otherwise the survivors keep
counting it toward quorum and re-adding its WireGuard peer.

This is a DESTRUCTIVE operation. Use --force to skip confirmation.

Examples:
  orama node wipe --env testnet                      # Wipe every node
  orama node wipe --env testnet --node 203.0.113.4       # Wipe one node
  orama node wipe --env testnet --nuclear             # Also remove shared binaries and accounts