Agent index (llms.txt) — fetch this first, then the page it names.

Release and rollout

How code gets from a checkout onto nodes. For installing a cluster the first time see Install a cluster from scratch; for the full upgrade protocol see Upgrades. This page is the contributor's view.

The model

Nodes never compile. You build one signed archive on your computer and every node unpacks it. A new version is a new archive and a rolling restart: extract can be parallel, restarts are serial, with a health gate between nodes.

The commands

cd core
orama maint build                                     # signed archive; prints its path
orama maint push --env <env> --archive <path>         # upload and verify on every node, restart nothing
orama node upgrade --env <env>                  # read every node's state, print the plan
orama node upgrade --env <env> --yes            # execute it
orama maint rollout --env <env> --yes                 # build + push + upgrade in one
orama maint rollout --env <env> --no-build --archive <path> --yes

A network whose release channel is published (stagenet follows nightly) is upgraded with orama upgrade instead: it takes the newest signed release, stages it on every node and rolls it one node at a time (Upgrades). The commands above are for a build of your own. --archive is required for push: the newest archive in /tmp may be another checkout's build.

Upgrading from 0.122.x the first time: see First upgrade from 0.122.x.

RootWallet must be open and unlocked for the whole run. The first run after a rebuild asks you to approve the new orama binary once (approval is keyed on the binary's hash).

Reproducible builds

Two orama maint build runs of one commit produce the same archive byte for byte, so the people who sign a release can each rebuild it and compare hashes before they sign. A release build sets the date from the commit: export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct). Go binaries are built with -trimpath, -buildvcs=false, a stripped linker and -mod=readonly. The third-party programs (Olric, IPFS Cluster, Caddy) are built from modules checked in to the repository, each with a go.sum, and CoreDNS, Kubo and RQLite are refused unless they match a pinned commit or SHA-256. The archive lists its entries in a fixed order with fixed owners, modes and times. A change to a pinned program means changing its version constant and its module together; a test holds the two. CI builds each published release's archive twice per architecture, fails if the two differ, and attaches the archive and its SHA-256 to the release. CI holds no signing key.

What else fixes the bytes: the build date is SOURCE_DATE_EPOCH (seconds since the epoch) when it is set, else the time of the build, and a value that is not a count of seconds fails the build; the date goes into the manifest and the binaries' version strings. The vault is built with zig build-exe -fstrip, because unstripped it holds the path of zig's per-build cache directory. The archive's entries carry the build date as their time, root as owner and mode 0755 (executable files and directories) or 0644, and its gzip header names nothing and carries no time. GOFLAGS, GOSUMDB, GONOSUMDB, GONOSUMCHECK, GOPRIVATE, GONOPROXY and GOINSECURE are removed from the environment of every go command the build runs, so the environment cannot switch verification off. CoreDNS is cloned at its tag and refused unless the checkout is the commit pinned in constants.CoreDNSCommit; its own go.sum and go.mod decide its dependencies, so nothing is upgraded to @latest. The version each module requires is the constant in core/pkg/constants/versions.go, and the digests of Kubo and RQLite are in constants/release_digests.go.

To bump a third-party program, change its constant, then in its module directory run go get -tool <package>@<version> (in caddy/, go get github.com/caddyserver/caddy/v2@v<version>) and commit the go.mod and go.sum. IPFS Cluster v1.1.6 requires a cockroachdb/swiss that does not compile with Go 1.27; its module pins a newer one, which is why core/thirdparty/ipfs-cluster/go.mod lists it. Beyond the pinned program, a module lists the fixed versions of the libraries that govulncheck reports reached (pion/dtls, quic-go, webtransport-go in IPFS Cluster's; golang.org/x/net, grpc, OpenTelemetry in Caddy's): the e2e scanners test runs govulncheck over caddy/ and over each pinned program's main package, so a new advisory fails the gate until the module is bumped or the advisory is accepted with a reason in e2e/features/scanners/govulncheck-accepted.yaml.

Releases, TUF and auto-update

A release is trusted when the DeBros TUF release root has signed the metadata that names its hash, and it needs no wallet signature on any node. All release keys are in the owner's RootWallet: one release key, listed in the root for the root, timestamp, snapshot and targets roles at threshold 1. Every signature is a human approval in the RootWallet desktop app, so a release is a short session at the owner's machine.

• +The root is adopted on a node by setup (the network's manifest pins its SHA-256 and the CLI embeds it), with orama node trust add-root, or by an archive built with orama maint build --release-root <root.json>. Nothing fetches a root by itself. After that the root rotates through numbered files (2.root.json, 3.root.json), each signed by the previous root's keys, which every node follows.
• +The repository is a static HTTPS site, https://releases.orama.network/: the metadata is served as files and targets/<path> answers 302 to a GitHub release asset of DeBrosDAO/orama. A channel is a path prefix in the one targets file: nightly/ (stagenet), main/ (the future testnet) and dev/<branch>/. A client refuses metadata signed under another root or below a threshold, an expired timestamp, a snapshot older than the newest one it has accepted (rollback protection), and a target whose bytes differ.
• +Every node runs orama-autoupdate.timer, which does nothing until the cluster stored a release repository and the node adopted a root. The cluster settings auto-update (off, notify by default, auto), update-channel, update-window and release-repo set the policy. With auto inside the window, nodes install the newest verified release one at a time under a cluster lock, behind the same health gate as a manual rolling upgrade; if the upgrade or the gate fails, the node puts the previous release back and the release is marked bad for the whole cluster, which stops the rollout.
• +A machine that runs the chain is never auto. Its binary and verifier go in through orama maint global stage-oramad --upgrade <plan>.
• +Auto-update installs only what the release root's key signed; it never signs anything.

Cutting a release (maintainers)

Build the archive unsigned, from a clean checkout, with the global layer (orama maint build --unsigned, amd64 and arm64 if both are needed), then:

# once: the repository's root. Prints the digest network manifests pin.
orama maint release init-root --dir ~/.orama/release-repo

# each release: list the archives on a channel and sign (3 approvals)
orama maint release cut --channel nightly \
  --archive /tmp/orama-0.3.1-linux-amd64.tar.gz --dir ~/.orama/release-repo
orama maint release cut --channel nightly --archive ... --dry-run   # plan only: signs and writes nothing

# upload: archives to the GitHub release, then metadata to the host, timestamp last
orama maint release publish --dir ~/.orama/release-repo

# keep a quiet repository believed: every 7 days (nightly, dev) or 30 (main)
orama maint release refresh-timestamp --channel nightly --dir ~/.orama/release-repo
orama maint release publish --dir ~/.orama/release-repo

# before the root's year is up
orama maint release renew-root --dir ~/.orama/release-repo

cut says what each approval is for before it asks: the targets file (which archives, which channel), the snapshot that names it, and the timestamp that names the snapshot. It writes only to --dir, checks that a client would accept the result before it writes, and keeps the newest three versions of the channel (--retention; the RootWallet agent signs at most 200 targets). A version must be dotted numeric and newer than the channel's newest; a published version is immutable (--replace is for a dev/<branch> build that reuses one). A second cut before publish is refused; publish can be run again if it stopped part way. It refuses an archive that is wallet-signed, names signers or a root, or lacks the global layer on amd64 (--allow-cluster-only). publish needs gh logged in with upload rights on DeBrosDAO/orama and ssh to the release host (--github-repo, --metadata-dest); --dry-run prints the commands. The release host serves /opt/orama-releases with website/deploy/releases.orama.network.nginx.conf, installed once like the site's own.

The timestamp is the freeze protection: nodes refuse metadata whose timestamp has expired, so skipping a refresh stops releases for every node on the channel.

Before the build, the chain's performance gate must pass (the numbers and the baseline machine are on Performance): on the baseline machine run cd chain && make bench && make bench-compare BASE=bench/baseline.txt, which fails when a benchmark got more than 25% slower in time per operation or stopped running, and run the fleet e2e with E2E_CHAIN_BASELINE_FILE set to the previous release's chain-perf/baseline.json. A release that keeps its numbers then records them as the next baseline (make bench-baseline).

See Upgrades and Build and sign.

Versions and releases

VERSION at the repository root is the single version. make -C core bump VER=<x.y.z> updates it everywhere; a release tag v<version> must match it or the release workflow fails. GoReleaser builds the CLI and node binaries for GitHub releases, Homebrew (stable only) and apt packages. The signed install archive that nodes use is built from a checkout with orama maint build, signed by your own wallet. CI also attaches an unsigned, reproducible archive to each release; it becomes installable through the TUF path above, not by being attached.

Before rolling out

• +Unit tests and make test (including the coverage gate) pass.
• +Docs updated for any behaviour change, in the same change.
• +If the change touches inter-node coordination, secrets, namespaces or deployments, read the upgrade page's mixed-version windows: pause namespace and deployment create and delete during the roll, and do not add a node until every cluster you are upgrading is on the same version.
• +If the change adds a migration, it must be applied before the gateway that requires it starts; migrations take a cluster-wide lock, and the gateway refuses to start if its required schema is not applied (orama maint node schema status).
• +RQLite v10 is one way. Take a registry backup first.

Verification between nodes

orama status report --env <env> --node <ip>     # that node: services, RQLite state, WireGuard peers
orama status report --env <env>                 # the cluster: leader stable, quorum ok, mesh ok, indices in sync

Proceed only when both pass. orama maint rollout gates on the same signals itself and stops the moment a node fails to rejoin.

What not to do

• +Do not stop all nodes, replace binaries, then start all nodes.
• +Do not run orama node upgrade --restart on several nodes in parallel.
• +Do not clear RQLite data directories unless you are rebuilding the whole cluster.
• +Do not stop orama-node on several nodes at once; it takes the index services with it.
• +Do not fix a node by hand. Fix in source, test, then roll out.

Environments

An environment is just a name for a cluster in your CLI. Promotion is rolling the same archive to the next one (orama maint push --env <next> --archive <same path>), which the nodes verify against their own trust anchor. Pushing the same archive to several clusters works only if each trusts the signer.

Stagenet: the chain and the global services

The stagenet is five machines (mew, mewtwo, gengar, magicarp, froakie; ssh aliases from ~/.ssh/config). orama setup --create-network makes it: it installs the signed release and the cluster node on all five, makes each machine's chain keys, builds the genesis from all five, starts the chains, registers the operator and the nodes, and writes networks/stagenet/ for the maintainer to publish (Create a network). The global services run beside the cluster node, co-located in the orama-global network namespace (Run a global node).

All five are chain validators: the bootstrap committee is the five of them. The last column is the node's id on the chain when the addresses are given to orama setup in this order with the default --name founder (not seed: the chain keeps seed<N> for the labels the zone publishes itself). Without BPF_FRAMEWORK systemd accepts SocketBindDeny but does not enforce it, so the deployment sandbox's bind check refuses a bind only on mew and mewtwo.

The five are made with one command (every reset of the stagenet gets a new chain id, orama-stagenet-N):

orama setup --create-network stagenet --chain-id orama-stagenet-N --release-root ./release-root.json --yes \
  --ip 57.129.166.16 --ip 57.129.166.17 --ip 161.97.184.199 --ip 161.97.184.202 --ip 161.97.151.255 \
  --host-key 57.129.166.16=SHA256:... --host-key 57.129.166.17=SHA256:... --host-key 161.97.184.199=SHA256:... \
  --host-key 161.97.184.202=SHA256:... --host-key 161.97.151.255=SHA256:...

chain/scripts/stagenet/deploy.sh is what is left of the old deploy script: the maintenance of the stagenet that runs. It refuses to run for a CHAIN_ID that does not contain -stagenet- or -devnet-, and it never starts, stops or reconfigures a cluster service.

cd chain/scripts/stagenet

./deploy.sh reset      # remove any earlier global install (also the legacy direct-unit one) and its state; do it before the next creation
./deploy.sh status
./deploy.sh faucet     # once, after the creation: the public faucet on every node (key, funding, node.yaml, one restart at a time)
./deploy.sh invariants
./deploy.sh gen-shielded                      # optional: the shielded wallet scenario for this chain
SHIELDED_SCENARIO=../../build/stagenet-shielded-scenario.json ./deploy.sh smoke   # gen-shielded prints this path

reset removes the orama-global-* units and state, the namespace, veth and nftables rulesets, the ufw rules tagged orama-global, puts net.ipv4.ip_forward back to the value the install recorded in /var/lib/orama-global/netns-prior-ip-forward, and removes the two lines (role: both, global_netns) the install added to the cluster's /opt/orama/.orama/preferences.yaml. smoke prints PASS, FAIL or SKIP for each check and exits non-zero if any failed. A SKIP names an environmental cause the script detected in the chain's state and is never used to hide a failure; the archive and storage checks SKIP on a network whose nodes run no archiver, which orama setup does not install.

faucet (once, on a created network) switches the public faucet on node by node. On each node it runs orama maint faucet init, which makes the key its gateway signs POST /v1/chain/faucet drips with; funds that account with FAUCET_FUND_NORAMA through orama chain faucet --env $ORAMA_ENV, a drip signed on the node by its seat key (a genesis account is not an option: the chain refuses a genesis that does not start at zero supply); writes chain.faucet.enabled: true into the node's node.yaml (remote/enable-faucet.sh); and restarts the node with orama node restart, whose health gate decides when the next node may go (docs/whitepaper/technical-reference/vol2/39-chain-architecture.md, "Local and stagenet scripts"). The genesis orama setup --create-network builds sets the faucet's largest drip to 10,000 ORAMA, since one orama setup is funded by one drip and needs the validator's 1,000 ORAMA self-bond on top of its bonds and fees. A key, an account that already holds its funds and a node.yaml that already has the block are left as they are, so the command can be run again.

Test clusters

Sandbox clusters create five Hetzner servers, install the archive and give you a throwaway cluster. For release confidence the owner runs the fleet suite (Testing).