Agent index (llms.txt) — fetch this first, then the page it names.

WireGuard Mesh Networking

All inter-node communication in the Orama Network happens over a WireGuard mesh overlay. WireGuard provides encrypted, high-performance tunnels between every node in the cluster, creating a private network that is invisible to the public internet.

How the Overlay Works

Each Orama node is assigned a private IP address on the 10.0.0.0/24 subnet. All service-to-service traffic (RQLite replication, Olric cache sync, IPFS peering, PubSub relay, etc.) uses these private IPs. The only traffic that traverses the public internet is:

• +SSH (port 22) -- for operator management access
• +HTTPS (port 443) -- for external client traffic to the gateway
• +WireGuard (UDP port 51820) -- the encrypted tunnel itself
• +DNS (port 53, TCP/UDP) -- only on nameserver nodes

Internal service ports (the index services on 10100 to 10114, tenant namespaces on 10000 to 10099, the SFU on 20000 to 30099, the chain's loopback ports, and so on) are never exposed to the public internet: they listen on the overlay address or loopback. The only exception is TURN, which a namespace with WebRTC opens on purpose (WebRTC).

Network Topology

The mesh is a full-mesh topology -- every node has a direct WireGuard tunnel to every other node:

  Node A (10.0.0.1)
    +-- tunnel --> Node B (10.0.0.2)
    +-- tunnel --> Node C (10.0.0.3)
    +-- tunnel --> Node D (10.0.0.4)

  Node B (10.0.0.2)
    +-- tunnel --> Node A (10.0.0.1)
    +-- tunnel --> Node C (10.0.0.3)
    +-- tunnel --> Node D (10.0.0.4)
  ...

This ensures any node can communicate directly with any other node with minimal latency. There is no routing through intermediary nodes.

How WireGuard is configured

Configuration is fully automated during node installation. You do not configure WireGuard by hand.

1. +Key generation. A WireGuard keypair is generated for the node. The private key never leaves the node.
2. +Address allocation. A join allocates the lowest free address in 10.0.0.2-254 with a plain insert into the registry's wireguard_peers table, retrying only on an address collision. Join and OramaOS enrolment both write through this one function (pkg/overlay), and the row stays unconfirmed until the node is seen in the node list, so a join that fails halfway leaves something the membership reconciler can remove. (The older approach of max+1 could roll past 10.0.0.254 into 10.0.1.x, outside the /24 that the firewall and the internal-auth check both accept, and two racing joins could delete each other's row.)
3. +Peer discovery. The joining node receives the peer list in the join response and brings up wg0 with a tunnel to every peer before any database, cache, storage or gateway service starts.
4. +Self-repair every 60 seconds. Every node checks its live tunnel configuration against the membership list in the registry. A peer whose endpoint or allowed IPs drifted is re-applied and the result is persisted to /etc/wireguard/wg0.conf (mode 0600). A peer whose public IP moved converges within a minute. The sync is additive from the node's own replica if the registry is unreachable, because Raft runs over the mesh and fixing the transport must not depend on the transport.
5. +Removal. orama node remove releases the node's mesh address by id and by address, and the membership reconciler drops a departed node's peer row.

The WireGuard unit (orama-namespace-wireguard@index) is independent of the supervisor orama-node: the mesh comes up at boot on its own, a restart of the node does not tear it down, and its stop is a no-op. Bring the interface down explicitly (wg-quick down wg0) only when that is the intent. The distribution's wg-quick@wg0 unit stays disabled and must not be started.

No peer-management API

There is no HTTP endpoint for adding, listing or removing mesh peers. A node joins only through the invite-gated join (or OramaOS enrolment, which also needs an invite), and peers converge through the 60-second self-repair above. The former /v1/internal/wg/peer, /peers and /peer/remove routes were removed because they let anyone holding the cluster secret add a machine to the mesh without an invite; an upgraded node answers them exactly as a path that never existed: 404 to a valid credential, 401 to none.

Key Management

Each node's WireGuard private key is generated during installation and stored on the node's filesystem. The corresponding public key is registered with the cluster for peer discovery. Private keys never leave the node.

Monitoring Mesh Health

Using orama status

The orama status mesh command shows WireGuard mesh connectivity across the cluster:

# View mesh connectivity for all nodes
orama status mesh --env devnet

# JSON output
orama status mesh --env devnet --json

# Filter to a specific node
orama status mesh --env devnet --node 203.0.113.10

The data comes from the cluster gateway's telemetry, not SSH: each node's report carries its WireGuard peers, and the view shows the peers against the N-1 a full mesh needs and each link's handshake age and traffic, so you can see which tunnels are healthy (recent handshakes) and which are stale. Add --ssh to collect from the nodes directly when no gateway answers.

Using orama maint inspect

The orama maint inspect command with the wg subsystem performs deeper SSH-based WireGuard health checks:

# Inspect WireGuard on all nodes
orama maint inspect --env devnet --subsystem wg

# With AI analysis of failures
orama maint inspect --env devnet --subsystem wg --ai

Using orama node doctor

On a single node, orama node doctor includes basic connectivity checks as part of its diagnostic sweep:

sudo orama node doctor

Direct WireGuard Commands

If you need to debug at a lower level, you can SSH into a node and use standard WireGuard tools:

# Show WireGuard interface status and peer handshakes
sudo wg show

# Check if the interface is up
ip link show wg0

# Ping another node on the mesh
ping 10.0.0.2

A healthy peer connection shows a latest handshake within the last 2 minutes. If a peer's handshake is older or missing, the tunnel may be down.

Troubleshooting

Firewall Blocking UDP 51820

Symptoms: No handshakes with any peers.

Resolution: Ensure UDP port 51820 is open inbound and outbound on your VPS provider's firewall and on UFW:

sudo ufw allow 51820/udp

Some VPS providers have an external firewall (security groups, network policies) in addition to the OS-level firewall. Check both.

Stale Handshakes After IP Change

Symptoms: Peer handshakes stop updating after a VPS IP change.

Resolution: If your node's public IP changes, the WireGuard endpoint registered with the cluster becomes stale. The 60-second peer sync re-applies it and a peer whose public IP moved converges within a minute. If it does not, check what the node believes before touching it (wg show wg0 dump, and sudo orama node logs node --since -10min | grep 'WireGuard peer sync completed': the line reports added, updated, removed and persisted; persisted=false means the mesh is right now but will regress at the next wg-quick up), then restart:

sudo orama node restart

NAT Issues

Symptoms: Intermittent connectivity, some peers unreachable.

Resolution: WireGuard uses UDP, which can be affected by NAT timeouts. The Orama WireGuard configuration includes persistent keepalive to maintain NAT mappings. If you are behind a particularly aggressive NAT, verify that your VPS has a direct public IP (not behind carrier-grade NAT).

All Peers Unreachable

Symptoms: No mesh connectivity at all, sudo wg show shows no handshakes.

Resolution:

1. +Verify the WireGuard interface is up: ip link show wg0
2. +Check that the WireGuard service is running: sudo orama node status
3. +Verify UDP 51820 is not blocked: sudo ufw status
4. +Check if the WireGuard configuration file exists and has peers listed
5. +Run sudo orama node doctor for a comprehensive diagnostic

Broken Pipe Errors During Rolling Upgrades

Symptoms: Other nodes briefly report a restarted node as "unreachable" with "broken pipe" errors.

Resolution: This is normal. Raft TCP connections re-establish within 1-2 minutes after a node restart. Wait and re-check before concluding there is a real problem.

Security

The WireGuard mesh provides several security guarantees:

• +Encryption -- all inter-node traffic is encrypted with ChaCha20-Poly1305
• +Authentication -- peers are authenticated by their public keys
• +No exposed services -- internal service ports are only reachable via the mesh
• +Forward secrecy -- WireGuard provides forward secrecy through its Noise protocol framework

The only ports that must be publicly accessible are: