Skip to content

Operations

Running JANUS day to day means watching the difference between what you declared, what the Controller compiled, what was delivered, and what a Gateway actually applied — and acting when those diverge.

The Controller records intent. The Gateway records reality. They are reconciled, and the difference is always observable. Nothing is inferred: a Gateway that received state is never reported as having applied it.

Units

Every JANUS component runs as a systemd unit:

Unit Runs User Enabled on install
janus-controller.service /usr/bin/janus-controller run --config /etc/janus/controller.yaml janus-controller yes (not started)
janus-webui.service /usr/bin/janus-webui -config /etc/janus/controller.yaml -api http://127.0.0.1:8081 -secure-cookies janus-controller yes (not started)
janus-portal.service /usr/bin/janus-portal -config /etc/janus/controller.yaml -api http://127.0.0.1:8081 -secure-cookies janus-controller yes (not started)
janus-gateway.service /usr/bin/janus-gateway run --config /etc/janus/gateway.yaml janus-gateway yes (not started)
janus-site-edge.service /usr/bin/janus-site-edge run --config /etc/janus/site-edge.yaml janus-site-edge yes (not started)
janus-site-edge-apply.service /usr/bin/janus-site-edge-apply serve --socket /run/janus/site-edge-apply.sock --service-user janus-site-edge --service-group janus-site-edge root (group janus-site-edge) yes and started
janus-clientd.service /usr/bin/janus-clientd --manifest /etc/janus-client/manifest.json --key-file /etc/janus-client/private.key --interface janus0 --socket /run/janus-client/client.sock janus-client yes and started
caddy.service (via a janus-caddy drop-in) /usr/bin/caddy run --environ --config /etc/janus/caddy/Caddyfile caddy package enabled by janus-caddy

janus-controller.service, janus-gateway.service and janus-site-edge.service each have a ConditionPathExists gate on their configuration file, so they do not start until that file exists. janus-clientd.service deliberately has no such gate, so it starts (and reports UNPROVISIONED) before it is provisioned.

Start, stop, restart and enable

# Controller host
sudo systemctl start   janus-controller janus-webui janus-portal
sudo systemctl stop    janus-controller janus-webui janus-portal
sudo systemctl restart janus-controller janus-webui janus-portal
sudo systemctl enable  janus-controller janus-webui janus-portal

# reference front door (with janus-caddy installed)
sudo systemctl enable --now caddy

# Gateway host
sudo systemctl start   janus-gateway
sudo systemctl restart janus-gateway
sudo systemctl enable  janus-gateway

# Site Edge host
sudo systemctl enable --now janus-site-edge

The two privileged helpers — janus-site-edge-apply.service (root, on the edge host) and the gateway-apply helper the gateway agent spawns — are started by their own units or on demand, and should not need manual intervention.

Logs

All units log to the journal:

journalctl -u janus-controller -n 100 --no-pager
journalctl -u janus-webui -n 100 --no-pager
journalctl -u janus-portal -n 100 --no-pager
journalctl -u janus-gateway -n 100 --no-pager
journalctl -u janus-site-edge -n 100 --no-pager
journalctl -u janus-site-edge-apply -n 100 --no-pager
journalctl -u caddy -n 100 --no-pager
  • The Controller logs its Git transport/trust/auth profile, migration status and reconciliation results, and never logs a secret.
  • Both the Controller and the WebUI log the listener address they actually bound.
  • The Gateway never prints its private key; status reports id, fingerprint and enrollment state only.
  • The Site Edge logs JSON to stdout.

Health checks

systemctl is-active janus-controller janus-webui janus-portal
curl -fsS http://127.0.0.1:8081/healthz        # process alive
curl -fsS http://127.0.0.1:8081/readyz         # ready + last-known-good revision
ss -tlnp | grep -E ':(8080|8081|8082|5432)\b'  # what is actually bound
janus-controller version
janus-gateway version
janus-gateway status

A running process is not a healthy service

/healthz proves the process is alive. /readyz returns 503 unless PostgreSQL is reachable and a last-known-good revision exists. Treat readyz as the real readiness signal.

The state model

Configuration moves through a defined sequence, and each stage is observable on its own:

Desired (Git)  ->  Compiled (Controller)  ->  Pending  ->  Delivering
               ->  Validated (Gateway)    ->  Applied

Observed (Git) — the most recent commit the Controller has detected
State Meaning Where it is observed
Desired the configuration exists in the Git desired-state repository git_revisions; the Operations screen
Compiled parsed, validated and compiled into a per-Gateway desired state, stored as an immutable JSONB artifact compiled_gateway_states (id, gateway_name, git_revision, compiled_at, desired_state, compiler_version)
Pending a compiled state exists in PostgreSQL but has not yet been delivered the Gateway's delivery status
Delivering a delivery attempt is in progress delivery status delivering
Validated the Gateway received, validated and persisted the candidate delivery status validated; last validated compiledStateID, last validated Git revision
Applied the Gateway applied the configuration and the data plane reflects it Gateway runtime status applied; appliedRevision
Observed the most recent Git commit on the configured branch the Controller has detected git_revisions; controller_state.observed_revision

Controller-tracked Gateway fields include: desired compiled-state id, desired Git revision, delivery status (pending/delivering/validated/failed), last delivery attempt, last delivered compiled-state id, last validated compiled-state id, last validated Git revision, last acknowledgement time and last delivery error. Gateway runtime status values are pending, compiled, apply_pending, applied, error. Controller singleton fields are observed_revision, last_valid_revision, last_reconciliation_id, last_reconciliation_at and last_reconciliation_ok.

delivered != applied

A Gateway that received state is not reported as having applied it until application succeeded. Always read the two fields separately. Application of a delivered candidate is requested by the Controller (delivery.autoApply) and requires the Gateway to have agent.applyHelperPath set.

Last-known-good (LKG)

The last-known-good revision is the most recent Git commit that parsed, passed semantic validation, compiled successfully for all enabled Gateways, passed ValidateGatewayDesiredState for every artifact, and was transactionally persisted to the runtime database. It is the effective desired state.

A revision becomes globally LKG only when every enabled Gateway validates and compiles; a single failure rejects the revision globally and the previous LKG is retained. Disabled Gateways do not block.

An invalid revision is marked invalid with its error recorded; the previous LKG is retained, the compiler is never called, no partial state is applied, and the Controller remains operational.

To inspect a rejected revision:

SELECT * FROM reconciliation_runs WHERE status = 'failed' ORDER BY started_at DESC LIMIT 1;
SELECT * FROM git_revisions WHERE commit_hash = '<revision>';

The error_detail in reconciliation_runs explains the failure.

Controller outage and last-known-good

The LKG survives:

  • a Controller restart — it is persisted in PostgreSQL;
  • a Git outage — it is retained during the outage;
  • invalid subsequent revisions — the LKG stays put while the newer revision is rejected.

PostgreSQL unreachable: reconciliation fails (no fake success is ever reported), /readyz returns 503, and the LKG is retained in memory and in the compiled-state directory but cannot be persisted durably. Restore connectivity and the Controller recovers automatically on the next poll cycle. If PostgreSQL is completely lost, the Controller starts fresh and the LKG is re-established from the next valid Git revision. A migration failure makes the Controller refuse to start (manual intervention required); a schema incompatibility is detected at startup and the Controller exits with an error.

Git unavailable: reconciliation fails at the fetch step, the LKG is retained, and the log shows ERROR git fetch failed: <redacted error>. The next poll retries. A trust failure (unknown or changed SSH host key, untrusted HTTPS certificate) fails closed — never relaxed. A configuration failure in the transport/auth/trust profile is caught at startup, before any Git process starts.

Under these conditions Git remains authoritative: a Git failure never creates an empty desired state and never tears down a healthy data plane.

Gateway recovery

  • On Gateway restart with a validated candidate, the candidate file is reloaded, its SHA-256 and semantic validation are re-verified, lastValidated reports correctly, and applied remains null until it is applied again.
  • Once approved, the Gateway authenticates every request with its key and reconnects after a restart without re-approval; re-addressing or renaming does not require re-enrolment.
  • Under autoApply, an approved Gateway keeps running even when the Controller is unreachable: the heartbeat is best-effort, and a Controller outage never stops already-applied state.
  • If a Gateway is offline, its delivery status becomes failed and the LKG is unchanged. When it comes online, the next retry — or a Controller restart — re-delivers the candidate.
  • On a Controller restart with a pending candidate, the candidate remains in PostgreSQL and the Controller retries delivery without requiring a new Git commit.
  • A revoked Gateway fails closed immediately and cannot silently re-enrol.

Reboot behaviour

  • The Gateway's runtime directories /run/netns and /run/access-concentrator are tmpfs and are re-created on every boot by the shipped tmpfiles.d/janus-gateway.conf entry. A reboot loses the per-Deployment network namespace handles and the per-domain provider-uplink NAT intents together; a re-apply restores both.
  • The Controller's LKG survives a reboot (it is persisted in PostgreSQL), and a pending candidate is retried after the restart.
  • On start, the Site Edge reconciler performs restart recovery: it reconstructs exactly the last-known-good already-applied state. An edge that has never applied anything reconstructs nothing.
  • The client (janus-clientd) is shut down by systemd; a restart re-runs a full reconcile.

Withdrawal semantics

Withdrawal is declarative and fail-closed. There is no imperative teardown.

  • Removing an AccessPolicy, removing its subject or grant, revoking a Device, or changing a Prefix subject withdraws the corresponding authorization.
  • The whole rule set is rebuilt atomically in one nftables batch, so there is no window in which the old authorization and the new classification coexist. Destination reclassification (for example INTERNAL becomes reachable under a new classification) denies new connections — there is no stale catch-all rule.
  • Deleting a configuration object is referentially fail-closed: an object still referenced by another cannot be deleted, and a refused delete commits nothing. Delete in dependency order.
  • Revoking a Gateway or a Site Edge identity is terminal: a revoked identity is never re-trusted.
  • The WebUI and API never tear down WireGuard, nftables or a route imperatively; the reconciler converges to the new desired state.

Routine operator tasks

Task Where
Confirm readiness curl -fsS http://127.0.0.1:8081/readyz
Confirm units are up systemctl is-active janus-controller janus-webui janus-portal
Approve a new Gateway janus enrollment list --state pending then janus enrollment approve <gateway-id>
Revoke a Gateway janus enrollment revoke <gateway-id> --reason "..."
Mint an API token janus-controller admin create-token --username admin --name <name> --scopes <scopes> --ttl <ttl>
Rotate the administrator password / break-glass janus-controller admin rotate-break-glass, enable-break-glass, disable-break-glass
Validate configuration off to the side janus-controller config validate / config compile / config migrate
Watch delivery and application the Operations screen, or the delivery/apply status fields
Upgrade a component install the new .deb, then restart the affected unit