Operations¶
Running JANUS day to day means watching the difference between what you declared, what the Controller compiled, what was delivered, and what a Gateway actually applied — and acting when those diverge.
The Controller records intent. The Gateway records reality. They are reconciled, and the difference is always observable. Nothing is inferred: a Gateway that received state is never reported as having applied it.
Units¶
Every JANUS component runs as a systemd unit:
| Unit | Runs | User | Enabled on install |
|---|---|---|---|
janus-controller.service |
/usr/bin/janus-controller run --config /etc/janus/controller.yaml |
janus-controller |
yes (not started) |
janus-webui.service |
/usr/bin/janus-webui -config /etc/janus/controller.yaml -api http://127.0.0.1:8081 -secure-cookies |
janus-controller |
yes (not started) |
janus-portal.service |
/usr/bin/janus-portal -config /etc/janus/controller.yaml -api http://127.0.0.1:8081 -secure-cookies |
janus-controller |
yes (not started) |
janus-gateway.service |
/usr/bin/janus-gateway run --config /etc/janus/gateway.yaml |
janus-gateway |
yes (not started) |
janus-site-edge.service |
/usr/bin/janus-site-edge run --config /etc/janus/site-edge.yaml |
janus-site-edge |
yes (not started) |
janus-site-edge-apply.service |
/usr/bin/janus-site-edge-apply serve --socket /run/janus/site-edge-apply.sock --service-user janus-site-edge --service-group janus-site-edge |
root (group janus-site-edge) |
yes and started |
janus-clientd.service |
/usr/bin/janus-clientd --manifest /etc/janus-client/manifest.json --key-file /etc/janus-client/private.key --interface janus0 --socket /run/janus-client/client.sock |
janus-client |
yes and started |
caddy.service (via a janus-caddy drop-in) |
/usr/bin/caddy run --environ --config /etc/janus/caddy/Caddyfile |
caddy package |
enabled by janus-caddy |
janus-controller.service, janus-gateway.service and janus-site-edge.service
each have a ConditionPathExists gate on their configuration file, so they do
not start until that file exists. janus-clientd.service deliberately has no
such gate, so it starts (and reports UNPROVISIONED) before it is provisioned.
Start, stop, restart and enable¶
# Controller host
sudo systemctl start janus-controller janus-webui janus-portal
sudo systemctl stop janus-controller janus-webui janus-portal
sudo systemctl restart janus-controller janus-webui janus-portal
sudo systemctl enable janus-controller janus-webui janus-portal
# reference front door (with janus-caddy installed)
sudo systemctl enable --now caddy
# Gateway host
sudo systemctl start janus-gateway
sudo systemctl restart janus-gateway
sudo systemctl enable janus-gateway
# Site Edge host
sudo systemctl enable --now janus-site-edge
The two privileged helpers — janus-site-edge-apply.service (root, on the edge
host) and the gateway-apply helper the gateway agent spawns — are started by
their own units or on demand, and should not need manual intervention.
Logs¶
All units log to the journal:
journalctl -u janus-controller -n 100 --no-pager
journalctl -u janus-webui -n 100 --no-pager
journalctl -u janus-portal -n 100 --no-pager
journalctl -u janus-gateway -n 100 --no-pager
journalctl -u janus-site-edge -n 100 --no-pager
journalctl -u janus-site-edge-apply -n 100 --no-pager
journalctl -u caddy -n 100 --no-pager
- The Controller logs its Git transport/trust/auth profile, migration status and reconciliation results, and never logs a secret.
- Both the Controller and the WebUI log the listener address they actually bound.
- The Gateway never prints its private key;
statusreports id, fingerprint and enrollment state only. - The Site Edge logs JSON to stdout.
Health checks¶
systemctl is-active janus-controller janus-webui janus-portal
curl -fsS http://127.0.0.1:8081/healthz # process alive
curl -fsS http://127.0.0.1:8081/readyz # ready + last-known-good revision
ss -tlnp | grep -E ':(8080|8081|8082|5432)\b' # what is actually bound
janus-controller version
janus-gateway version
janus-gateway status
A running process is not a healthy service
/healthz proves the process is alive. /readyz returns 503 unless
PostgreSQL is reachable and a last-known-good revision exists. Treat
readyz as the real readiness signal.
The state model¶
Configuration moves through a defined sequence, and each stage is observable on its own:
Desired (Git) -> Compiled (Controller) -> Pending -> Delivering
-> Validated (Gateway) -> Applied
Observed (Git) — the most recent commit the Controller has detected
| State | Meaning | Where it is observed |
|---|---|---|
| Desired | the configuration exists in the Git desired-state repository | git_revisions; the Operations screen |
| Compiled | parsed, validated and compiled into a per-Gateway desired state, stored as an immutable JSONB artifact | compiled_gateway_states (id, gateway_name, git_revision, compiled_at, desired_state, compiler_version) |
| Pending | a compiled state exists in PostgreSQL but has not yet been delivered | the Gateway's delivery status |
| Delivering | a delivery attempt is in progress | delivery status delivering |
| Validated | the Gateway received, validated and persisted the candidate | delivery status validated; last validated compiledStateID, last validated Git revision |
| Applied | the Gateway applied the configuration and the data plane reflects it | Gateway runtime status applied; appliedRevision |
| Observed | the most recent Git commit on the configured branch the Controller has detected | git_revisions; controller_state.observed_revision |
Controller-tracked Gateway fields include: desired compiled-state id, desired Git
revision, delivery status (pending/delivering/validated/failed), last
delivery attempt, last delivered compiled-state id, last validated compiled-state
id, last validated Git revision, last acknowledgement time and last delivery
error. Gateway runtime status values are pending, compiled, apply_pending,
applied, error. Controller singleton fields are observed_revision,
last_valid_revision, last_reconciliation_id, last_reconciliation_at and
last_reconciliation_ok.
delivered != applied
A Gateway that received state is not reported as having applied it until
application succeeded. Always read the two fields separately. Application of
a delivered candidate is requested by the Controller (delivery.autoApply)
and requires the Gateway to have agent.applyHelperPath set.
Last-known-good (LKG)¶
The last-known-good revision is the most recent Git commit that parsed,
passed semantic validation, compiled successfully for all enabled Gateways,
passed ValidateGatewayDesiredState for every artifact, and was transactionally
persisted to the runtime database. It is the effective desired state.
A revision becomes globally LKG only when every enabled Gateway validates and compiles; a single failure rejects the revision globally and the previous LKG is retained. Disabled Gateways do not block.
An invalid revision is marked invalid with its error recorded; the previous
LKG is retained, the compiler is never called, no partial state is applied, and
the Controller remains operational.
To inspect a rejected revision:
SELECT * FROM reconciliation_runs WHERE status = 'failed' ORDER BY started_at DESC LIMIT 1;
SELECT * FROM git_revisions WHERE commit_hash = '<revision>';
The error_detail in reconciliation_runs explains the failure.
Controller outage and last-known-good¶
The LKG survives:
- a Controller restart — it is persisted in PostgreSQL;
- a Git outage — it is retained during the outage;
- invalid subsequent revisions — the LKG stays put while the newer revision is rejected.
PostgreSQL unreachable: reconciliation fails (no fake success is ever
reported), /readyz returns 503, and the LKG is retained in memory and in the
compiled-state directory but cannot be persisted durably. Restore connectivity
and the Controller recovers automatically on the next poll cycle. If PostgreSQL
is completely lost, the Controller starts fresh and the LKG is re-established
from the next valid Git revision. A migration failure makes the Controller
refuse to start (manual intervention required); a schema incompatibility is
detected at startup and the Controller exits with an error.
Git unavailable: reconciliation fails at the fetch step, the LKG is retained,
and the log shows ERROR git fetch failed: <redacted error>. The next poll
retries. A trust failure (unknown or changed SSH host key, untrusted HTTPS
certificate) fails closed — never relaxed. A configuration failure in the
transport/auth/trust profile is caught at startup, before any Git process starts.
Under these conditions Git remains authoritative: a Git failure never creates an empty desired state and never tears down a healthy data plane.
Gateway recovery¶
- On Gateway restart with a validated candidate, the candidate file is
reloaded, its SHA-256 and semantic validation are re-verified,
lastValidatedreports correctly, andappliedremains null until it is applied again. - Once approved, the Gateway authenticates every request with its key and reconnects after a restart without re-approval; re-addressing or renaming does not require re-enrolment.
- Under
autoApply, an approved Gateway keeps running even when the Controller is unreachable: the heartbeat is best-effort, and a Controller outage never stops already-applied state. - If a Gateway is offline, its delivery status becomes
failedand the LKG is unchanged. When it comes online, the next retry — or a Controller restart — re-delivers the candidate. - On a Controller restart with a pending candidate, the candidate remains in PostgreSQL and the Controller retries delivery without requiring a new Git commit.
- A revoked Gateway fails closed immediately and cannot silently re-enrol.
Reboot behaviour¶
- The Gateway's runtime directories
/run/netnsand/run/access-concentratoraretmpfsand are re-created on every boot by the shippedtmpfiles.d/janus-gateway.confentry. A reboot loses the per-Deployment network namespace handles and the per-domain provider-uplink NAT intents together; a re-apply restores both. - The Controller's LKG survives a reboot (it is persisted in PostgreSQL), and a pending candidate is retried after the restart.
- On start, the Site Edge reconciler performs restart recovery: it reconstructs exactly the last-known-good already-applied state. An edge that has never applied anything reconstructs nothing.
- The client (
janus-clientd) is shut down by systemd; a restart re-runs a full reconcile.
Withdrawal semantics¶
Withdrawal is declarative and fail-closed. There is no imperative teardown.
- Removing an AccessPolicy, removing its subject or grant, revoking a Device, or changing a Prefix subject withdraws the corresponding authorization.
- The whole rule set is rebuilt atomically in one nftables batch, so there is no window in which the old authorization and the new classification coexist. Destination reclassification (for example INTERNAL becomes reachable under a new classification) denies new connections — there is no stale catch-all rule.
- Deleting a configuration object is referentially fail-closed: an object still referenced by another cannot be deleted, and a refused delete commits nothing. Delete in dependency order.
- Revoking a Gateway or a Site Edge identity is terminal: a revoked identity is never re-trusted.
- The WebUI and API never tear down WireGuard, nftables or a route imperatively; the reconciler converges to the new desired state.
Routine operator tasks¶
| Task | Where |
|---|---|
| Confirm readiness | curl -fsS http://127.0.0.1:8081/readyz |
| Confirm units are up | systemctl is-active janus-controller janus-webui janus-portal |
| Approve a new Gateway | janus enrollment list --state pending then janus enrollment approve <gateway-id> |
| Revoke a Gateway | janus enrollment revoke <gateway-id> --reason "..." |
| Mint an API token | janus-controller admin create-token --username admin --name <name> --scopes <scopes> --ttl <ttl> |
| Rotate the administrator password / break-glass | janus-controller admin rotate-break-glass, enable-break-glass, disable-break-glass |
| Validate configuration off to the side | janus-controller config validate / config compile / config migrate |
| Watch delivery and application | the Operations screen, or the delivery/apply status fields |
| Upgrade a component | install the new .deb, then restart the affected unit |
Related¶
- Configuration — authoring the desired state.
- Troubleshooting — when a layer fails.
- WebUI — the operator and self-service surfaces.
- Gateway agent — delivery and application.
- Data plane — what is enforced.
- Site Edge — the provider-facing participant.