Troubleshooting¶
When something "is not working", separate the layers before changing anything. A failure at one layer shows up differently from a failure at another, and changing configuration to mask a delivery problem usually makes the next diagnosis harder.
Diagnosis methodology¶
Work down these layers in order and stop at the first that is wrong:
| Layer | Question | First command or surface |
|---|---|---|
| 1. Configuration | Is the intent correct and present in Git? | janus-controller config validate, the Operations screen, the WebUI's Validate and review |
| 2. Compilation | Did the Controller compile it into a desired state for the Gateway that should carry it? | reconciliation_runs, compiled_gateway_states, journalctl -u janus-controller |
| 3. Delivery | Did the Gateway receive that desired state? | the Gateway's delivery status; journalctl -u janus-controller |
| 4. Application | Did the Gateway apply it, and does it report so? | janus-gateway status; the Gateway runtime status (applied, apply_pending, error) |
| 5. Data plane | Is the flow permitted by the applied policy? | the applied nftables state; the Gateway's apply logs |
Start every diagnosis from a baseline:
# Controller host
systemctl is-active janus-controller janus-webui janus-portal
curl -fsS http://127.0.0.1:8081/healthz
curl -fsS http://127.0.0.1:8081/readyz
journalctl -u janus-controller -n 100 --no-pager
# Gateway host
systemctl is-active janus-gateway
janus-gateway status
journalctl -u janus-gateway -n 100 --no-pager
Never widen trust to make a symptom go away
JANUS fails closed by design. Disabling host-key or certificate verification, or dropping a policy to "see if that was it", converts a diagnosable failure into an invisible one.
Gateway must be enrolled before anything is delivered¶
If a Gateway is not enrolled, approved and talking to the Controller, nothing downstream can work. Check enrolment first.
| Symptom | What to check | Likely cause |
|---|---|---|
janus-gateway enroll cannot reach the Controller |
controller.url and controller.caFile in /etc/janus/gateway.yaml; reachability of the HTTPS front door from the Gateway host |
wrong URL, or a CA the Gateway does not trust |
janus-gateway status shows state pending and never advances |
on the Controller: janus enrollment list --state pending |
the request was created but no administrator has approved the fingerprint — compare it out of band, then janus enrollment approve <gateway-id> |
| Reachability error in the Gateway log | journalctl -u janus-gateway — the underlying TLS/HTTP error is logged |
a certificate the Gateway does not trust is a caFile problem, never a reason to disable verification |
janus-gateway status shows state revoked |
— | the identity was revoked centrally; it fails closed and cannot silently re-enrol. Re-trusting needs a new explicit decision |
Administrative routes answer 503 |
— | no administrator exists yet; run janus-controller admin bootstrap once |
| The Gateway restarted and will not re-connect | janus-gateway status |
it reconnects without re-approval once trusted; a changed address or hostname does not require re-enrolment, so suspect the transport (url/caFile) or the Controller's reachability instead |
Not delivering state¶
| Symptom | What to check | Likely cause |
|---|---|---|
Delivery status failed |
journalctl -u janus-controller, and the Gateway's agent.listen (:9443) |
connection timeout or refused — the Gateway is offline or unreachable |
Delivery status stays pending |
whether the delivery: block exists in /etc/janus/controller.yaml |
with no delivery: block the Controller reconciles and compiles but does not deliver — a legitimate state with no Gateway nodes yet |
| Delivery fails with an authentication error | the gateway's agent.tls.*, the Controller's delivery.gateways[].caCert/tlsCert/tlsKey; certificate expiry, SAN and signing CA; private key permissions (0600) |
a TLS or certificate mismatch between Controller and Gateway |
Candidate rejected as INVALID_DESIRED_STATE |
re-compile locally with janus-controller config compile; verify the Git revision content; Controller and Gateway schema versions |
the desired state is invalid, or the schema versions are incompatible |
Candidate rejected as CHECKSUM_MISMATCH |
compiler version on Controller and Gateway | payload corruption or a compiler-version mismatch; re-delivery retries with a fresh candidate |
Delivered but never applied |
delivery.autoApply in controller.yaml; agent.applyHelperPath in gateway.yaml |
application is requested by the Controller and requires the Gateway's privileged helper; delivered != applied |
| Controller restarted with a pending candidate | — | normal: the candidate remains in PostgreSQL and the Controller retries delivery without a new Git commit |
Policy not taking effect¶
| Symptom | What to check | Likely cause |
|---|---|---|
| A permitted flow is dropped | the whole composition: valid source identity, the AccessPolicy, the destination classification, any Site Edge direction admission, an existing route, and the service/port constraint | a policy with no services grants nothing; every layer may narrow, no layer widens another |
| A flow to a specific destination is denied | whether the destination is classified INTERNAL when the grant is Internet-only (and vice versa) | an Internet grant never authorizes internal space; authorize an internal destination separately |
| A flow through a Site Edge is dropped | the edge's spec.roles |
reaching the Internet from a customer source needs ingress; an egress-only edge denies that NEW flow even with a valid policy |
| DNS does not work under an Internet grant | the services in the policy | there is no hidden port-53 exception; DNS is reachable only if the services allow it |
| A revoked Device keeps its access | whether the applied state reflects the withdrawal | withdrawal takes effect on the next applied state; the whole rule set is rebuilt atomically, so there is no stale rule to find |
| The policy is right but the Gateway enforces nothing | janus-gateway status — runtime status should be applied, not apply_pending/error; agent.applyHelperPath |
the privileged helper is not enabled, or application has not run |
The Gateway is applied at the wrong revision |
the desired Git revision vs the applied revision | delivery or application has not converged; reconcile first, do not edit the data plane by hand |
Database failures¶
| Symptom | What to check | Likely cause |
|---|---|---|
/readyz returns 503 |
reachability of PostgreSQL from the Controller host | PostgreSQL is unreachable — reconciliation fails, but the last-known-good is retained and the Controller recovers on the next poll cycle once connectivity is restored |
| Controller refuses to start after an upgrade | SELECT * FROM schema_migrations |
a migration failed; manual intervention is required |
| Controller exits at startup with an error | — | schema incompatibility: the Controller checks the schema version and exits if it does not match |
| Reconciliation reports no success | reconciliation_runs where status = 'failed' |
PostgreSQL unreachable or a rejected revision — no fake success is ever reported |
If PostgreSQL is completely lost, the Controller starts fresh and the LKG is re-established from the next valid Git revision.
Git failures¶
| Symptom | What to check | Likely cause |
|---|---|---|
ERROR git fetch failed: … in the Controller log |
Git server reachability; the git: block |
the Git transport or server is unavailable; the LKG is retained and the next poll retries |
Host key verification failed. |
the configured sshKnownHostsFile |
the host key is unknown or changed. Install it through the deployment process; do not disable host-key checking |
Permission denied (publickey) |
git.auth.sshKeyFile, the deploy-key registration |
the configured key is not authorized for the repository, or the wrong key file is configured |
Authentication failed for '…' |
git.auth.httpsSecretFile / httpsSecretEnv |
the token or app password is wrong, expired or revoked |
ERROR: Repository not found. |
the identity's access to the repository | the credential has no access to that repository |
SSL certificate problem: unable to get local issuer certificate |
the OS CA store, or the configured trust.kind: custom-ca bundle |
the issuing CA is not trusted; install it into the configured bundle — verification is never disabled |
A revision is marked invalid in git_revisions |
the recorded error, journalctl -u janus-controller |
validation failed; the LKG is unchanged and the Controller remains operational. Fix the configuration and push a new commit |
| Startup fails with a configuration message | — | the transport/auth/trust combination is validated before any Git process starts (for example git.auth.kind is required, or repository URL must not embed credentials) |
| A write is refused | the outcome | CONFLICT (remote advanced; re-read and retry), PUSH_FAILED (remote rejected; local commit discarded), PUSH_INDETERMINATE (transport failed before a verdict; re-read and verify). An ambiguous outcome is never reported as success |
Site Edge symptoms¶
A Site Edge is a separate product with its own identity and enrolment model. Its full troubleshooting table lives in Site Edge. The most common symptoms:
| Symptom | What to check | Likely cause |
|---|---|---|
| The edge routes nothing | whether Resources are bound with spec.siteEdge and enabled |
a declared lan.prefixes entry with no bound Resource is never accepted for provider transit |
| Customer traffic to the Internet is denied | the edge's spec.roles |
Internet egress from a customer source needs ingress |
Enrolment returns 401 |
the (deployment, site, edge-name) binding, and for a pre-registered edge spec.identity.publicKey |
the binding does not match an enabled SiteEdge, or the presented key does not equal the pre-registered one — the 401 is deliberately uniform and reveals nothing |
janus-site-edge status shows never-enrolled |
the central SiteEdge object |
the binding does not match an expected SiteEdge, or pre-registration does not match |
janus-site-edge status shows pending-approval |
the WebUI Site Edges page | possession was proven; an administrator has not yet approved the appliance. It authenticates nothing until then |
| The customer interface changed | JANUS owns only the edge's WireGuard interface and its own marked routes | it should not change the customer-side interface; collect ip route show and the apply logs |
The edge's own CLI (janus-site-edge identity, status, config, enroll,
leave) reports what the edge believes about itself; systemctl status and
journalctl -u janus-site-edge (and -u janus-site-edge-apply) report what it is
doing.
Related¶
- Operations — the state model and how each state is observed.
- Configuration — authoring and validating intent.
- Controller installation and Gateway installation — the reference workflows.
- Site Edge — the edge's own symptom table.
- Security model — why JANUS fails closed.