Skip to content

Troubleshooting

When something "is not working", separate the layers before changing anything. A failure at one layer shows up differently from a failure at another, and changing configuration to mask a delivery problem usually makes the next diagnosis harder.

Diagnosis methodology

Work down these layers in order and stop at the first that is wrong:

Layer Question First command or surface
1. Configuration Is the intent correct and present in Git? janus-controller config validate, the Operations screen, the WebUI's Validate and review
2. Compilation Did the Controller compile it into a desired state for the Gateway that should carry it? reconciliation_runs, compiled_gateway_states, journalctl -u janus-controller
3. Delivery Did the Gateway receive that desired state? the Gateway's delivery status; journalctl -u janus-controller
4. Application Did the Gateway apply it, and does it report so? janus-gateway status; the Gateway runtime status (applied, apply_pending, error)
5. Data plane Is the flow permitted by the applied policy? the applied nftables state; the Gateway's apply logs

Start every diagnosis from a baseline:

# Controller host
systemctl is-active janus-controller janus-webui janus-portal
curl -fsS http://127.0.0.1:8081/healthz
curl -fsS http://127.0.0.1:8081/readyz
journalctl -u janus-controller -n 100 --no-pager

# Gateway host
systemctl is-active janus-gateway
janus-gateway status
journalctl -u janus-gateway -n 100 --no-pager

Never widen trust to make a symptom go away

JANUS fails closed by design. Disabling host-key or certificate verification, or dropping a policy to "see if that was it", converts a diagnosable failure into an invisible one.

Gateway must be enrolled before anything is delivered

If a Gateway is not enrolled, approved and talking to the Controller, nothing downstream can work. Check enrolment first.

Symptom What to check Likely cause
janus-gateway enroll cannot reach the Controller controller.url and controller.caFile in /etc/janus/gateway.yaml; reachability of the HTTPS front door from the Gateway host wrong URL, or a CA the Gateway does not trust
janus-gateway status shows state pending and never advances on the Controller: janus enrollment list --state pending the request was created but no administrator has approved the fingerprint — compare it out of band, then janus enrollment approve <gateway-id>
Reachability error in the Gateway log journalctl -u janus-gateway — the underlying TLS/HTTP error is logged a certificate the Gateway does not trust is a caFile problem, never a reason to disable verification
janus-gateway status shows state revoked — the identity was revoked centrally; it fails closed and cannot silently re-enrol. Re-trusting needs a new explicit decision
Administrative routes answer 503 — no administrator exists yet; run janus-controller admin bootstrap once
The Gateway restarted and will not re-connect janus-gateway status it reconnects without re-approval once trusted; a changed address or hostname does not require re-enrolment, so suspect the transport (url/caFile) or the Controller's reachability instead

Not delivering state

Symptom What to check Likely cause
Delivery status failed journalctl -u janus-controller, and the Gateway's agent.listen (:9443) connection timeout or refused — the Gateway is offline or unreachable
Delivery status stays pending whether the delivery: block exists in /etc/janus/controller.yaml with no delivery: block the Controller reconciles and compiles but does not deliver — a legitimate state with no Gateway nodes yet
Delivery fails with an authentication error the gateway's agent.tls.*, the Controller's delivery.gateways[].caCert/tlsCert/tlsKey; certificate expiry, SAN and signing CA; private key permissions (0600) a TLS or certificate mismatch between Controller and Gateway
Candidate rejected as INVALID_DESIRED_STATE re-compile locally with janus-controller config compile; verify the Git revision content; Controller and Gateway schema versions the desired state is invalid, or the schema versions are incompatible
Candidate rejected as CHECKSUM_MISMATCH compiler version on Controller and Gateway payload corruption or a compiler-version mismatch; re-delivery retries with a fresh candidate
Delivered but never applied delivery.autoApply in controller.yaml; agent.applyHelperPath in gateway.yaml application is requested by the Controller and requires the Gateway's privileged helper; delivered != applied
Controller restarted with a pending candidate — normal: the candidate remains in PostgreSQL and the Controller retries delivery without a new Git commit

Policy not taking effect

Symptom What to check Likely cause
A permitted flow is dropped the whole composition: valid source identity, the AccessPolicy, the destination classification, any Site Edge direction admission, an existing route, and the service/port constraint a policy with no services grants nothing; every layer may narrow, no layer widens another
A flow to a specific destination is denied whether the destination is classified INTERNAL when the grant is Internet-only (and vice versa) an Internet grant never authorizes internal space; authorize an internal destination separately
A flow through a Site Edge is dropped the edge's spec.roles reaching the Internet from a customer source needs ingress; an egress-only edge denies that NEW flow even with a valid policy
DNS does not work under an Internet grant the services in the policy there is no hidden port-53 exception; DNS is reachable only if the services allow it
A revoked Device keeps its access whether the applied state reflects the withdrawal withdrawal takes effect on the next applied state; the whole rule set is rebuilt atomically, so there is no stale rule to find
The policy is right but the Gateway enforces nothing janus-gateway status — runtime status should be applied, not apply_pending/error; agent.applyHelperPath the privileged helper is not enabled, or application has not run
The Gateway is applied at the wrong revision the desired Git revision vs the applied revision delivery or application has not converged; reconcile first, do not edit the data plane by hand

Database failures

Symptom What to check Likely cause
/readyz returns 503 reachability of PostgreSQL from the Controller host PostgreSQL is unreachable — reconciliation fails, but the last-known-good is retained and the Controller recovers on the next poll cycle once connectivity is restored
Controller refuses to start after an upgrade SELECT * FROM schema_migrations a migration failed; manual intervention is required
Controller exits at startup with an error — schema incompatibility: the Controller checks the schema version and exits if it does not match
Reconciliation reports no success reconciliation_runs where status = 'failed' PostgreSQL unreachable or a rejected revision — no fake success is ever reported

If PostgreSQL is completely lost, the Controller starts fresh and the LKG is re-established from the next valid Git revision.

Git failures

Symptom What to check Likely cause
ERROR git fetch failed: … in the Controller log Git server reachability; the git: block the Git transport or server is unavailable; the LKG is retained and the next poll retries
Host key verification failed. the configured sshKnownHostsFile the host key is unknown or changed. Install it through the deployment process; do not disable host-key checking
Permission denied (publickey) git.auth.sshKeyFile, the deploy-key registration the configured key is not authorized for the repository, or the wrong key file is configured
Authentication failed for '…' git.auth.httpsSecretFile / httpsSecretEnv the token or app password is wrong, expired or revoked
ERROR: Repository not found. the identity's access to the repository the credential has no access to that repository
SSL certificate problem: unable to get local issuer certificate the OS CA store, or the configured trust.kind: custom-ca bundle the issuing CA is not trusted; install it into the configured bundle — verification is never disabled
A revision is marked invalid in git_revisions the recorded error, journalctl -u janus-controller validation failed; the LKG is unchanged and the Controller remains operational. Fix the configuration and push a new commit
Startup fails with a configuration message — the transport/auth/trust combination is validated before any Git process starts (for example git.auth.kind is required, or repository URL must not embed credentials)
A write is refused the outcome CONFLICT (remote advanced; re-read and retry), PUSH_FAILED (remote rejected; local commit discarded), PUSH_INDETERMINATE (transport failed before a verdict; re-read and verify). An ambiguous outcome is never reported as success

Site Edge symptoms

A Site Edge is a separate product with its own identity and enrolment model. Its full troubleshooting table lives in Site Edge. The most common symptoms:

Symptom What to check Likely cause
The edge routes nothing whether Resources are bound with spec.siteEdge and enabled a declared lan.prefixes entry with no bound Resource is never accepted for provider transit
Customer traffic to the Internet is denied the edge's spec.roles Internet egress from a customer source needs ingress
Enrolment returns 401 the (deployment, site, edge-name) binding, and for a pre-registered edge spec.identity.publicKey the binding does not match an enabled SiteEdge, or the presented key does not equal the pre-registered one — the 401 is deliberately uniform and reveals nothing
janus-site-edge status shows never-enrolled the central SiteEdge object the binding does not match an expected SiteEdge, or pre-registration does not match
janus-site-edge status shows pending-approval the WebUI Site Edges page possession was proven; an administrator has not yet approved the appliance. It authenticates nothing until then
The customer interface changed JANUS owns only the edge's WireGuard interface and its own marked routes it should not change the customer-side interface; collect ip route show and the apply logs

The edge's own CLI (janus-site-edge identity, status, config, enroll, leave) reports what the edge believes about itself; systemctl status and journalctl -u janus-site-edge (and -u janus-site-edge-apply) report what it is doing.