# Server operations

> Logs, health checks, the storage warning and the errors zidane-server and its clients print, with their causes and fixes.

## Logs

The server writes its log to stderr. At start, it prints where it listens, then any notes and warnings:

```text
✓ zidane server listening on ws://0.0.0.0:4280 and /data/.zidane/server.sock (data /data/.zidane, pid 7)
· Stored the bootstrap token from ZIDANE_SERVER_BOOTSTRAP_TOKEN as tok_3f2a… "bootstrap". Revoke it once device tokens exist: zidane server token revoke bootstrap.
! warning: behind a proxy without trusted proxies: X-Forwarded-* headers from any peer are believed …
```

| Source | Shows |
| - | - |
| stderr, always | The listening line, warnings, a broken token file, shutdown (`zidane server: SIGTERM, shutting down`) |
| stderr, with `--verbose` (or `ZIDANE_SERVER_VERBOSE=1`) | Every connection, authentication, refusal and disconnect, as `[server]` and `[ipc-ws]` lines |
| `server-audit.jsonl` | Connections, tokens, sync, installs: [the audit log](https://zidane.si/docs/server-security.md#audit-log) |
| `server-audit.failures.jsonl` | Refused upgrades, aggregated per address and minute |

Under systemd: `journalctl -u zidane-server -f`. In Docker: `docker logs -f zidane`.

## Health

| Command | Prints on success | Exit |
| - | - | - |
| `zidane-server health` | `ok server "<name>" v<version> protocol <n> pid <pid> up <s>s` | 0, else 1 |
| `zidane-server health --http` | `ok http://127.0.0.1:4280/healthz` | 0 on a 200, else 1 |
| `zidane-server wait-idle` | `idle`, or `idle after <s>s` | 0 once idle (or when no server answers), 1 when work is still going at `--timeout` (10 minutes by default) |

`wait-idle` asks the host over the unix socket until no run, compaction, `/` command or background task is in flight. Open terminals do not count. Run it before a restart (an upgrade, a renewed certificate), because a restart cuts off running tool calls. `daemon.info` reports the same as `activeWork`.

`health` asks the host over the unix socket, without a token. `--timeout <ms>` bounds it (default 5000).

`health --http` requests `GET /healthz` on the configured port. When the server listens on every interface, the request goes to loopback. It does not check the TLS certificate. From another machine, run:

```bash
curl -fsS https://zidane.example.com/healthz
```

## Channels

`zidane-server channels check` checks the channels config offline and exits 1 on errors. `channels status` lists the running server's channel integrations. `channels reload` makes the running server read its config again. It sends `SIGHUP` to the pid the server reports over its unix socket, so run it where the server runs (in a container, through `docker exec`). `reload` sends nothing when no channel runs. A server that started without channels exits on `SIGHUP` and needs a restart. If its config did not load, fix it and send `SIGHUP` yourself. See [Channels](https://zidane.si/docs/channels.md).

## Storage warning

In a Linux container, the server looks at its mounts at start. If its data dir is not on a persistent mount, it prints this warning:

```text
! warning: the data dir is not on a persistent volume; it will be lost when the container is recreated (/data/.zidane is on an anonymous volume, a new one for each container). Mount a named volume or a host directory on it …
```

The cause in parentheses is one of three: the container's own filesystem, a tmpfs, or an anonymous volume. Docker makes an anonymous volume for the image's `VOLUME` when nothing is mounted, and a new one each time it creates the container. Mount a named volume or a host directory on `/data`. On Coolify, use Persistent Storage › Volume Mount.

## Troubleshooting

### At start

| Message | Cause | Fix |
| - | - | - |
| `Refusing to listen on 0.0.0.0 without TLS: device tokens and transcripts would cross the network in the clear.` | A non-loopback `--host` without TLS or a proxy | `--tls-cert` and `--tls-key`, or `--behind-proxy`, or `--host 127.0.0.1` |
| `TLS on the server and behind-proxy are exclusive: …` | A certificate with `behind-proxy` on, often the image's default | `ZIDANE_SERVER_BEHIND_PROXY=0` and an empty `ZIDANE_SERVER_TRUSTED_PROXIES`, or drop the certificate |
| `TLS needs both a certificate and a key: …` | Only one of the two is set | Set both |
| `… only applies behind a proxy: also set --behind-proxy …` | Trusted proxies without `--behind-proxy` | Add `--behind-proxy`, or clear the list |
| ``Invalid ZIDANE_SERVER_BOOTSTRAP_TOKEN: expected a token from `zidane server token generate` …`` | Not a `zds_` token | Paste the output of `token generate` |
| `ZIDANE_SERVER_BOOTSTRAP_TOKEN and ZIDANE_SERVER_BOOTSTRAP_TOKEN_FILE are both set: set one, not both.` | Both variables | Keep one |
| `warning: … holds a revoked token …: ignored.` | The bootstrap value was revoked | Remove the variable, or set a fresh token |
| `Invalid <file> key "x": expected …`, `unknown key "x"` | A bad `server.json` | Fix the key. The message lists the known keys |
| `Cannot read the TLS certificate <file>: ENOENT.` | A wrong path, or a file the server's user cannot read | Fix the path or its permissions |
| `EADDRINUSE` | Another process holds the port | Stop it, or change `--port` |
| `warning: behind a proxy without trusted proxies …` | `--behind-proxy` with an empty list | Set `--trusted-proxies` to your proxy |
| `warning: the TLS certificate <file> expires on <date> …`, `… expired on …` | The certificate the server serves is within 14 days of its end, or past it | Renew it, then restart once idle |
| `warning: the TLS certificate or key changed on disk …` | A renewal: the server keeps serving the pair it read at start | Restart once idle (`zidane-server wait-idle`, then restart) |

### When a client connects

| Message | Cause | Fix |
| - | - | - |
| `token refused (401)` | A missing, unknown or revoked token | Create a token for the device |
| `forbidden (403): the server only accepts connections through its HTTPS proxy` | Behind a proxy, a request that skipped it, or a proxy that sends no `X-Forwarded-Proto: https` | Connect through the proxy's `wss://` URL, or fix the proxy's headers |
| `forbidden (403): the server only accepts connections from its trusted proxies` | The proxy's address is outside `--trusted-proxies` | Add it |
| `forbidden (403): this client's origin is not allowed by the server (allowedOrigins)` | A browser origin missing from the allowlist | Add it to `ZIDANE_SERVER_ALLOWED_ORIGINS` |
| `the server redirects (301); not following it with the token` | An `http` to `https` redirect, or a wrong path | Use the final `wss://` URL |
| `Refusing ws://host: the token would travel unencrypted.` | `ws://` to a remote host | Use `wss://`, or an SSH tunnel to `ws://127.0.0.1` |
| `Protocol version mismatch with <host>: the server speaks v13, this client v12. Upgrade the client.` (status `version mismatch`) | Different protocol versions | Upgrade the side it names. The client checks again every 5 minutes, or at once with Retry |
| `A different server answers at <host>'s address` | The server was reinstalled, or the name points elsewhere | Remove the host and add it again |
| `too many connections or connection attempts (429)` | Too many upgrades from one address, or too many connections for one token | Close stale windows. If the connections are yours, raise the caps |
| `TLS failed: …` | A private or self-signed certificate this computer does not trust | Trust it in the system's certificate store, or list it in `NODE_EXTRA_CA_CERTS` |
| `… can be read by other users (mode 644) …` | A `--token-file` other users can read | `chmod 600` the file |

### While connected

| Symptom | Cause | Fix |
| - | - | - |
| The connection closes with 1009 | A message over `ZIDANE_SERVER_MAX_MESSAGE_BYTES` (32 MiB) | Smaller attachments, or raise the cap |
| `Rate limit exceeded: slow down and retry.` (-32012) | Over 50 requests per second (burst 200) on one connection | Batch requests, or raise `ZIDANE_SERVER_RATE_LIMIT` |
| `The reply is over … bytes` (-32013) | A reply too large for one message | Ask for less, for example a smaller `limit` on `sessions.loadEvents` |
| Long sessions open slowly over a slow link | Transcript pages go out uncompressed | Set `ZIDANE_SERVER_COMPRESS_LARGE_MESSAGES=1` (`compressLargeMessages`): messages of 64 KiB or more go out compressed to clients that accept it, at some CPU cost per message |
| A window drops and reloads its session | Missed heartbeats, a slow consumer, or a server restart | Make sure that the proxy's idle timeout is longer than 20 seconds. The window resumes without help |
| Tool approvals never come | Tool approvals set to Ask or Model decision, and only a browser client connected | Answer from the TUI or the desktop app ([why](https://zidane.si/docs/ui-adapters.md#limits)) |
| A new session lands in an odd folder | The session got no folder, so it uses the first workspace root. A session started from a git project on your computer runs in a clone of it, `<first root>/<repository name>` | Pick a folder in the new-session flow |
| A send from a local project fails with a clone error | The server could not clone the project's `origin`: no git credentials there, no network, or another folder of that name in the first root | Set up git credentials on the server, or pick a folder in the new-session flow |