Troubleshooting Index

This page is the cross-component entry point for troubleshooting an Open ACE deployment. It maps symptoms to the guide that owns the problem. If you already know which component is at fault, go directly to that guide's own Troubleshooting section — the linked entries below all have one.

Symptom Index

Symptom	Likely component	Go to
WebUI iframe is blank; browser console reports mixed content on an HTTPS page	Reverse proxy / webui over HTTPS	NGINX.md — Problem A and "Blank iframe" in Troubleshooting
Remote agent stays offline; agent logs show repeated HTTP 413 on /api/remote/agent/message	nginx client_max_body_size (default 1 MB)	NGINX.md — Problem E
Remote terminal disconnects after exactly 5 minutes of idle time ("Connection closed. Reconnecting...")	nginx proxy_read_timeout	NGINX.md — Problem F
Remote session API returns "Not Found" when accessed through the proxy	API paths rewritten by global sub_filter	NGINX.md — Troubleshooting, "Remote session returns Not Found"
Remote machine cannot register: install script returns 404; agent cannot connect; agent stuck in a WebSocket 405 reconnect loop	Remote workspace server side / registration token	REMOTE_WORKSPACE.md — Troubleshooting
Agent won't connect; CLI tool not found; browser terminal not connecting; session sync not working	Remote agent on the machine	REMOTE_AGENT.md — Troubleshooting
alembic upgrade fails; /readyz returns 503 with schema_version incompatible / behind head	Database schema migration	SCHEMA_MIGRATION_GUIDE.md — Troubleshooting; for in-place upgrade failures and rollback see UPGRADING.md — Migration Failure Handling
Sandbox pod abnormal, workspace creation rejected with fail-closed reason codes	Sandbox backend (legacy / OpenSandbox)	SANDBOX_BACKENDS.md — section 10, Troubleshooting
Service or container fails to start: port already in use, database locked / connection refused, permission issues, code-server installation verification	Deployment / runtime environment	DEPLOYMENT.md — Troubleshooting
Multi-user WebUI fails to start or a user's workspace instance misbehaves (startup errors, runtime issues)	Multi-user workspace	MULTI_USER_WORKSPACE.md — Troubleshooting
Backup CronJob fails, restore job errors, backup integrity check fails	Database backup tooling	DATABASE_BACKUP.md — Recovery procedure and common failures
SSO login fails; browser URL carries ?sso_error=<code> (signature failure, clock skew, redirect_uri_not_allowed, ...)	SSO provider configuration	SSO_CONFIG.md — Testing and Troubleshooting
Feishu / DingTalk user or group names are not resolving in imported sessions	Feishu / DingTalk integration	FEISHU_CONFIG.md / DINGTALK_CONFIG.md — Troubleshooting
Pod stuck not ready; liveness/readiness probes failing	Health probes and dependencies	OPERATIONS.md — Health and Monitoring Endpoints
Cluster-level deploy, scaling, or ingress problems	Kubernetes manifests	KUBERNETES.md

Collecting Diagnostics

Before digging into a specific guide, gather this baseline:

1. Readiness and liveness (web service, port 19888):

curl -s http://localhost:19888/readyz | python3 -m json.tool
curl -s http://localhost:19888/livez

/readyz reports each check (database, schema_version, config_dir, workspace_dir, encryption_keys, init_status, security_mode, ssh_sync, frontend_build) and, on 503, an action hint — for example alembic upgrade head when the schema is behind head.

2. Security baseline:

curl -s http://localhost:19888/security-status | python3 -m json.tool

3. Scheduler worker (port 9090): curl http://localhost:9090/health returns is_leader; /metrics exposes Prometheus metrics.

4. Logs:

   • Docker Compose: docker compose ps, then docker compose logs -f open-ace and docker compose logs -f scheduler. The app also writes to /app/logs inside the container, bind-mounted to ./logs on the host.
   • Kubernetes: kubectl -n open-ace get pods, kubectl -n open-ace describe pod <pod> (probe failure events), kubectl -n open-ace logs <pod>.

5. SSH sync alert files: a failed secure SSH sync writes /var/log/openace/ssh-sync-failure.warning (human-readable, with remediation hints) and /var/log/openace/ssh-sync-failure.json (structured). While the .warning file exists, /readyz fails its ssh_sync check. Details are in /var/log/openace/ssh-sync.log. Delete the warning file only after fixing the cause.