# DockTail Cloud DockTail Cloud is optional, opt-in monitoring for DockTail-managed services across one or more hosts. It is a hosted dashboard that watches what DockTail exposes and tells you *why* something is unreachable. [Explore DockTail Cloud](https://docktail.org/cloud/) or [open the dashboard](https://cloud.docktail.org/login). DockTail Cloud is a paid service; the agent code that reports to it is the same open-source DockTail. - **Account.** You sign up at [cloud.docktail.org](https://cloud.docktail.org/login) yourself. An AI agent cannot create the account, the workspace or the subscription for you; once they exist, it can help with everything else. - **Plans.** Homelab, Team and Business, priced by the number of monitored hosts. Current prices and host limits are on [plans and pricing](https://docktail.org/cloud/#pricing). - **Trial.** Homelab and Team start with a 30-day free trial. It needs a card and turns into the paid plan when it ends unless you cancel. Business starts at its regular price. - **No plan, no hosts.** A workspace connects no host until it has a plan or a running trial; before that the dashboard shows demo data. ## What You Get - **One view of every host and service.** The full catalog of DockTail-managed services, plus a read-only inventory of the host's other containers, with health history. - **The useful distinction.** A plain uptime check stops at "down." Cloud already has the Docker and Tailscale context, so it separates a *container* problem (exit code, Docker OOM event, restart loop) from an *exposure* problem (the container answers locally, but the Tailscale control plane says the service is awaiting approval, not advertised by this host, or gone) from a *host* problem (the box stopped sending heartbeats). - **Incidents with the evidence attached.** Docker failure events and a bounded log tail captured at the moment of failure, so you start from the reason instead of from a graph. - **Alerts and recoveries.** Notification when a service breaks and when it comes back — including hosts that drop off entirely, since detection runs in Cloud rather than on the box. Reporting rides along with the normal agent — there is no separate binary. The same DockTail container gains cloud reporting when a workspace key is present, and stays completely inert without one — the only trace is a single info line at startup noting that DockTail Cloud is not connected. ## How To Enable Before you start: - **Tailscale API credentials.** Configure `TAILSCALE_OAUTH_CLIENT_ID`/`TAILSCALE_OAUTH_CLIENT_SECRET` (or `TAILSCALE_API_KEY`) first, as in [Tailscale Admin Setup](https://docktail.org/docs/tailscale-admin/#tailscale-admin-setup). Cloud's [tailnet vantage](https://docktail.org/docs/cloud/#tailnet-health) reads the control plane through them, and one credentialed host per tailnet is enough. With none, Cloud reports "no Tailscale credentials" instead of approval and advertisement state; everything else still works. - **A working DockTail.** The host is tagged, and its services already show up on the tailnet without Cloud. Then: 1. Sign in at [cloud.docktail.org](https://cloud.docktail.org/login), creating your account if you do not have one. 2. Create a workspace. 3. Choose a plan, or start the 30-day trial on Homelab or Team. 4. Copy the workspace key (`dtc_...`) the onboarding shows under **Connect your first host**. Later keys are created under [Settings → Agent keys](https://cloud.docktail.org/settings/agent-keys). 5. Set `DOCKTAIL_CLOUD_KEY` on the DockTail agent and recreate the container (`docker compose up -d`). Do this within the key's enrollment window, one hour by default (see [Identity](https://docktail.org/docs/cloud/#identity)). 6. The host appears on [Hosts](https://cloud.docktail.org/hosts), and [`docktail health`](https://docktail.org/docs/reference/#health-check) reports `cloud connected`. That is the only configuration — the cloud endpoint is built into the agent. A complete Compose file with it is [`docker-compose.cloud.yaml`](https://github.com/marvinvr/docktail/blob/main/docker-compose.cloud.yaml). ```yaml services: docktail: image: ghcr.io/marvinvr/docktail:latest restart: unless-stopped volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /var/run/tailscale:/var/run/tailscale environment: - TAILSCALE_OAUTH_CLIENT_ID=${TAILSCALE_OAUTH_CLIENT_ID} - TAILSCALE_OAUTH_CLIENT_SECRET=${TAILSCALE_OAUTH_CLIENT_SECRET} # Optional. Enables DockTail Cloud reporting. - DOCKTAIL_CLOUD_KEY=${DOCKTAIL_CLOUD_KEY} ``` With no key set, no connection is opened and DockTail runs exactly as before. `docktail health` reports the connection state (`connecting`, `connected`, `disconnected`, `rejected` with the reason, or `failed` when the module could not start, for example with a malformed key) without ever making the container unhealthy; see [Health Check](https://docktail.org/docs/reference/#health-check). When the cloud accepts the connection the agent also compares its clock with the cloud's and warns if they are more than 60 seconds apart, since event and check times would then be off; enable time sync (NTP) on the host. If Cloud marks a host as unmonitored (for example, the host sits past the workspace's host cap), the agent keeps the connection open with heartbeats and occasional catalog snapshots and pauses checks, Docker events, metrics, tailnet state, tailnet-health answers, and log excerpts. Cloud pushes an updated configuration over the same connection when that changes; the agent does not need a restart. ## Cloud Environment Variables | Variable | Default | Description | | --- | --- | --- | | `DOCKTAIL_CLOUD_KEY` | - | Workspace key (`dtc_...`) from the cloud dashboard. Enables reporting. Inert when unset. | | `DOCKTAIL_LOG_LEVEL` | `info` | Read, but currently has no effect: the cloud module logs at the level `LOG_LEVEL` sets. | | `DOCKTAIL_CHECK_INTERVAL` | `30s` | How often local-vantage checks run (5s–5m). | | `DOCKTAIL_HOST_ROOT` | `/host` | Where the host's root filesystem is bind-mounted, for whole-host disk usage (see [Disk Usage](https://docktail.org/docs/cloud/#disk-usage)). Only used when that path exists. | For local development, `DOCKTAIL_CLOUD_URL` overrides the built-in endpoint. Plaintext `ws://` is accepted only for loopback endpoints unless `DOCKTAIL_CLOUD_ALLOW_INSECURE=true` is also set. Do not use that escape hatch in production; non-loopback production endpoints must use `wss://`. ## What It Sends The hosted control plane (the dashboard and ingest service behind `wss://ingest.docktail.org`) is a separate, proprietary product. The reporting module in this repository sends operational metadata and bounded incident log tails to it over an outbound-only WebSocket Secure (WSS) connection. When enabled, the agent reports the following operational data: - Periodic snapshots of DockTail-managed services, including stopped containers, plus refreshes after successful reconciles. - A read-only inventory of the host's **other** containers — the ones *not* published with `docktail.*` labels, plus any labelled `docktail.cloud.ignore=true`, including stopped ones — with name, image, state/health, ports, and live CPU/memory. These containers are not actively probed; they are listed on the dashboard so you can see the host's whole Docker footprint, and can be explicitly watched for Docker-event-driven incidents and alerts. - Docker failure events, including container exit codes, out-of-memory (OOM) kills, health-status changes, and restart loops. Docker reports an OOM event whenever the kernel kills *any* process in a container, so the agent also says whether the container kept running. A killed child process (a worker, a transcoder) in a container that stays up is recorded as an event, not reported as the service going down; only an OOM kill that stops the container is. - Local-vantage check results. Checks default to TCP; cloud-managed config or the container's own [`docktail.cloud.check.*` labels](https://docktail.org/docs/labels/#docktail-cloud-labels) may select HTTP, a relative path, and expected status, but the destination always comes from the agent's local service discovery. HTTP checks are plain HTTP to the container and do not follow redirects; with an expected status set, only that exact status passes, otherwise any answer below 500 counts as up. Each probe allows up to 15 seconds, so a slow service that still answers is reported with its latency (Cloud can flag it as degraded) rather than as down. - Each service's valid `docktail.cloud.*` label values, so the dashboard can show which settings are set by label. - Whole-host vitals, sampled every 30 seconds: CPU, memory and swap usage, load average, temperature where the machine has sensors, and per-filesystem disk usage (mount point, total, used, and available bytes, at most 16 filesystems). These describe the machine, not the containers; nothing is stored as history, each report replaces the last. Disk is read on Linux only, from `/proc/mounts` and `statfs`, and covers real filesystems only — network mounts such as NFS and CIFS are deliberately skipped, so an unresponsive NAS can never stall reporting. Docker inside a Proxmox LXC needs no extra setup: the container's `/proc` shows the physical node, so the agent reports the CT's memory instead — its limit as Docker reports it, and as used the combined working set of all running containers (the same figure `docker stats` shows, without the CT's few non-container processes). Load average and swap are omitted there, because only the node's figures are visible. - Tailscale control-plane service state, when Cloud asks for it (see [Tailnet Health](https://docktail.org/docs/cloud/#tailnet-health)). - For a Funnel-exposed service, this node's MagicDNS name — the public address its Funnel answers on (see [Public Health](https://docktail.org/docs/cloud/#public-health)). - Bounded incident log excerpts. Cloud enables this by default and you can turn it off for the whole workspace or for an individual service, in the dashboard or with the `docktail.cloud.logs=off` label; the agent captures nothing while the mode is off. Before sending, the agent best-effort redacts common Authorization/Bearer credentials, passwords, tokens, API keys, credential URLs, JWTs, and private-key blocks, then applies the 40-line/8-KiB caps. Redaction cannot recognize every application-specific secret, so turn capture off if your logs carry secrets those patterns will not match. If the connection to Cloud drops, the agent keeps watching Docker and buffers the failure events it sees — plus the log tail for each, captured at the moment of the failure rather than on reconnect — then replays them with their original timestamps once the link is back. A container that crashes and recovers during a network blip or a Cloud deployment still produces an incident and an alert. The buffer is held in memory only, is capped (1024 frames / 1 MiB, newest kept), and frames older than an hour are discarded rather than replayed: by then Cloud has already alerted that the host stopped reporting, so re-raising hours-old container failures would only add noise. Periodic data — snapshots, check results, host metrics — is not buffered, since the next report replaces it anyway. ## Disk Usage With the standard setup above — only the Docker socket mounted — the agent sees one filesystem: its own container root. That is less limiting than it sounds, because the container root passes through to the filesystem that actually holds `/var/lib/docker`, which on a normal install is the host's root filesystem and the usual reason a self-hosted box runs out of space. Cloud reports it as `/`. To see every filesystem the host has — a separate `/mnt/data`, a media disk, a backup volume — bind-mount the host's root into the container read-only: ```yaml volumes: - /var/run/docker.sock:/var/run/docker.sock:ro # Optional. Reports disk usage for every host filesystem, not just the # one backing /var/lib/docker. - /:/host:ro ``` The agent then reads the host's mount table and reports each filesystem under its real path (`/mnt/data`, not `/host/mnt/data`). The mount is read-only and used for nothing else; if you prefer a different mount point, set `DOCKTAIL_HOST_ROOT` to it. Without it, nothing else changes — every other metric is already the host's. ## What It Never Does - No remote command execution, deployment, or shell access. The protocol is metadata-only and has no exec, deploy, or shell message types. - It never executes anything on the host on behalf of the cloud. - Cloud configuration cannot select an arbitrary network destination; checks are bound to services discovered locally from Docker. A tailnet-health request carries only service names the agent itself published, and carries no credentials. - Tailscale credentials stay on the host. Cloud never receives an OAuth client, an API key, or a Tailscale token — only the metadata the agent reads with them. - The connection is outbound only; the agent dials the ingest endpoint and nothing dials in. ## Tailnet Health Cloud's tailnet vantage answers one question: does the Tailscale *control plane* agree that this host is advertising the service and that the service is approved? Answering it needs Tailscale API credentials, so it only works when `TAILSCALE_OAUTH_CLIENT_ID`/`TAILSCALE_OAUTH_CLIENT_SECRET` or `TAILSCALE_API_KEY` is already configured (see [Tailscale Admin Setup](https://docktail.org/docs/tailscale-admin/#tailscale-admin-setup)). No new credential, and none of them ever leaves the host: Cloud asks a question, the agent reads the Tailscale API and answers with metadata. It reads the control plane rather than the host's own `tailscale serve` config because serve state only says "this host believes it is serving." It cannot see that the service is waiting for approval in the Admin Console, that its definition was deleted, or that the advertisement never reached the control plane — cases where the host looks perfectly healthy and nobody can reach the service. This is an approval and advertisement oracle, not a reachability ping. Tailscale exposes no health or liveness probe for a service, so a service Cloud reports as `connected` is one the control plane considers advertised and approved — nothing more. Actual reachability still comes from the local vantage and, for Funnel services, the public one below. Cloud drives the polling: it asks exactly one credentialed host per tailnet on a slow cadence (service state changes when a human approves something), and that answer covers every host on the tailnet. Agents never poll on their own, and each agent enforces its own minimum interval of 60 seconds between control-plane reads, re-serving its previous answer if asked sooner — your Tailscale API quota is yours, and nothing on the Cloud side can spend more of it than that. ## Public Health For a service with Funnel enabled, Cloud makes an ordinary HTTPS request to its public Funnel address from the open internet and reports whether it answered. That is the one check the agent cannot run for you: from inside the tailnet, MagicDNS resolves a Funnel name to a tailnet address, so a request from this host takes the tailnet path and always succeeds. Only an observer outside your tailnet sees what a visitor sees. The agent's part is to say *where* — it reports this node's MagicDNS name alongside the Funnel port and path it configured, so Cloud requests exactly the URL you published instead of guessing one. Nothing else is sent, no credential is involved, and a service without Funnel enabled is never requested at all. HTTP and HTTPS funnels are checked; a `tcp` or `tls-terminated-tcp` funnel is not, because Tailscale's ingress accepts the connection itself and such a check would report healthy even with the host gone. Without credentials the agent still says so explicitly: its hello advertises that it understands tailnet health but has no credentials for it, so Cloud reports "no Tailscale credentials" and links to [Tailscale Admin Setup](https://docktail.org/docs/tailscale-admin/#tailscale-admin-setup) rather than showing an outage — or telling you to upgrade an agent that is already current. Local checks, Docker events, metrics, and incidents are unaffected. ## Identity Each host is identified by its Docker engine ID, used as a stable fingerprint. A workspace key can enroll multiple hosts while its enrollment window is open (one hour by default; the dashboard's agent key settings offer other lengths when a key is created or its enrollment reopened). After the window closes, the key continues to authenticate the hosts it already enrolled but cannot add another fingerprint until an operator reopens enrollment in the Cloud dashboard. An agent waiting for a reopened window retries automatically at a low rate. The agent also reports its own Tailscale node ID and tailnet name when it has a tailnet. The Services API calls the stable identity `nodeId`; it is the same identity exposed as `Self.ID` by `tailscale status --json`. Cloud uses that exact match to associate each service advertisement with its host. The tailnet name groups the hosts that share a control plane, so Cloud knows which single host to ask for tailnet health. Neither is required — without them the agent simply reports fewer signals. ## Connection Problems When Cloud refuses a connection, the agent logs the reason code and what to do about it (`cloud: connection rejected. …` when it will retry, `cloud: stopped reporting until this container restarts. …` when it will not). The explanation repeats about every 30 minutes, so it stays near the end of `docker logs`. DockTail itself keeps serving your services either way; only reporting stops. | Reason | Meaning | What the agent does | Fix | | --- | --- | --- | --- | | `http_401`, `invalid_key` | The workspace key was revoked, its workspace was deleted, or it is mistyped. | Stops. | Create a new agent key at [cloud.docktail.org/settings/agent-keys](https://cloud.docktail.org/settings/agent-keys), set it as `DOCKTAIL_CLOUD_KEY`, and recreate the container (`docker compose up -d`). | | `blocked` | This host was blocked in the dashboard. | Checks again about every 15 minutes for up to 24 hours, then stops. | Unblock it at [cloud.docktail.org/hosts](https://cloud.docktail.org/hosts). After 24 hours, also restart the container. | | `protocol_mismatch` | This DockTail image speaks a wire protocol Cloud no longer accepts. | Checks again about every 15 minutes for up to 24 hours, then stops. | Pull the latest DockTail image and recreate the container. | | `http_403` | Something between the host and Cloud (a proxy, firewall, or WAF) refused the connection. | Checks again about every 15 minutes for up to 24 hours, then stops. | Allow outbound WebSocket connections to Cloud. After 24 hours, also restart the container. | | `enrollment_closed` | The key's enrollment window closed before this host joined. | Retries every 30–60 seconds. | Reopen enrollment for the key at [cloud.docktail.org/settings/agent-keys](https://cloud.docktail.org/settings/agent-keys), or recreate the container with a new key. | | `over_cap` | The workspace has no plan yet, or has reached its plan's host limit. | Retries every 30–60 seconds. | Choose a plan or upgrade it at [cloud.docktail.org/settings/billing](https://cloud.docktail.org/settings/billing), or remove an offline host. The agent reconnects on its own once there is room. | A key cannot be changed while the container runs, so rotating keys always means setting the new `DOCKTAIL_CLOUD_KEY` and recreating the container. Network errors and temporary Cloud outages are not rejections: the agent reconnects on its own with backoff. ## API And MCP Server DockTail Cloud has a REST API and a remote MCP server, so an AI agent (Claude Code, Cursor, Codex, VS Code, Claude Desktop) or a script can read the same hosts, services, incidents, Docker events and captured logs as the dashboard, and find the root cause of an incident. Overview: [docktail.org/cloud/agents](https://docktail.org/cloud/agents/). Both need a workspace with a plan or a running trial, and an API key that a person creates in Cloud: 1. Open the **MCP** page ([cloud.docktail.org/mcp](https://cloud.docktail.org/mcp)) or the **API** page ([cloud.docktail.org/api-access](https://cloud.docktail.org/api-access)) and choose **Create key**. 2. Pick a scope. `read` (the default) reaches every read endpoint and tool, which is enough to investigate any incident. `read_write` can also acknowledge and resolve incidents and start or stop maintenance windows. 3. Copy the key (`dtk_...`). It is shown once. API keys (`dtk_...`) are separate from the workspace key (`dtc_...`) the DockTail agent uses; neither works in place of the other. **MCP server** - Endpoint: `https://cloud.docktail.org/api/mcp`, Streamable HTTP, stateless. - Auth: `Authorization: Bearer dtk_...`. - Claude Code: `claude mcp add --transport http docktail https://cloud.docktail.org/api/mcp --header "Authorization: Bearer dtk_..."`. The MCP page in Cloud shows the configuration for the other clients; Claude Desktop connects through the `mcp-remote` bridge. - Read tools: `get_workspace`, `list_hosts`, `get_host`, `list_services`, `get_service`, `list_incidents`, `get_incident`, `get_incident_logs`, `list_events`, `list_maintenance_windows`. - Write tools (`read_write` keys only): `acknowledge_incident`, `resolve_incident`, `start_maintenance`, `stop_maintenance`. - Start with `get_workspace`. Use write tools only when the user asks for that change. **REST API** - Base URL: `https://cloud.docktail.org/api/v1`, same bearer key. - OpenAPI 3.1 document: `https://cloud.docktail.org/api/v1/openapi.json` (no key needed). - Reads: `GET /workspace`, `/hosts`, `/hosts/{id}`, `/services`, `/services/{id}`, `/incidents`, `/incidents/{id}`, `/incidents/{id}/logs`, `/events`, `/maintenance`. - Writes (`read_write`): `POST /incidents/{id}/ack`, `/incidents/{id}/resolve`, `/maintenance/start`, `/maintenance/stop`. - Responses are camelCase JSON, request bodies snake_case. Errors are `{"error": ""}`. ```bash curl -H "Authorization: Bearer $DOCKTAIL_API_KEY" \ "https://cloud.docktail.org/api/v1/incidents?status=open" ``` API and MCP share one rate limit of 120 requests per key per minute. Keys are revoked on the same MCP and API pages; revoking one leaves the others working. Neither the API nor the MCP server can run anything on your hosts: they only read Cloud's data and, with `read_write`, change incident and maintenance state in Cloud. Writes are logged under the key's name in Settings → Activity.