Monitoring a Space App

Plugins → Space Apps → Details & logs on the app. The process monitoring section sits at the top of the dialog and refreshes every 3 seconds. Present in both the Web UI and the desktop app, from the same endpoint with the same numbers.

A Space App is a process the daemon starts and then mostly forgets about. When it misbehaves the questions are always the same: is it running, since when, how many times has it restarted, how much CPU and RAM is it using, who is it talking to, and what does the log say. Each answer used to live somewhere different.

Reading the status line

running   pid 67399   port 4740   up 1m 45s   1 launch   sandbox: seatbelt
health 200 · 0ms      [Restart] [Open] [Open folder]

• running / running but not answering / not running are three different states. "Supervised by the daemon" does not mean "working": an app returning 500 or hanging is still a live process. The verdict comes from a real health check (the runtime.healthPath from the manifest, default /), with the status code and the latency.
• launch count — counted since the daemon started. A number that climbs steadily is a crash loop, which every other screen renders identically to a healthy app: the supervisor restarts it after each death, so whenever you look it says "running". Past 3 the panel says so outright and points at the log.
• sandbox — the mechanism actually applied to the running process (seatbelt / bubblewrap / nothing shown when it runs free), recorded at launch rather than read back from configuration: configuration can have been edited since, and the thing that is running is the thing worth reporting. See Sandboxing a Space App.

"Running" and "started by the daemon" are different things

The daemon does not relaunch a healthy app: if the app's fixed port answers, it uses that process as-is. That happens constantly — every app that survives a daemon restart lands in this category.

The consequence is that the launcher has no child record for it. The first version of this screen therefore reported not running for an app that was serving perfectly well.

How it is detected now: with no child record, the daemon probes the port the manifest knows about (runtime.port, or the runtime.url it recorded after a successful run), does one machine-wide lsof pass to find the pid holding that port, and then measures CPU/RAM and uptime from ps like any other app. It shows up as:

running (outside the daemon)   pid 18274 · 1h 12m        unknown   needs restart

• **unknown**, not seatbelt: this daemon did not build a profile for that process, so it cannot claim it is confined. If the configuration says sandbox-on and the process was adopted, it is almost certainly not confined — hence the row always reads needs restart.
• No launch count: inventing a number for a process you did not start is meaningless.

Why every app used to say "outside the daemon"

Three bugs in a chain, all fixed:

1. The daemon only caught SIGINT. It awaited ctrl_c() while the desktop app stops it with kill -TERM followed by SIGKILL 800 ms later. The shutdown block — where the app launcher's shutdown runs — had never executed the way people actually quit. Every Space App survived every exit.
2. Shutdown was too slow to fit. It killed apps one at a time with a 2-second grace each: a few dozen apps needed a minute, inside an 800 ms window. Now it sends SIGTERM to all of them, waits once for 300 ms, and SIGKILLs the remainder — measured at about 300 ms in total.
3. A new daemon adopted the old corpses. Seeing the fixed port still answering, it just used that process — so an app could run for weeks on old code, from an old directory, with no sandbox at all. Now the daemon reclaims the port: it kills that process and relaunches properly.

Reclaiming is guarded — the daemon only kills when the process's working directory is inside the app's installation directory (lsof -d cwd). Your own dev server that happens to share a port is left alone and logged; a daemon that kills unfamiliar processes at every startup would be worse than the problem.

The supervisor learned the same lesson: "the port answers" used to mean healthy, so a stranger appearing mid-session was never noticed. Now a port answering with no child record is treated as work to do — one reclaim attempt per daemon run (if it fails, note it and stop; do not retry every tick).

outside the daemon / unknown is therefore a rare exception now, not the default state after every restart. All three fixes landed in v0.4.5, so they need a current daemon. To clear out apps left behind by an older one:

pkill -f "$HOME/senclaw/workspace/space-apps/"

CPU / RAM

Measured per process group (pgid), not per pid: npm start spawns sh → npm → node and the memory worth knowing about is in the child. The table lists each process (pid, CPU %, RAM MB, uptime, command) — enough to see an app that spawns a dozen children, or which one is eating the machine.

The numbers come from the machine's ps, through the same parser the Sandbox page uses.

Network

Two layers, because they answer different questions:

1. Open sockets (lsof by pid): a LISTEN socket proves the app really is serving its port; ESTABLISHED sockets show who it is talking to — including loopback (the daemon, other apps).
2. The allowlist proxy, when the app runs sandboxed in "only these sites" mode: how many requests were allowed and denied, and which domains were just denied. This is usually the answer to "the app is broken and I cannot see why" — it needs a site nobody listed. Add it with the Sandbox button.

If the machine has no lsof, the socket section is empty with a note rather than breaking the panel. It is strictly read-only: this function must never become lsof -t … | kill, which is exactly how an earlier incident killed the daemon itself.

Everything you need to reproduce it by hand

The bottom of the panel is what you need to run the app yourself in a terminal:

Plus Open (open the app's URL in the system browser) and Open folder (reveal the app directory in Finder/Explorer, on desktop). The log contents are in the Logs section just below, reloading every 2 seconds, with a clear button.

Watching the whole fleet

Plugins → Sandbox carries a Space Apps — per-app sandbox card: one row per app with a server process, refreshing every 5 seconds.

The first column is the valuable one, because it catches what no other screen sees: an app configured with a sandbox but currently running without one, since the profile is fixed at launch while the configuration can be edited afterwards. Those rows read needs restart, with the restart button beside them.

Three buttons per row, in "look → change → act" order: a heartbeat opens that app's detailed monitoring dialog, a flask opens its Sandbox dialog, and a circular arrow restarts it.

The list paginates at 10 apps per page (47 installed apps is an ordinary number, and one long list buries every other card). Sortable by state (the default), name, sandbox on/off, network, CPU/RAM or launch count — the web sorts by clicking a column header, the desktop by a "Sort" selector with a direction toggle. The default is state, not name: with 47 apps and a handful running, an A→Z list opens on idle apps while the interesting ones sit on page three. Changing the sort returns you to page 1, because page numbers from the old order mean nothing.

The whole card is one API call, GET /api/space/apps/sandbox-overview — a single ps for the entire list rather than one per app.

The runtime API

GET /api/space/apps/:id/runtime returns a snapshot:

{
  "running": true,
  "launches": 1,
  "process": { "pid": 67399, "pgid": 67399, "port": 4740,
               "url": "http://127.0.0.1:4740", "uptimeMs": 105000,
               "isolation": "seatbelt" },
  "health":  { "url": "…/api/status", "ok": true, "status": 200, "ms": 0 },
  "resources": { "cpu": 0.0, "rssMb": 10.9, "processes": [] },
  "network": { "connections": [ { "proto": "TCP", "local": "127.0.0.1:4740",
                                 "remote": null, "state": "LISTEN" } ],
               "proxy": { "port": 59876, "stats": { "allowed": 1, "denied": 2,
                          "recentDenied": ["wikipedia.org"] } } },
  "sandbox": { "enabled": true, "readMode": "open", "network": "hosts", "hosts": [] },
  "log": { "path": "…/.senclaw/runtime.log", "bytes": 14342 },
  "launch": { "cwd": "…", "command": "./ba", "env": [["PORT", "4740"]] }
}

Everything is best-effort: a missing lsof, a process that dies mid-measurement, a health check that times out — each becomes a note in the payload rather than a failed request. A monitoring screen that returns 500 exactly when the thing it monitors breaks is useless at the only moment it matters.

Further reading

• Sandboxing a Space App — configuring the cage this screen reports on.
• Running code in a sandbox — the engine and its tools.
• Building a Space App — where runtime.port, runtime.start and runtime.healthPath come from.

In the SenClaw repo

The endpoint and socket reading live in src/gateway/ui_server/space_runtime.rs, the pid/uptime/launch bookkeeping in src/gateway/ui_server/space_mcp.rs (runtime_info), and the per-group CPU/RAM measurement in src/sandbox/monitor.rs (stats_for_groups). The single-app UIs are web/src/components/space/AppRuntimePanel.tsx and desktop_app/lib/features/plugins/space_app_runtime_panel.dart; the fleet card is web/src/components/plugins/SandboxAppsCard.tsx and the _appsCard section of desktop_app/lib/features/plugins/sandbox_panel.dart.