sf-unstick — recover from stuck starflow runs

Symptoms:

• /api/runs shows many rows with status: "running" that started minutes or hours ago.
• /api/status shows activeRuns: [] even though those rows say otherwise.
• New POST /api/pipelines/<name>/trigger calls return a runId but the run never produces events and stays running forever.

Root cause is always the same: starflow-server lost its in-memory queue state — restart, OOM, deploy, crash — and the DB still believes those runs are in flight. The DB is the durable source of truth; the in-memory queue is not. So on the next boot every row that says running is by definition an orphan.

The runtime now self-heals at boot: StarflowServer._reconcileOrphansOnBoot calls WorkflowStateAdapter.reconcileOrphans() shortly after the storage adapter comes up, marking every running row as error with the message "orphaned: server restarted before completion". New deployments and ordinary systemctl restart starflow-server ops are pushbutton — nothing to do.

The two recovery paths below cover the cases where you need a manual nudge.

---

Path 1 — starflow is alive, you just want to wipe orphans

Use this when /health returns 200 and /api/status is responsive, but /api/runs is full of stale running rows from a prior crash that finished before this codepath shipped.

Locally:

npm run sf:unstick

What it does:

1. Materialises the celestial vault to pick up STARFLOW_URL + STARFLOW_API_TOKEN.
2. Counts current running rows via /api/runs?limit=500.
3. Calls POST /api/admin/reconcile-orphans (Bearer-gated). The server updates every running row to error and returns the count.
4. Re-counts to confirm.

Equivalent through the Starflow CI pipeline:

sf run sf-mark-orphaned-runs

The pipeline runs the same script on the starflow host. Pick the pipeline form when you want the action recorded as a run in the audit log; pick the local form when starflow is misbehaving and you don’t want the recovery itself to be a workflow run.

Path 2 — starflow’s own runner is wedged

Use this when /api/admin/reconcile-orphans returns success but new runs still hang at running with no events, OR when /health is slow / 5xx. The fix is a process restart — the boot-time reconciler will mark the orphans as part of coming back up.

npm run sf:unstick:remote

What it does:

1. Materialises secrets (as Path 1) — needs STARFLOW_URL to poll /health afterwards.
2. Counts running rows via the API (last view before bouncing).
3. SSHes to the starflow host (entmoot by default) and runs systemctl restart starflow-server. SSH credentials default to ~/.ssh/entmoot_hetzner_ed25519; override with --key= or STARFLOW_SSH_KEY.
4. Polls /health for up to 60s.
5. Re-counts running to confirm the boot reconciler did its job.

This step requires SSH access to the host. Operators with that access should be a short list; if you don’t have it, escalate.

Why both?

The API path is fast and safe — no process bounce, no risk of dropping a SSE connection, no momentary 5xx. Use it whenever it works.

The SSH path is the bigger hammer — it covers the case where the API itself is unhealthy and the orphan-marking action can’t be served.

When to suspect a bug instead

Self-healing on boot should make this runbook a rare reach. If you find yourself running these more than once a week, the underlying issue is probably not orphan cleanup — it’s whatever keeps killing the process. Check journalctl for OOM kills, look at memory usage, and audit any recent deploys for regressions.