Skip to content
Celestial Intelligence Docs

Starflow-server state — 2026-05-13 21:30 UTC

2026-05-25 amendment (task M1): the public URL renamed from starflow-prod.celestialintelligence.co → starflow.celestialintelligence.co (dropping the -prod internal-env suffix that leaked into customer-visible surfaces). At the same time the server moved behind Caddy on entmoot for HTTPS — previously raw HTTP on port 7702. The old hostname now 301s to the new one + the GitHub webhook (hook id 605920001) was repointed to https://starflow.celestialintelligence.co/webhooks/github. References below preserve the historical URL for accuracy.

Discovered while migrating scribe-newswatch off GitHub Actions. Two issues found; one fixed, one captured for next session.

Fixed in this session

Section titled “Fixed in this session”

  • GitHub webhook URL was stale. The repo’s webhook (hooks/605920001) pointed at celestial-starflow-prod.fly.dev (dead since the Railway migration). Every delivery for ~a month returned 502. Updated to https://starflow-server-production.up.railway.app/webhooks/github and synced the STARFLOW_WEBHOOK_SECRET between Railway env and GitHub config. Ping now returns 200; push events now create pipeline runs.

  • Railway watchPatterns on starflow-server were too broad. Included package-lock.json / package.json / tsconfig.json / packages/storage/** / packages/supabase/migrations/** — almost every commit matched, redeploying starflow-server (and wiping its in-memory run state) on every push. Tightened to packages/starflow-* + workspace deps + the two yaml configs starflow loads at startup (starflow.yaml, starsystem.prod.yaml).

Update — 2026-05-23

Section titled “Update — 2026-05-23”

The two outstanding items below are both resolved:

  • Crash-loop on packages/starflow-* pushes — Dockerfile still had COPY starsystem.prod.yaml after ADR-017 retired the file. Every matched push failed with failed to compute cache key. Fixed in 556346f; Railway watchPatterns also pruned to drop the orphan entry. Deploy 5686c87 confirmed green.
  • Runs stuck in ‘running’ forever — server.ts:259 .catch() logged the error but never called stateAdapter.onRunError(), so any throw before engine.ts:220’s graceful-error path left the workflow_runs row at status='running' indefinitely. Fixed in 5686c87. Failed runs now correctly transition to status='error' with the error message persisted.

What this does not fix:

  • Whether the Alpine image actually has bash/git/node to run pipeline run: steps. We’ll find out the next time a real pipeline fires — the error (if any) will now be visible in /api/runs.
  • Mid-execution hangs (node handler that never throws or returns) — still need a separate timeout mechanism.

Final update — 2026-05-23 (round 3): RESOLVED via Hetzner migration

Section titled “Final update — 2026-05-23 (round 3): RESOLVED via Hetzner migration”

Status: fully resolved. starflow-server is off Railway and running on the existing Hetzner box (entmoot-db-node, 5.161.83.131) alongside scribe.newswatch. Smoke pipeline completes status=success in ~108ms with all events persisted. The original “runs hang in ‘running’ forever” bug is gone — was 100% the stubbed Promise issue from packages/starflow-server/stubs/.

What the migration shipped:

  1. esbuild bundle (commit e29cdb3). npm run build now produces dist/entrypoint.bundle.mjs (6.8 MB, one self-contained file with every @celestial/* workspace dep inlined). Production node never resolves @celestial/storage etc. at runtime. See packages/starflow-server/esbuild.config.mjs for the config + the pinned external native modules.

  2. Hetzner deploy via ss (overlay in packages/starflow/starflow.ssmod.yaml). type: external, host: 5.161.83.131. _artifact.workspace_packages collapsed to just @celestial/starflow-server (bundle does the rest). _artifact.cmd points at the bundle. Saves ~80% artifact size and eliminates toposort fragility.

  3. DNS via Cloudflare (NOT Porkbun — celestialintelligence.co nameservers are at CF). A record starflow-prod.celestialintelligence.co → 5.161.83.131, TTL 300, proxied=false (port 7702 needs direct).

  4. GitHub webhook URL swap (hook id 605920001) from the Railway subdomain to http://starflow-prod.celestialintelligence.co:7702/webhooks/github. Redelivery test returned 200 immediately. Shared secret was already in the vault; no re-sync needed.

  5. Railway service deleted (service id 0b5da70a-...). The crash-loop emails are over.

Outstanding follow-ups (small, not blocking):

  • Post-extract npm rebuild for native deps. Today this is a manual SSH after each ss artifact deploy — the build machine is darwin-arm64 and ships its better-sqlite3 binary, which won’t run on linux-x64. Either add a post-extract hook to ss artifact deploy, or set npm_config_target_platform=linux npm_config_target_arch=x64 during the artifact’s npm install step. Tracked as task #99.

  • TLS / HTTPS in front of port 7702. Currently plain HTTP. Webhook HMAC verification means payload integrity is fine, but hygiene says put Cloudflare in front (would require dropping to a CF-proxiable port or running the app on 443/8443 with cert via Caddy). Not urgent — GitHub is fine with plain HTTP and the secret never travels in cleartext.

  • starsystem.prod.yaml temp file. ss artifact deploy doesn’t yet accept --workspace-file; we work around by cping the legacy-shim-generated yaml into place. Tracked as part of audit task §73 (CLI wart) — fix by adding --workspace-file to artifact-deploy.

Update — 2026-05-23 (round 2): a deeper bug surfaced

Section titled “Update — 2026-05-23 (round 2): a deeper bug surfaced”

Tested the fix with a real push to packages/lens/README.md (commit 3e5e74b). 5 pipelines fired correctly (ci, lens-dev, docs-dev, marketing-dev, sync-subtrees) — webhook → run-row creation works cleanly. But all five stayed at status='running' events=[] for 10+ minutes with no error.

Added a smoke pipeline (one echo step, commit c2f2db5) and triggered it manually. echo "starflow smoke ok" also hangs at running/events: [] for 90+ seconds with no error.

Conclusions:

  • Not a slim-image issue (echo is busybox builtin).
  • Not “real pipelines are slow” — even instant work hangs.
  • The engine queue / handler execution path is hanging without throwing. onRunError doesn’t fire because nothing throws.

Most likely cause: one of the stubbed packages in packages/starflow-server/stubs/ (@celestial/scribe, @celestial/storage, @celestial/agent-host-ports) has a method that returns a never-settling Promise instead of throwing or returning. The engine’s first await on a step handler hangs forever.

The drift-detection test added in commit 0929f0d was a code smell flagging this risk — the stub architecture is fighting us.

Recommended path forward: stop stubbing. Either rebuild the Dockerfile to use the real workspace packages (likely requires dropping Alpine for a Debian-based image so the submodule deps build), or migrate starflow-server back to Fly.io (where it was before Railway) where real workspace support is first-class. The runbook author has been on Railway for a month chasing stub bugs; this is the second silent failure mode found in that time.

Status: diagnosed; fix deferred pending provider/architecture decision.

Original (resolved) — Still outstanding

Section titled “Original (resolved) — Still outstanding”

Pipeline runs are created but never appear to execute past “running” state with zero events.

Confirmed:

  • POST /api/pipelines/<name>/trigger returns { ok: true, runId }
  • The run row is inserted into workflow_runs as status=‘running’
  • engine.execute() is fired
  • activeRuns.set(runId, ...) happens
  • BUT: status never moves to completed / failed; events: [] forever; durationMs: null; completedAt: null

Suspicion (not confirmed): the bash run: steps in my pipelines require tools (git clone, npm ci, npm run build, node) that the slim starflow-server Docker image may not fully support, OR something earlier in engine.execute() throws and the .catch() on line ~259 of server.ts logs to console (not captured in /api/runs).

What to dig into next session:

  • Tail Railway deployment logs while triggering a manual run; look for the console.error("[starflow] run ${runId} fatal error: …") branch from server.ts ~262.
  • Check whether the engine’s run: node handler can actually shell out in the container — might need apk add bash or similar at the Dockerfile level.
  • Verify workflow_runs table updates actually fire from the state adapter (the table insert is at server.ts:244-247; the updates happen elsewhere — maybe MirroringWorkflowStateAdapter or the inner ScribeStateAdapter is failing silently).

What still works regardless

Section titled “What still works regardless”
  • Webhook receives + matches pipelines + creates run rows ✓
  • Manual trigger via POST /api/pipelines/<name>/trigger creates run rows ✓
  • Schedule-based runs registered at startup (nightly, spacecraft- scheduled-agent) per server logs ✓
  • The artifact build (ss artifact build scribe-newswatch --env=prod) works perfectly when run locally — we just can’t trigger it via starflow yet.

Acceptable short-term workflow: keep building artifacts on the laptop; once execution is unblocked, the same pipeline runs hands-off on starflow.