Scheduler tick has race condition — multi-node deploy will double-fire recorder starts #103

New issue

Closed

opened 2026-05-26 18:18:31 -04:00 by zgaetano · 1 comment

zgaetano commented

2026-05-26 18:18:31 -04:00

Owner

Fixed in 04ce096. scheduler.js now wraps each tick in pg_try_advisory_lock(8210301) so exactly one replica processes a given interval. Pending → starting and running → stopping transitions are atomic via UPDATE … FOR UPDATE SKIP LOCKED, so even without the lock a row can only be claimed once.

Fixed in 04ce096. `scheduler.js` now wraps each tick in `pg_try_advisory_lock(8210301)` so exactly one replica processes a given interval. Pending → starting and running → stopping transitions are atomic via `UPDATE … FOR UPDATE SKIP LOCKED`, so even without the lock a row can only be claimed once.

~~zgaetano referenced this issue 2026-05-26 18:22:49 -04:00~~

Tracker: 2026-05-26 audit session — 27 bugs logged (#100-#127) #128

zgaetano commented

2026-05-26 19:33:19 -04:00

Author

Owner

Fix Plan — #103 Scheduler race condition (multi-node double-fire)

Root cause: scheduler.js:31-130 polls every 15s. Two mam-api instances both SELECT same pending row, both call /start, second races on container create.

Fix — use SELECT ... FOR UPDATE SKIP LOCKED:

// In scheduler tick:
const client = await pool.connect();
try {
  await client.query("BEGIN");
  const { rows: [job] } = await client.query(
    `UPDATE recorder_schedules SET status = "triggering"
     WHERE id = (
       SELECT id FROM recorder_schedules
       WHERE status = "pending" AND next_run <= NOW()
       ORDER BY next_run ASC
       FOR UPDATE SKIP LOCKED
       LIMIT 1
     )
     RETURNING *`
  );
  await client.query("COMMIT");
  if (job) await startRecorder(job);
} finally {
  client.release();
}

This ensures only one instance claims each pending row.

Files: src/scheduler.js:31-130
Effort: ~1h
**Priority: P1 — only triggers in multi-node deploy

## Fix Plan — #103 Scheduler race condition (multi-node double-fire) **Root cause:** `scheduler.js:31-130` polls every 15s. Two mam-api instances both SELECT same `pending` row, both call `/start`, second races on container create. **Fix — use `SELECT ... FOR UPDATE SKIP LOCKED`:** ```js // In scheduler tick: const client = await pool.connect(); try { await client.query("BEGIN"); const { rows: [job] } = await client.query( `UPDATE recorder_schedules SET status = "triggering" WHERE id = ( SELECT id FROM recorder_schedules WHERE status = "pending" AND next_run <= NOW() ORDER BY next_run ASC FOR UPDATE SKIP LOCKED LIMIT 1 ) RETURNING *` ); await client.query("COMMIT"); if (job) await startRecorder(job); } finally { client.release(); } ``` This ensures only one instance claims each pending row. **Files:** `src/scheduler.js:31-130` **Effort:** ~1h **Priority: P1 — only triggers in multi-node deploy

zgaetano referenced this issue

2026-05-26 19:42:16 -04:00

Tracker: 2026-05-26 audit session — 27 bugs logged (#100-#127) #128

zgaetano referenced this issue from a commit

2026-05-26 22:06:25 -04:00

chore: 1.2 ship-prep sweep — close 38 issues

zgaetano closed this issue

2026-05-26 22:09:41 -04:00