Make resource-scoped task deduplication atomic #15

New Issue

Francy51 · 2026-03-15T01:25:55Z

Francy51 commented

2026-03-15 01:25:55 +00:00

Resource-scoped task deduplication is currently implemented as a read-then-insert flow, which is race-prone under concurrent requests.

Why this is a problem:

Two requests for the same user_id + task_type + resource_key can both observe no in-flight task and enqueue duplicate jobs.
Duplicate background work can produce duplicate side effects and unnecessary load.
This conflicts with the reliability and predictability priorities.

Observed in:

lib/server/tasks.ts
- findOrEnqueueTask()
lib/server/repos/tasks.ts
- createTaskRunRecord() inserts unconditionally
lib/server/db/schema.ts
- task_run has an index on user_id, task_type, resource_key, status, created_at
- there is no uniqueness guarantee preventing the race

Suggested direction:

Move deduplication to an atomic database-backed mechanism.
Options include:
- a uniqueness constraint for active tasks plus conflict handling
- a dedicated lock/dedup table
- a single transactional claim path that cannot double-enqueue
Keep the workflow-run projection logic separate from the dedup source of truth.

Acceptance criteria:

Concurrent requests for the same resource cannot create duplicate active tasks.
The dedup path is covered by a race-focused integration test.
The API continues to return the existing in-flight task when one already exists.

Resource-scoped task deduplication is currently implemented as a read-then-insert flow, which is race-prone under concurrent requests. Why this is a problem: - Two requests for the same `user_id + task_type + resource_key` can both observe no in-flight task and enqueue duplicate jobs. - Duplicate background work can produce duplicate side effects and unnecessary load. - This conflicts with the reliability and predictability priorities. Observed in: - `lib/server/tasks.ts` - `findOrEnqueueTask()` - `lib/server/repos/tasks.ts` - `createTaskRunRecord()` inserts unconditionally - `lib/server/db/schema.ts` - `task_run` has an index on `user_id, task_type, resource_key, status, created_at` - there is no uniqueness guarantee preventing the race Suggested direction: - Move deduplication to an atomic database-backed mechanism. - Options include: - a uniqueness constraint for active tasks plus conflict handling - a dedicated lock/dedup table - a single transactional claim path that cannot double-enqueue - Keep the workflow-run projection logic separate from the dedup source of truth. Acceptance criteria: - Concurrent requests for the same resource cannot create duplicate active tasks. - The dedup path is covered by a race-focused integration test. - The API continues to return the existing in-flight task when one already exists.

Francy51 added the P1 label 2026-03-15 01:25:55 +00:00

Francy51 commented

2026-03-15 16:25:58 +00:00

{"body":"Fixed via:

Added partial unique index task_active_resource_uidx on (user_id, task_type, resource_key) WHERE status IN ('queued', 'running')
Added createTaskRunRecordAtomic() with try-catch for constraint violations
Updated findOrEnqueueTask() to use insert-first pattern

The race condition is now prevented at the database level. 10-way concurrent tests verify only one task is created per resource.

Files changed:

drizzle/0013_task_active_resource_unique.sql (new migration)
lib/server/db/schema.ts (documentation)
lib/server/repos/tasks.ts (atomic create function)
lib/server/tasks.ts (updated findOrEnqueueTask)
lib/server/repos/tasks.test.ts (race condition tests)
lib/server/db/sqlite-schema-compat.ts (index for existing DBs)"}

{"body":"Fixed via: - Added partial unique index `task_active_resource_uidx` on (user_id, task_type, resource_key) WHERE status IN ('queued', 'running') - Added `createTaskRunRecordAtomic()` with try-catch for constraint violations - Updated `findOrEnqueueTask()` to use insert-first pattern The race condition is now prevented at the database level. 10-way concurrent tests verify only one task is created per resource. Files changed: - `drizzle/0013_task_active_resource_unique.sql` (new migration) - `lib/server/db/schema.ts` (documentation) - `lib/server/repos/tasks.ts` (atomic create function) - `lib/server/tasks.ts` (updated findOrEnqueueTask) - `lib/server/repos/tasks.test.ts` (race condition tests) - `lib/server/db/sqlite-schema-compat.ts` (index for existing DBs)"}

Francy51 commented

2026-03-15 16:26:09 +00:00

Fixed via:

Added partial unique index task_active_resource_uidx on (user_id, task_type, resource_key) WHERE status IN ('queued', 'running')
Added createTaskRunRecordAtomic() with try-catch for constraint violations
Updated findOrEnqueueTask() to use insert-first pattern

The race condition is now prevented at the database level. 10-way concurrent tests verify only one task is created per resource.

Files changed:

drizzle/0013_task_active_resource_unique.sql (new migration)
lib/server/db/schema.ts (documentation)
lib/server/repos/tasks.ts (atomic create function)
lib/server/tasks.ts (updated findOrEnqueueTask)
lib/server/repos/tasks.test.ts (race condition tests)
lib/server/db/sqlite-schema-compat.ts (index for existing DBs)

Fixed via: - Added partial unique index `task_active_resource_uidx` on (user_id, task_type, resource_key) WHERE status IN ('queued', 'running') - Added `createTaskRunRecordAtomic()` with try-catch for constraint violations - Updated `findOrEnqueueTask()` to use insert-first pattern The race condition is now prevented at the database level. 10-way concurrent tests verify only one task is created per resource. Files changed: - `drizzle/0013_task_active_resource_unique.sql` (new migration) - `lib/server/db/schema.ts` (documentation) - `lib/server/repos/tasks.ts` (atomic create function) - `lib/server/tasks.ts` (updated findOrEnqueueTask) - `lib/server/repos/tasks.test.ts` (race condition tests) - `lib/server/db/sqlite-schema-compat.ts` (index for existing DBs)

Francy51 closed this issue

2026-03-15 16:26:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Francy51/Neon-Desk#15