Switch poll loop to per-instance dispatched jobs #18

New issue

Closed

opened 2026-04-26 01:26:43 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-04-26 01:26:43 +02:00

Owner

Context

PollInstancesCommand::handle() polls instances sequentially within a single command invocation. With withoutOverlapping(5) and timeout(10) per HTTP request, the budget is roughly 50 seconds of slow instances before the 5-minute lock window backs up.

For v0.1's expected 1-5 instances this is fine. Beyond ~5 instances, sequential polling becomes a bottleneck (especially if any instance returns slowly).

Goal

Dispatch each instance's poll as a separate queued job. The command becomes a thin dispatcher that fires N jobs and returns immediately. The queue worker handles concurrency.

Acceptance criteria

PollInstancesCommand::handle() dispatches a job per enabled instance instead of calling the action directly.
New PollFediverseInstanceJob (or similar) wraps PollFediverseAction::execute($instance).
Failure isolation moves from the command's per-iteration try/catch to the job's failed() method (Laravel queue convention).
The command's exit code semantics change: it can no longer report "any instance failed" since failures happen async. New semantic: exit 0 if dispatch succeeded for all instances; exit 1 only if dispatch itself failed (e.g. queue connection refused).
The scheduler entry in routes/console.php stays the same.
Tests updated: PollInstancesCommandTest asserts jobs are dispatched (using Bus::fake()) instead of action invocations.
Add a new test exercising job behavior (action invoked correctly, failure handling).

Risks

Queue worker must be running in production. Likely already true for crawler work but worth verifying.
Job retry semantics — default Laravel job retry can cause duplicate UrlDiscovered events if the action partially succeeds. Decide: tries=1 (no retry, lose events on failure) or tries=3 with idempotency in #4's listener.

Trigger condition

File this when instance count is approaching the v0.1 sequential budget, OR when slow instances are observed causing scheduled lock-window misses.

## Context `PollInstancesCommand::handle()` polls instances **sequentially** within a single command invocation. With `withoutOverlapping(5)` and `timeout(10)` per HTTP request, the budget is roughly 50 seconds of slow instances before the 5-minute lock window backs up. For v0.1's expected 1-5 instances this is fine. Beyond ~5 instances, sequential polling becomes a bottleneck (especially if any instance returns slowly). ## Goal Dispatch each instance's poll as a separate queued job. The command becomes a thin dispatcher that fires N jobs and returns immediately. The queue worker handles concurrency. ## Acceptance criteria - `PollInstancesCommand::handle()` dispatches a job per enabled instance instead of calling the action directly. - New `PollFediverseInstanceJob` (or similar) wraps `PollFediverseAction::execute($instance)`. - Failure isolation moves from the command's per-iteration try/catch to the job's `failed()` method (Laravel queue convention). - The command's exit code semantics change: it can no longer report "any instance failed" since failures happen async. New semantic: exit 0 if dispatch succeeded for all instances; exit 1 only if dispatch itself failed (e.g. queue connection refused). - The scheduler entry in `routes/console.php` stays the same. - Tests updated: `PollInstancesCommandTest` asserts jobs are dispatched (using `Bus::fake()`) instead of action invocations. - Add a new test exercising job behavior (action invoked correctly, failure handling). ## Risks - Queue worker must be running in production. Likely already true for crawler work but worth verifying. - Job retry semantics — default Laravel job retry can cause duplicate `UrlDiscovered` events if the action partially succeeds. Decide: `tries=1` (no retry, lose events on failure) or `tries=3` with idempotency in #4's listener. ## Trigger condition File this when instance count is approaching the v0.1 sequential budget, OR when slow instances are observed causing scheduled lock-window misses.