lvl0/fedi-feed-router
1
0
Fork 0
Code Issues 27 Pull requests Projects Releases 13 Packages Wiki Activity Actions

Detect and notify on failed queue jobs that have sat unretried #131

New issue
Open
opened 2026-08-10 22:58:08 +02:00 by myrmidex · 0 comments
myrmidex commented 2026-08-10 22:58:08 +02:00
Owner
Copy link

Summary

SyncChannelPostsJob failed on 2026-03-08 and sat in the failed-jobs table, unretried and unnoticed, for five months. It was discovered by accident while investigating #123. The duplicate-detection backstop was silently absent the entire time.

Horizon keeps failed jobs for 7 days (config/horizon.php:trim.failed = 10080), then prunes them. A job that exhausts its retries and is pruned by Horizon disappears without a trace — no notification, no log entry, no surviving record.

Current state

  • config/horizon.php sets tries = 1 on the supervisor — jobs get one attempt, no retries
  • Horizon's LongWaitDetected event exists (waits.redis:default = 60) but nothing listens for it
  • CheckFeedStalenessJob demonstrates the notification-deduplication pattern: query for an existing unread notification of the same type before sending
  • NotificationTypeEnum has FEED_STALE, FEED_EMPTY, PUBLISH_FAILED, CREDENTIAL_EXPIRED — no JOB_FAILED or equivalent
  • The failed_jobs table is a Laravel built-in — it has id, connection, queue, payload, exception, failed_at

Design

Approach: scheduled check against the failed_jobs table

Horizon prunes after 7 days, so the check must run more frequently than that. A daily scheduled job that queries failed_jobs for entries newer than the last run is simple and covers any queue, not just sync.

Alternatives considered:

  • Horizon JobFailed event — real-time but requires a listener that ships with Horizon; no simpler than the scheduled approach and misses jobs that failed before the listener was deployed
  • Queue::failing() callback in AppServiceProvider — catches every failure but fires per-job, making deduplication harder and risking notification storms when a systemic issue fails every job

The scheduled approach is consistent with how CheckFeedStalenessJob already works and composes well with the existing notification deduplication pattern.

Notification deduplication

Reuse the CheckFeedStalenessJob pattern: before sending, check for an existing unread JOB_FAILED notification. A single notification per failure is enough — the user can clear it once acted on.

What to include in the notification

  • Job class name (extracted from the failed_jobs payload)
  • Queue name
  • Failed-at timestamp
  • Exception message (first ~200 chars)

Tasks

  • Add JOB_FAILED case to NotificationTypeEnum
  • New CheckFailedJobsJob scheduled job (daily, withoutOverlapping, onOneServer)
  • Job queries failed_jobs for entries since the last notification (or a configurable lookback window)
  • Notification sent via NotificationService with deduplication
  • Configurable via Setting — failed_job_notification_enabled + failed_job_lookback_hours, with 0 to disable
  • Settings UI entries (dark-mode compatible per #88)
  • API surface on SettingsController

Edge cases

  • Horizon not running: If Redis is down or Horizon is stopped, the scheduled job still runs and can't find failed jobs — no false positive, but the check itself should not fail
  • Notification storm on systemic failure: A single root cause (e.g., Lemmy instance down) could fail every publish job. Deduplication by type prevents this; one JOB_FAILED notification covers all
  • Horizon prunes before we check: The lookback window must be shorter than Horizon's trim (7 days). Daily check with a 24h window is safe
  • Job class not in the app: A failed job from a third-party package should still produce a legible notification

Acceptance criteria

  • A failed job appearing in failed_jobs produces a notification within 24 hours
  • Notification appears in the UI bell, not just the log
  • Deduplicated — one unread JOB_FAILED regardless of how many jobs have failed
  • Disabled when failed_job_notification_enabled = false
  • The check itself does not fail if Redis/Horizon is unavailable
  • Tests: failed job produces notification, no failed jobs = no notification, dedup holds, disabled = no-op

Related

  • #123 — the duplicate-post bug where the silent SyncChannelPostsJob failure was discovered
  • #116 — empty-feed detection; same class of problem (silent failure of a background process)
## Summary `SyncChannelPostsJob` failed on 2026-03-08 and sat in the failed-jobs table, unretried and unnoticed, for five months. It was discovered by accident while investigating #123. The duplicate-detection backstop was silently absent the entire time. Horizon keeps failed jobs for 7 days (`config/horizon.php:trim.failed = 10080`), then prunes them. A job that exhausts its retries and is pruned by Horizon disappears without a trace — no notification, no log entry, no surviving record. ## Current state - `config/horizon.php` sets `tries = 1` on the supervisor — jobs get one attempt, no retries - Horizon's `LongWaitDetected` event exists (`waits.redis:default = 60`) but nothing listens for it - `CheckFeedStalenessJob` demonstrates the notification-deduplication pattern: query for an existing unread notification of the same type before sending - `NotificationTypeEnum` has `FEED_STALE`, `FEED_EMPTY`, `PUBLISH_FAILED`, `CREDENTIAL_EXPIRED` — no `JOB_FAILED` or equivalent - The `failed_jobs` table is a Laravel built-in — it has `id`, `connection`, `queue`, `payload`, `exception`, `failed_at` ## Design ### Approach: scheduled check against the `failed_jobs` table Horizon prunes after 7 days, so the check must run more frequently than that. A daily scheduled job that queries `failed_jobs` for entries newer than the last run is simple and covers any queue, not just `sync`. Alternatives considered: - **Horizon `JobFailed` event** — real-time but requires a listener that ships with Horizon; no simpler than the scheduled approach and misses jobs that failed before the listener was deployed - **`Queue::failing()` callback in `AppServiceProvider`** — catches every failure but fires per-job, making deduplication harder and risking notification storms when a systemic issue fails every job The scheduled approach is consistent with how `CheckFeedStalenessJob` already works and composes well with the existing notification deduplication pattern. ### Notification deduplication Reuse the `CheckFeedStalenessJob` pattern: before sending, check for an existing unread `JOB_FAILED` notification. A single notification per failure is enough — the user can clear it once acted on. ### What to include in the notification - Job class name (extracted from the failed_jobs payload) - Queue name - Failed-at timestamp - Exception message (first ~200 chars) ## Tasks - [ ] Add `JOB_FAILED` case to `NotificationTypeEnum` - [ ] New `CheckFailedJobsJob` scheduled job (daily, `withoutOverlapping`, `onOneServer`) - [ ] Job queries `failed_jobs` for entries since the last notification (or a configurable lookback window) - [ ] Notification sent via `NotificationService` with deduplication - [ ] Configurable via `Setting` — `failed_job_notification_enabled` + `failed_job_lookback_hours`, with `0` to disable - [ ] Settings UI entries (dark-mode compatible per #88) - [ ] API surface on `SettingsController` ## Edge cases - **Horizon not running**: If Redis is down or Horizon is stopped, the scheduled job still runs and can't find failed jobs — no false positive, but the check itself should not fail - **Notification storm on systemic failure**: A single root cause (e.g., Lemmy instance down) could fail every publish job. Deduplication by type prevents this; one `JOB_FAILED` notification covers all - **Horizon prunes before we check**: The lookback window must be shorter than Horizon's trim (7 days). Daily check with a 24h window is safe - **Job class not in the app**: A failed job from a third-party package should still produce a legible notification ## Acceptance criteria - [ ] A failed job appearing in `failed_jobs` produces a notification within 24 hours - [ ] Notification appears in the UI bell, not just the log - [ ] Deduplicated — one unread `JOB_FAILED` regardless of how many jobs have failed - [ ] Disabled when `failed_job_notification_enabled = false` - [ ] The check itself does not fail if Redis/Horizon is unavailable - [ ] Tests: failed job produces notification, no failed jobs = no notification, dedup holds, disabled = no-op ## Related - #123 — the duplicate-post bug where the silent `SyncChannelPostsJob` failure was discovered - #116 — empty-feed detection; same class of problem (silent failure of a background process)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.4.2
v1.4.1
v1.4.0
v1.3.7
v1.3.6
v1.3.5
v1.3.4
v1.3.3
v1.3.2
v1.3.1
v1.3.0
v1.2.0
v1.1.0
v1.0.0
v0.2.0
v0.1.0
Labels
Clear labels   
bug

Something is not working

  
devops

All things Ci/CD

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
good first issue

Interested in contributing? Get started here.

  
layout

UI/UX

  
next major release

Include in next Major Release

  
next minor release

Include in next minor release

  
question

More information is needed

  
research

   
testing

All sorts of testing: UI, automation, ...

No labels
bug
devops
duplicate
enhancement
good first issue
layout
next major release
next minor release
question
research
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lvl0/fedi-feed-router#131
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?