Belga discovery skips the newest article — offset=1 should be offset=0 #158

New issue

Closed

opened 2026-08-16 17:35:28 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-16 17:35:28 +02:00

Owner

Summary

The Belga API url in config/feed.php uses offset=1. The parameter is 0-based, so every fetch silently drops the newest press release. An article only becomes visible to FFR once a newer one is published and pushes it into position 2 — meaning the most recent Belga article is always missing, for hours at a time.

Evidence

Same endpoint, same count, differing only in offset (2026-08-16):

offset=0  →  50 rows, first id 35545, last id 35398
offset=1  →  50 rows, first id 35544, last id 35395

Both report _meta.total = 14732. offset=1 is an item offset, not a page number, and it starts one item late.

Observed on PROD

v1.4.1, feed 2 (Belga), inspected 2026-08-16:

Newest article in FFR: 35543, published 2026-08-16T06:01:17
Missing: 35544, published 2026-08-16T06:07:47 — six minutes later
Belga fetches ran at 14:35, 15:05 and 15:22 that day, each logging "Fetched 50 articles from Belga", and none of them picked up 35544

35544 was the newest article from 06:07 onward, so it was dropped by every fetch for ~11 hours. It would only have been ingested after 35545 appeared at 17:32:55.

Why it went unnoticed

The failure is invisible from inside the app:

The fetch reports success and a full count of 50 — FetchWebsiteArticlesAction pushes every article it listed, new or already-known, so "Fetched 50" says nothing about how many were new
No error, no warning, no FEED_EMPTY (the collection is never empty)
The skipped article does eventually arrive, so the backlog looks complete in hindsight — only the newest article is ever absent

Fix

config/feed.php, providers.belga.languages.en.url: offset=1 → offset=0.

A config change alone is not sufficient, and shipping one on its own would make things worse. feeds.url is read from the database at fetch time; config/feed.php only ever seeds new rows via CreateFeedAction's firstOrCreate(['url' => $url], …). Because the url is the lookup key, changing it in config does not update the existing row — it fails to match and inserts a second belga feed, leaving the stale one active.

This needs the established pattern: a data migration keyed on provider, modelled on 2024_01_01_000012_sync_belga_feed_to_website_discovery.php, which adopts the row already on the target url and deactivates superseded ones. Two schema constraints apply — feeds.url is UNIQUE while feeds.provider is not, and routes.feed_id cascades on delete, so superseded rows must be deactivated rather than deleted.

Acceptance criteria

config/feed.php uses offset=0 for the Belga endpoint
A data migration updates the existing belga feed row, keyed on provider, not on the old url
No duplicate belga feed row is created; any superseded row is deactivated, not deleted
The migration is idempotent and safe on environments already on the target url
Migration touches no network (see #150)
Test covers the migration against a row on the old url and a row already on the new one

#157 "Articles validated while a feed has no active route are permanently unroutable" — filed from the same investigation, but a distinct defect. #157 concerns articles stranded after they are saved; this concerns articles never fetched at all. Both contributed to "no Belga articles" being reported.
#115 / #116 — same silent-failure family: the fetch reports success while yielding less than it should. Note #116's warning would not catch this, since the collection is non-empty.
#124 — Belga language configuration. Confirmed during this investigation that newsroom 70 is English-only: language=NL and language=FR both return an empty data array.

## Summary The Belga API url in `config/feed.php` uses `offset=1`. The parameter is **0-based**, so every fetch silently drops the newest press release. An article only becomes visible to FFR once a *newer* one is published and pushes it into position 2 — meaning the most recent Belga article is always missing, for hours at a time. ## Evidence Same endpoint, same `count`, differing only in `offset` (2026-08-16): ``` offset=0 → 50 rows, first id 35545, last id 35398 offset=1 → 50 rows, first id 35544, last id 35395 ``` Both report `_meta.total = 14732`. `offset=1` is an item offset, not a page number, and it starts one item late. ## Observed on PROD v1.4.1, feed 2 (Belga), inspected 2026-08-16: - Newest article in FFR: **35543**, published `2026-08-16T06:01:17` - Missing: **35544**, published `2026-08-16T06:07:47` — six minutes later - Belga fetches ran at 14:35, 15:05 and 15:22 that day, each logging "Fetched 50 articles from Belga", and none of them picked up 35544 35544 was the newest article from 06:07 onward, so it was dropped by every fetch for ~11 hours. It would only have been ingested after 35545 appeared at `17:32:55`. ## Why it went unnoticed The failure is invisible from inside the app: - The fetch reports success and a full count of 50 — `FetchWebsiteArticlesAction` pushes every article it listed, new or already-known, so "Fetched 50" says nothing about how many were new - No error, no warning, no `FEED_EMPTY` (the collection is never empty) - The skipped article does eventually arrive, so the backlog looks complete in hindsight — only the *newest* article is ever absent ## Fix `config/feed.php`, `providers.belga.languages.en.url`: `offset=1` → `offset=0`. **A config change alone is not sufficient**, and shipping one on its own would make things worse. `feeds.url` is read from the **database** at fetch time; `config/feed.php` only ever seeds new rows via `CreateFeedAction`'s `firstOrCreate(['url' => $url], …)`. Because the url is the lookup key, changing it in config does not update the existing row — it fails to match and inserts a **second** belga feed, leaving the stale one active. This needs the established pattern: a data migration keyed on `provider`, modelled on `2024_01_01_000012_sync_belga_feed_to_website_discovery.php`, which adopts the row already on the target url and deactivates superseded ones. Two schema constraints apply — `feeds.url` is UNIQUE while `feeds.provider` is not, and `routes.feed_id` cascades on delete, so superseded rows must be deactivated rather than deleted. ## Acceptance criteria - [ ] `config/feed.php` uses `offset=0` for the Belga endpoint - [ ] A data migration updates the existing belga feed row, keyed on `provider`, not on the old url - [ ] No duplicate belga feed row is created; any superseded row is deactivated, not deleted - [ ] The migration is idempotent and safe on environments already on the target url - [ ] Migration touches no network (see #150) - [ ] Test covers the migration against a row on the old url and a row already on the new one ## Related - #157 "Articles validated while a feed has no active route are permanently unroutable" — filed from the same investigation, but a distinct defect. #157 concerns articles stranded *after* they are saved; this concerns articles never fetched at all. Both contributed to "no Belga articles" being reported. - #115 / #116 — same silent-failure family: the fetch reports success while yielding less than it should. Note #116's warning would **not** catch this, since the collection is non-empty. - #124 — Belga language configuration. Confirmed during this investigation that newsroom 70 is English-only: `language=NL` and `language=FR` both return an empty `data` array.