lvl0/fedi-feed-router

Fork 0

Belga News feed produces no articles — RSS feed retired, discovery approach TBD #115

New issue

Closed

opened 2026-08-01 08:50:25 +02:00 by myrmidex · 2 comments

myrmidex commented

2026-08-01 08:50:25 +02:00

Owner

Summary

The Belga News feed produces no articles. Root cause confirmed: the RSS feed no longer exists.

The discovery job runs normally and reports success, so this failed silently. The remaining work is deciding how to discover Belga articles without an RSS feed — that question is open and needs research before implementation.

Root cause

Feed config (prod)

rss | https://www.belganewsagency.eu/feed | active:1 | last_fetched:2026-08-01 06:52:01

The job is running (fetched that morning), so this is not a queue or scheduler problem.

The feed URL is dead

https://www.belganewsagency.eu/feed   -> 308 redirect to /feed/
https://www.belganewsagency.eu/feed/  -> 404 (14 KB HTML error page)

ArticleFetcher::getArticlesFromRssFeed() passes that HTML to simplexml_load_string(), which fails, logs "Failed to parse RSS feed XML", and returns an empty collection. The job then completes and updates last_fetched_at — hence the silent failure.

No alternative feed exists

404 on all of: /rss, /rss.xml, /feed.xml, /en/feed, /atom.xml, /index.xml.

The homepage (200 OK) declares no <link rel="alternate"> RSS/Atom element. The public feed appears to have been retired.

Why the VRT approach does not transfer

No Belga homepage parser exists — verified:

Only three Belga files in the repo: BelgaArticleParser, BelgaArticlePageParser, BelgaArticlePageParserTest
VrtHomepageParserAdapter is the only implementation of HomepageParserInterface
config/feed.php → belga.parsers has article and article_page, no homepage key; type => 'rss'

The site is client-side rendered. It is a Next.js app (27 /_next/ references, __NEXT_DATA__ present). Server-side fetch yields:

70 KB of HTML with only 7 unique links, none of them articles
__NEXT_DATA__ payload ~2 KB, containing zero article slugs

VrtHomepageParser works by regex over server-rendered HTML. That approach will find nothing here.

Other routes already ruled out:

Checked	Result
`/api/articles`, `/api/news`	404
`api.belganewsagency.eu`	DNS does not resolve
`/sitemap.xml`	200, but a sitemap index with one stale entry (`belgashare.be`, lastmod 2025-09-30), no article URLs
`/sitemap_index.xml`	404

Open question — research before implementing

Do not start building a parser until the discovery source is settled. The options differ by an order of magnitude in cost:

Find the client-side data endpoint. The article list is populated by an XHR/fetch after page load. Inspect the site in browser DevTools (Network → XHR/Fetch → reload) to identify it. If a stable public JSON endpoint exists, this is simpler and more robust than the VRT scraper — JSON parsing instead of regex, no markup fragility. Do this first; it determines everything else.
Headless browser rendering. Would work, but adds Chromium to the container plus memory overhead and a new failure class, for a single source. Disproportionate unless Belga is important enough to justify it.
Alternative Belga property. Their own sitemap references belgashare.be — worth checking whether that or another Belga property exposes a usable feed.
Retire the feed. If no practical route exists, deactivate it rather than leave it silently failing.

Once an approach is chosen, implementation follows the existing pattern: a thin adapter implementing HomepageParserInterface (see VrtHomepageParserAdapter), a parser class doing extraction, both registered in config/feed.php under belga.parsers.homepage, with type changed from rss to website.

Acceptance criteria

Discovery approach chosen and rationale recorded on this ticket
Belga feed produces articles again, or the feed is deactivated with the reason documented
If a parser is added: registered in config/feed.php, feed type updated to website
Tests use local fixtures, not live HTTP (project rule — tests must work offline)
Regression coverage for the failure mode

A feed whose URL 404s currently updates last_fetched_at and reports success, with only a log warning — nothing surfaces in the UI and no notification fires. CheckFeedStalenessJob and the FEED_STALE notification already exist and could plausibly cover "fetched repeatedly but returned zero articles". That gap applies to all feeds, not just Belga, and is worth its own ticket.

## Summary The Belga News feed produces no articles. **Root cause confirmed: the RSS feed no longer exists.** The discovery job runs normally and reports success, so this failed silently. The remaining work is deciding *how* to discover Belga articles without an RSS feed — that question is open and needs research before implementation. ## Root cause ### Feed config (prod) ``` rss | https://www.belganewsagency.eu/feed | active:1 | last_fetched:2026-08-01 06:52:01 ``` The job is running (fetched that morning), so this is not a queue or scheduler problem. ### The feed URL is dead ``` https://www.belganewsagency.eu/feed -> 308 redirect to /feed/ https://www.belganewsagency.eu/feed/ -> 404 (14 KB HTML error page) ``` `ArticleFetcher::getArticlesFromRssFeed()` passes that HTML to `simplexml_load_string()`, which fails, logs "Failed to parse RSS feed XML", and returns an empty collection. The job then completes and updates `last_fetched_at` — hence the silent failure. ### No alternative feed exists 404 on all of: `/rss`, `/rss.xml`, `/feed.xml`, `/en/feed`, `/atom.xml`, `/index.xml`. The homepage (200 OK) declares no `<link rel="alternate">` RSS/Atom element. The public feed appears to have been retired. ## Why the VRT approach does not transfer No Belga homepage parser exists — verified: - Only three Belga files in the repo: `BelgaArticleParser`, `BelgaArticlePageParser`, `BelgaArticlePageParserTest` - `VrtHomepageParserAdapter` is the **only** implementation of `HomepageParserInterface` - `config/feed.php` → `belga.parsers` has `article` and `article_page`, no `homepage` key; `type => 'rss'` **The site is client-side rendered.** It is a Next.js app (27 `/_next/` references, `__NEXT_DATA__` present). Server-side fetch yields: - 70 KB of HTML with only **7 unique links**, none of them articles - `__NEXT_DATA__` payload ~2 KB, containing **zero article slugs** `VrtHomepageParser` works by regex over server-rendered HTML. That approach will find nothing here. Other routes already ruled out: | Checked | Result | |---|---| | `/api/articles`, `/api/news` | 404 | | `api.belganewsagency.eu` | DNS does not resolve | | `/sitemap.xml` | 200, but a sitemap index with one stale entry (`belgashare.be`, lastmod 2025-09-30), no article URLs | | `/sitemap_index.xml` | 404 | ## Open question — research before implementing **Do not start building a parser until the discovery source is settled.** The options differ by an order of magnitude in cost: 1. **Find the client-side data endpoint.** The article list is populated by an XHR/fetch after page load. Inspect the site in browser DevTools (Network → XHR/Fetch → reload) to identify it. If a stable public JSON endpoint exists, this is *simpler and more robust* than the VRT scraper — JSON parsing instead of regex, no markup fragility. **Do this first; it determines everything else.** 2. **Headless browser rendering.** Would work, but adds Chromium to the container plus memory overhead and a new failure class, for a single source. Disproportionate unless Belga is important enough to justify it. 3. **Alternative Belga property.** Their own sitemap references `belgashare.be` — worth checking whether that or another Belga property exposes a usable feed. 4. **Retire the feed.** If no practical route exists, deactivate it rather than leave it silently failing. Once an approach is chosen, implementation follows the existing pattern: a thin adapter implementing `HomepageParserInterface` (see `VrtHomepageParserAdapter`), a parser class doing extraction, both registered in `config/feed.php` under `belga.parsers.homepage`, with `type` changed from `rss` to `website`. ## Acceptance criteria - [ ] Discovery approach chosen and rationale recorded on this ticket - [ ] Belga feed produces articles again, **or** the feed is deactivated with the reason documented - [ ] If a parser is added: registered in `config/feed.php`, feed `type` updated to `website` - [ ] Tests use local fixtures, not live HTTP (project rule — tests must work offline) - [ ] Regression coverage for the failure mode ## Related A feed whose URL 404s currently updates `last_fetched_at` and reports success, with only a log warning — nothing surfaces in the UI and no notification fires. `CheckFeedStalenessJob` and the `FEED_STALE` notification already exist and could plausibly cover "fetched repeatedly but returned zero articles". That gap applies to all feeds, not just Belga, and is worth its own ticket.

myrmidex added the

bug

label 2026-08-01 08:50:25 +02:00

myrmidex self-assigned this 2026-08-01 08:50:25 +02:00

myrmidex commented

2026-08-01 11:21:12 +02:00

Author

Owner

Root cause found — the RSS feed is gone

Diagnostics from production and against the live site.

Feed config (prod)

rss | https://www.belganewsagency.eu/feed | active:1 | last_fetched:2026-08-01 06:52:01

The job is running (fetched this morning), so this is not a queue or scheduler problem.

The feed URL is dead

https://www.belganewsagency.eu/feed   -> 308 redirect to /feed/
https://www.belganewsagency.eu/feed/  -> 404 (14 KB HTML error page)

getArticlesFromRssFeed() passes that HTML to simplexml_load_string(), which fails, logs "Failed to parse RSS feed XML", and returns an empty collection. The job then completes successfully and updates last_fetched_at — which is why this failed silently with no visible error.

No alternative feed exists

All returned 404: /rss, /rss.xml, /feed.xml, /en/feed, /atom.xml, /index.xml.

The homepage (200 OK) declares no <link rel="alternate"> RSS/Atom element. The site appears to have retired its public feed entirely.

Switching to website scraping is harder than the VRT case

Confirmed first that no Belga homepage parser exists in any form:

Only three Belga files in the repo: BelgaArticleParser, BelgaArticlePageParser, and BelgaArticlePageParserTest
VrtHomepageParserAdapter is the only class implementing HomepageParserInterface
config/feed.php has belga.parsers with article and article_page but no homepage key, and type => 'rss'

The blocker: the site is client-side rendered

The homepage is a Next.js app (27 /_next/ references, __NEXT_DATA__ present). Fetching it server-side yields:

70 KB of HTML containing only 7 unique links, none of them articles
__NEXT_DATA__ payload is only ~2 KB and contains zero article slugs

Article content is loaded client-side after page load. A regex-over-HTML approach like VrtHomepageParser will find nothing.

Other discovery routes checked:

/api/articles, /api/news → 404
api.belganewsagency.eu → DNS does not resolve
/sitemap.xml → 200, but it is a sitemap index with a single stale entry (belgashare.be, lastmod 2025-09-30) and no article URLs
/sitemap_index.xml → 404

Open question before implementation

Server-side scraping of the homepage is not viable as-is. Options, roughly in order of cost:

Find the client-side data endpoint — inspect the site's network requests in a browser to see what XHR/fetch call populates the article list. If there is a stable JSON endpoint, this becomes straightforward and more robust than scraping. Worth doing before anything else.
Headless browser rendering — would work, but adds a significant runtime dependency to the container and is a large change for one source.
Alternative Belga source — check whether belgashare.be (referenced in their own sitemap) or another Belga property exposes a usable feed.
Drop the Belga feed — if no practical route exists, deactivate it rather than leave it silently failing.

Recommend option 1 first; it determines whether this is a small parser addition or a much larger piece of work.

Separate issue worth noting

A feed whose URL 404s updates last_fetched_at and reports success. There is a warning in the log, but nothing surfaces in the UI and no notification fires. Given CheckFeedStalenessJob and the FEED_STALE notification already exist, a feed returning zero articles over repeated fetches arguably deserves the same treatment. That is a general robustness gap, not specific to Belga — suggest a separate ticket.

## Root cause found — the RSS feed is gone Diagnostics from production and against the live site. ### Feed config (prod) ``` rss | https://www.belganewsagency.eu/feed | active:1 | last_fetched:2026-08-01 06:52:01 ``` The job **is** running (fetched this morning), so this is not a queue or scheduler problem. ### The feed URL is dead ``` https://www.belganewsagency.eu/feed -> 308 redirect to /feed/ https://www.belganewsagency.eu/feed/ -> 404 (14 KB HTML error page) ``` `getArticlesFromRssFeed()` passes that HTML to `simplexml_load_string()`, which fails, logs "Failed to parse RSS feed XML", and returns an empty collection. The job then completes successfully and updates `last_fetched_at` — which is why this failed silently with no visible error. ### No alternative feed exists All returned 404: `/rss`, `/rss.xml`, `/feed.xml`, `/en/feed`, `/atom.xml`, `/index.xml`. The homepage (200 OK) declares **no** `<link rel="alternate">` RSS/Atom element. The site appears to have retired its public feed entirely. ## Switching to website scraping is harder than the VRT case Confirmed first that no Belga homepage parser exists in any form: - Only three Belga files in the repo: `BelgaArticleParser`, `BelgaArticlePageParser`, and `BelgaArticlePageParserTest` - `VrtHomepageParserAdapter` is the **only** class implementing `HomepageParserInterface` - `config/feed.php` has `belga.parsers` with `article` and `article_page` but no `homepage` key, and `type => 'rss'` ### The blocker: the site is client-side rendered The homepage is a **Next.js** app (27 `/_next/` references, `__NEXT_DATA__` present). Fetching it server-side yields: - 70 KB of HTML containing only **7 unique links**, none of them articles - `__NEXT_DATA__` payload is only ~2 KB and contains **zero article slugs** Article content is loaded client-side after page load. A regex-over-HTML approach like `VrtHomepageParser` will find nothing. Other discovery routes checked: - `/api/articles`, `/api/news` → 404 - `api.belganewsagency.eu` → DNS does not resolve - `/sitemap.xml` → 200, but it is a sitemap index with a **single stale entry** (`belgashare.be`, lastmod 2025-09-30) and no article URLs - `/sitemap_index.xml` → 404 ## Open question before implementation Server-side scraping of the homepage is **not viable as-is**. Options, roughly in order of cost: 1. **Find the client-side data endpoint** — inspect the site's network requests in a browser to see what XHR/fetch call populates the article list. If there is a stable JSON endpoint, this becomes straightforward and more robust than scraping. **Worth doing before anything else.** 2. **Headless browser rendering** — would work, but adds a significant runtime dependency to the container and is a large change for one source. 3. **Alternative Belga source** — check whether `belgashare.be` (referenced in their own sitemap) or another Belga property exposes a usable feed. 4. **Drop the Belga feed** — if no practical route exists, deactivate it rather than leave it silently failing. Recommend option 1 first; it determines whether this is a small parser addition or a much larger piece of work. ## Separate issue worth noting A feed whose URL 404s updates `last_fetched_at` and reports success. There is a warning in the log, but nothing surfaces in the UI and no notification fires. Given `CheckFeedStalenessJob` and the `FEED_STALE` notification already exist, a feed returning zero articles over repeated fetches arguably deserves the same treatment. That is a general robustness gap, not specific to Belga — suggest a separate ticket.

myrmidex changed title from ~~No articles being discovered from the Belga News feed~~ to Belga News feed produces no articles — RSS feed retired, discovery approach TBD

2026-08-01 11:22:35 +02:00

myrmidex added this to the v1.3.5 milestone 2026-08-01 11:24:17 +02:00

myrmidex commented

2026-08-01 11:27:08 +02:00

Author

Owner

Approach settled — option 1, public JSON API found

DevTools inspection found the endpoint the site uses to populate its article list.

The endpoint

https://capi.belga.press/belgapress/api/public/pressreleases
    ?offset=1&count=6&search=&start=&end=&newsroomId=70&language=EN

Verified working server-side — curl with no Origin, no Referer, no cookies, no auth returns 200 application/json. The CORS headers are for the browser's benefit; they do not gate server-to-server requests.

newsroomId=70 identifies the Belga News Agency newsroom (it matches newsroomData.id in the site's __NEXT_DATA__).

Response shape

{
  "data": [ { "id": 35285,
              "publishDate": "2026-08-01T06:24:19",
              "coverImageUrl": "...",
              "newsroom": {...},
              "content": { "en": { "title": "...", "lead": "...",
                                   "coverImageUrl": "...",
                                   "categories": [{"id":3,"name":"SPORTS"}] } } } ],
  "_links": { "next": "...offset=7...", "prev": "...", "self": "..." },
  "_meta": { "total": 14650 }
}

content is keyed by language code (en), which maps cleanly onto the existing $language constructor parameter in the parser adapters
Pagination via offset/count with _links.next provided; count can be raised above 6
_meta.total was 14650 at time of writing

Article URLs

The API returns no URL field. Article pages are reachable by ID:

https://www.belganewsagency.eu/press-releases/{id}/

Verified 200 for IDs 35285, 35282, 35271, 35269. Other patterns (/{id}, /pressreleases/{id}, belgashare.be/pressreleases/{id}) all 404.

So extractArticleUrls() builds URLs from data[].id rather than scraping hrefs.

The existing article parser still works

Article pages are server-rendered enough for BelgaArticlePageParser: og:title and og:description are present in the raw HTML and match the API values exactly. No changes needed to BelgaArticleParser / BelgaArticlePageParser.

Sample data returned (fetched 2026-08-01):

35285  2026-08-01T06:24:19  FIFA cancels plans for World Cup commercialisation...
35282  2026-07-31T16:59:26  Excluding Spain from Schengen is "in no way a solution"...
35281  2026-07-31T16:52:04  Belgium offers tankers and other vehicles for fire fighting...

Revised implementation plan

BelgaHomepageParser — calls the API, returns article URLs built from IDs. Unlike VrtHomepageParser this is JSON parsing, not regex over HTML, so it is more robust.
BelgaHomepageParserAdapter — implements HomepageParserInterface, follows VrtHomepageParserAdapter, takes $language and maps it to the language= query parameter.
config/feed.php — add belga.parsers.homepage, change belga.type from rss to website, update the en URL.
Update the prod feed row: type → website, and the URL to the homepage.
Tests against a saved JSON fixture, no live HTTP.

Open questions for implementation

count value — the site requests 6. defaults.max_articles_per_fetch in config/feed.php is 50. Worth confirming the API accepts a larger count rather than assuming.
Endpoint stability — this is an internal API for their front-end, not a documented public one. It could change without notice. Failure would be visible (HTTP error rather than silent empty parse), but worth noting as a maintenance risk in a way an RSS feed would not be.
start/end parameters — appear to accept date bounds; could be used to fetch only recent articles rather than paginating from the newest.

## Approach settled — option 1, public JSON API found DevTools inspection found the endpoint the site uses to populate its article list. ### The endpoint ``` https://capi.belga.press/belgapress/api/public/pressreleases ?offset=1&count=6&search=&start=&end=&newsroomId=70&language=EN ``` **Verified working server-side** — `curl` with no `Origin`, no `Referer`, no cookies, no auth returns `200 application/json`. The CORS headers are for the browser's benefit; they do not gate server-to-server requests. `newsroomId=70` identifies the Belga News Agency newsroom (it matches `newsroomData.id` in the site's `__NEXT_DATA__`). ### Response shape ```json { "data": [ { "id": 35285, "publishDate": "2026-08-01T06:24:19", "coverImageUrl": "...", "newsroom": {...}, "content": { "en": { "title": "...", "lead": "...", "coverImageUrl": "...", "categories": [{"id":3,"name":"SPORTS"}] } } } ], "_links": { "next": "...offset=7...", "prev": "...", "self": "..." }, "_meta": { "total": 14650 } } ``` - `content` is keyed by language code (`en`), which maps cleanly onto the existing `$language` constructor parameter in the parser adapters - Pagination via `offset`/`count` with `_links.next` provided; `count` can be raised above 6 - `_meta.total` was 14650 at time of writing ### Article URLs The API returns **no URL field**. Article pages are reachable by ID: ``` https://www.belganewsagency.eu/press-releases/{id}/ ``` Verified 200 for IDs 35285, 35282, 35271, 35269. Other patterns (`/{id}`, `/pressreleases/{id}`, `belgashare.be/pressreleases/{id}`) all 404. So `extractArticleUrls()` builds URLs from `data[].id` rather than scraping hrefs. ### The existing article parser still works Article pages are server-rendered enough for `BelgaArticlePageParser`: `og:title` and `og:description` are present in the raw HTML and match the API values exactly. **No changes needed to `BelgaArticleParser` / `BelgaArticlePageParser`.** Sample data returned (fetched 2026-08-01): ``` 35285 2026-08-01T06:24:19 FIFA cancels plans for World Cup commercialisation... 35282 2026-07-31T16:59:26 Excluding Spain from Schengen is "in no way a solution"... 35281 2026-07-31T16:52:04 Belgium offers tankers and other vehicles for fire fighting... ``` ## Revised implementation plan 1. `BelgaHomepageParser` — calls the API, returns article URLs built from IDs. Unlike `VrtHomepageParser` this is JSON parsing, not regex over HTML, so it is more robust. 2. `BelgaHomepageParserAdapter` — implements `HomepageParserInterface`, follows `VrtHomepageParserAdapter`, takes `$language` and maps it to the `language=` query parameter. 3. `config/feed.php` — add `belga.parsers.homepage`, change `belga.type` from `rss` to `website`, update the `en` URL. 4. Update the prod feed row: `type` → `website`, and the URL to the homepage. 5. Tests against a saved JSON fixture, no live HTTP. ## Open questions for implementation - **`count` value** — the site requests 6. `defaults.max_articles_per_fetch` in `config/feed.php` is 50. Worth confirming the API accepts a larger `count` rather than assuming. - **Endpoint stability** — this is an internal API for their front-end, not a documented public one. It could change without notice. Failure would be visible (HTTP error rather than silent empty parse), but worth noting as a maintenance risk in a way an RSS feed would not be. - **`start`/`end` parameters** — appear to accept date bounds; could be used to fetch only recent articles rather than paginating from the newest.

myrmidex referenced this issue

2026-08-01 11:28:31 +02:00

Detect and warn about feeds that fetch successfully but return no articles #116

myrmidex referenced this issue

2026-08-01 16:34:23 +02:00

Thumbnail URLs are not HTML-entity decoded, so custom_thumbnail fails #117

myrmidex referenced this issue

2026-08-01 16:35:52 +02:00

Add end-to-end tests covering the full feed-to-published-post pipeline #118

myrmidex referenced this issue

2026-08-02 10:12:34 +02:00

Concurrent publish attempts create duplicate Lemmy posts #123