Add Reuters as a feed provider #139

New issue

Open

opened 2026-08-13 20:25:09 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-13 20:25:09 +02:00

Owner

Summary

Add reuters as a feed provider so Reuters articles can be routed and published like VRT, Belga and Guardian content.

Blocked — reuters.com sits behind bot protection

Investigated 2026-08-13. The obvious approach does not work. Every candidate feed URL returns HTTP 401:

URL	Result
`https://www.reuters.com/rssfeed/worldNews`	401
`https://www.reuters.com/world/rss`	401
`https://www.reuters.com/arc/outboundfeeds/rss/?outputType=xml`	404
`https://feeds.reuters.com/reuters/worldNews`	does not resolve
`https://www.reuters.com/` (homepage)	401

The 401 body is a DataDome challenge, not an auth prompt:

<p id="cmsg">Please enable JS and disable any ad blocker</p>
<script data-cfasync="false">var dd={'rt':'c','cid':'AHrlqAAAAAMAdCi8bVOdcjAA1duXZQ==', ...

A browser User-Agent does not get past it. This blocks the whole site to non-browser clients, so HttpFetcher::fetchHtml() cannot reach Reuters content as things stand.

Options, none free

Licensed Reuters API — Reuters sells content access. Correct and reliable; costs money and needs credential handling.
Third-party aggregator — some services republish Reuters headlines as RSS. Cheaper, but adds a dependency on someone else's uptime and terms, and content is usually truncated.
Headless-browser fetching — would defeat the challenge but means running a browser in the publish path. Heavy for a self-hosted app, and adversarial to a site that has explicitly opted out.

Option 3 is worth thinking twice about: DataDome is an explicit signal that Reuters does not want automated fetching. Working around it is a licensing/ToS question, not just a technical one.

Relationship to #34 "Custom RSS feeds"

If Reuters content turns out to be reachable through a third-party RSS URL, #34 may cover this without a dedicated provider — a user could simply paste that URL in. Worth resolving #34 first and re-evaluating whether this ticket still needs its own parser pair.

If it does go ahead

Follow the #37 "Add The Guardian as an RSS feed" precedent — Guardian is the closest match, being type: 'rss' and needing no homepage scraper:

config/feed.php — a reuters entry with type, is_active, languages, and a parsers map
app/Services/Parsers/ReutersArticleParser.php and ReutersArticlePageParser.php
extractThumbnail() should return a sized image rather than a full-resolution og:image — see #138 "Thumbnails too large"

Acceptance criteria

A content source is agreed that does not require defeating bot protection
Reuters provider registered in config/feed.php
Article and article-page parsers implemented with tests against fixtures
A Reuters feed can be created through the UI and produces articles

## Summary Add `reuters` as a feed provider so Reuters articles can be routed and published like VRT, Belga and Guardian content. ## Blocked — reuters.com sits behind bot protection Investigated 2026-08-13. **The obvious approach does not work.** Every candidate feed URL returns HTTP 401: | URL | Result | |---|---| | `https://www.reuters.com/rssfeed/worldNews` | 401 | | `https://www.reuters.com/world/rss` | 401 | | `https://www.reuters.com/arc/outboundfeeds/rss/?outputType=xml` | 404 | | `https://feeds.reuters.com/reuters/worldNews` | does not resolve | | `https://www.reuters.com/` (homepage) | 401 | The 401 body is a **DataDome challenge**, not an auth prompt: ```html <p id="cmsg">Please enable JS and disable any ad blocker</p> <script data-cfasync="false">var dd={'rt':'c','cid':'AHrlqAAAAAMAdCi8bVOdcjAA1duXZQ==', ... ``` A browser User-Agent does not get past it. This blocks the whole site to non-browser clients, so `HttpFetcher::fetchHtml()` cannot reach Reuters content as things stand. ## Options, none free 1. **Licensed Reuters API** — Reuters sells content access. Correct and reliable; costs money and needs credential handling. 2. **Third-party aggregator** — some services republish Reuters headlines as RSS. Cheaper, but adds a dependency on someone else's uptime and terms, and content is usually truncated. 3. **Headless-browser fetching** — would defeat the challenge but means running a browser in the publish path. Heavy for a self-hosted app, and adversarial to a site that has explicitly opted out. Option 3 is worth thinking twice about: DataDome is an explicit signal that Reuters does not want automated fetching. Working around it is a licensing/ToS question, not just a technical one. ## Relationship to #34 "Custom RSS feeds" If Reuters content turns out to be reachable through a third-party RSS URL, #34 may cover this without a dedicated provider — a user could simply paste that URL in. **Worth resolving #34 first** and re-evaluating whether this ticket still needs its own parser pair. ## If it does go ahead Follow the #37 "Add The Guardian as an RSS feed" precedent — Guardian is the closest match, being `type: 'rss'` and needing no homepage scraper: - `config/feed.php` — a `reuters` entry with `type`, `is_active`, `languages`, and a `parsers` map - `app/Services/Parsers/ReutersArticleParser.php` and `ReutersArticlePageParser.php` - `extractThumbnail()` should return a **sized** image rather than a full-resolution `og:image` — see #138 "Thumbnails too large" ## Acceptance criteria - [ ] A content source is agreed that does not require defeating bot protection - [ ] Reuters provider registered in `config/feed.php` - [ ] Article and article-page parsers implemented with tests against fixtures - [ ] A Reuters feed can be created through the UI and produces articles