Research user-definable parsing for non-RSS (website) feeds #143

New issue

Open

opened 2026-08-13 21:25:06 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-13 21:25:06 +02:00

Owner

Summary

Adding a website-scraped source today requires a code change and a deploy. This ticket is research only: work out how a user could define, adapt, or override parsing rules for a non-RSS feed themselves, and produce a recommendation plus follow-up implementation tickets.

No implementation in this ticket.

Problem

RSS feeds are effectively declarative — point at a URL and the generic XML path in ArticleFetcher::getArticlesFromRssFeed() handles it. Website feeds are the opposite: every source needs hand-written PHP.

Adding one source currently means:

A homepage parser class (extract article URLs from the index page)
An article-page parser class (extract title, description, full article, thumbnail)
An adapter implementing HomepageParserInterface / ArticleParserInterface
A config/feed.php entry wiring the class names in
A release and deploy

So the app can only ever scrape sources its maintainer has written code for. A self-hosted user who wants their local news site has no path at all.

Current architecture (as of this ticket)

Resolution is config-driven but the parsers themselves are hardcoded classes:

HomepageParserFactory::getParserForFeed() and ArticleParserFactory::getParserForFeed() read config("feed.providers.{$feed->provider}.parsers.*") and instantiate the mapped class.
ArticleParserFactory::getParser() (used by ArticleFetcher::fetchArticleData()) instead loops a hardcoded $parsers array and picks the first whose canParse($url) matches — a second, separate resolution path that ignores the feed's provider entirely. Any design has to account for both.

The three existing providers are not shaped alike, which constrains the format:

VRT — VrtHomepageParser is a single preg_match_all over raw HTML with the language interpolated into the pattern. Language-parameterised.
Belga — BelgaHomepageParser is not HTML at all. It decodes a JSON API response, plucks data[].id, and templates those into article URLs via a sprintf pattern.
Guardian — RSS, so no homepage parser at all.

VrtArticlePageParser is already a chain of prioritised fallbacks (og:title → h1 → <title>), which is a strong hint that the natural user-facing format is an ordered list of extraction rules per field, first match wins.

Relevant existing facts:

feeds.settings is a json column, in $fillable, cast to array, and currently unused — likely storage for per-feed rules with no migration needed.
feeds.provider is NOT NULL ($table->string('provider')), and CreateFeedAction derives the URL from config("feed.providers.{$provider}.languages.{$langCode}.url"). A user-defined feed has no provider config entry, so this path needs rethinking.
No DOM or CSS-selector library is installed — composer.json has no symfony/dom-crawler, symfony/css-selector, or equivalent. All current parsing is regex over raw strings.

Research questions

Rule format

CSS selectors, XPath, or regex? Selectors are far more approachable for users; regex is what exists today.
How are JSON sources (Belga) expressed in the same format — a separate rule type, or a JSON-path variant?
How is the URL-templating case (Belga's id → article URL) expressed declaratively?
How is VRT's language parameterisation carried over?

Dependency

Does this justify adding symfony/dom-crawler + symfony/css-selector? Weigh against the project's low-scaffolding preference.

Layering

Can a user override part of a shipped provider's parsing (e.g. just the thumbnail rule) while inheriting the rest? Config defaults + per-feed settings overrides, or full replacement only?
Does a user-defined feed still need a provider value, given the NOT NULL column?

Security — user-supplied extraction rules are executed server-side:

Catastrophic backtracking / ReDoS if regex is user-supplied.
SSRF: a user-defined feed points HttpFetcher at an arbitrary URL. Assess what the current fetcher already does or does not guard against.
Whether this is a real concern given the app's trust model (self-hosted, admin-only) — document the assumption either way rather than leaving it implicit.

Raw JSON editor vs guided field-by-field builder.
A "test these rules against a live URL and show me what was extracted" preview is probably the difference between usable and unusable. How hard is that to build?
What does a rule that matches nothing look like to the user? Ties into #116 "Detect and warn about feeds that fetch successfully but return no articles" (closed).

Migration

Can VRT and Belga be expressed in the new format, or do they stay as code? Being unable to express the two existing website sources would be a strong signal the format is too weak.

Non-goals

No implementation — output is a recommendation and follow-up tickets.
No JS execution or headless browsing. See #139 "Add Reuters as a feed provider" for why that is its own can of worms.
No change to the RSS path.

Relationship to other tickets

#34 "Custom RSS feeds" — complementary, not overlapping. #34 is explicitly scoped to "RSS feeds only (no website scraping for custom feeds)". This ticket is the website-scraping half. The two share a likely prerequisite: a system-vs-custom feed distinction and a feed-creation UI that does not depend on a config/feed.php provider entry. Worth deciding whether that shared groundwork lands in #34 or in a third ticket.
#139 "Add Reuters as a feed provider" — already notes it may not need a dedicated provider if a generic path exists.
#124 "Investigate multi-language support for the Belga feed" — the language-parameterisation question overlaps directly.

Deliverable

Written recommendation on the ticket: rule format, storage, dependency decision, layering model
Assessment of whether VRT and Belga can be expressed in the proposed format
Security assessment (ReDoS, SSRF) with the trust model stated explicitly
Proposed UI approach, including the test/preview affordance
Follow-up implementation tickets created, sized and ordered

## Summary Adding a website-scraped source today requires a code change and a deploy. This ticket is **research only**: work out how a user could define, adapt, or override parsing rules for a non-RSS feed themselves, and produce a recommendation plus follow-up implementation tickets. No implementation in this ticket. ## Problem RSS feeds are effectively declarative — point at a URL and the generic XML path in `ArticleFetcher::getArticlesFromRssFeed()` handles it. Website feeds are the opposite: every source needs hand-written PHP. Adding one source currently means: 1. A homepage parser class (extract article URLs from the index page) 2. An article-page parser class (extract title, description, full article, thumbnail) 3. An adapter implementing `HomepageParserInterface` / `ArticleParserInterface` 4. A `config/feed.php` entry wiring the class names in 5. A release and deploy So the app can only ever scrape sources its maintainer has written code for. A self-hosted user who wants their local news site has no path at all. ## Current architecture (as of this ticket) Resolution is config-driven but the parsers themselves are hardcoded classes: - `HomepageParserFactory::getParserForFeed()` and `ArticleParserFactory::getParserForFeed()` read `config("feed.providers.{$feed->provider}.parsers.*")` and instantiate the mapped class. - `ArticleParserFactory::getParser()` (used by `ArticleFetcher::fetchArticleData()`) instead loops a hardcoded `$parsers` array and picks the first whose `canParse($url)` matches — a **second, separate resolution path** that ignores the feed's provider entirely. Any design has to account for both. The three existing providers are not shaped alike, which constrains the format: - **VRT** — `VrtHomepageParser` is a single `preg_match_all` over raw HTML with the language interpolated into the pattern. Language-parameterised. - **Belga** — `BelgaHomepageParser` is not HTML at all. It decodes a JSON API response, plucks `data[].id`, and templates those into article URLs via a `sprintf` pattern. - **Guardian** — RSS, so no homepage parser at all. `VrtArticlePageParser` is already a chain of prioritised fallbacks (`og:title` → `h1` → `<title>`), which is a strong hint that the natural user-facing format is an **ordered list of extraction rules per field**, first match wins. Relevant existing facts: - `feeds.settings` is a `json` column, in `$fillable`, cast to `array`, and currently unused — likely storage for per-feed rules with no migration needed. - `feeds.provider` is `NOT NULL` (`$table->string('provider')`), and `CreateFeedAction` derives the URL from `config("feed.providers.{$provider}.languages.{$langCode}.url")`. A user-defined feed has no provider config entry, so this path needs rethinking. - **No DOM or CSS-selector library is installed** — `composer.json` has no `symfony/dom-crawler`, `symfony/css-selector`, or equivalent. All current parsing is regex over raw strings. ## Research questions **Rule format** - CSS selectors, XPath, or regex? Selectors are far more approachable for users; regex is what exists today. - How are JSON sources (Belga) expressed in the same format — a separate rule type, or a JSON-path variant? - How is the URL-templating case (Belga's `id` → article URL) expressed declaratively? - How is VRT's language parameterisation carried over? **Dependency** - Does this justify adding `symfony/dom-crawler` + `symfony/css-selector`? Weigh against the project's low-scaffolding preference. **Layering** - Can a user override *part* of a shipped provider's parsing (e.g. just the thumbnail rule) while inheriting the rest? Config defaults + per-feed `settings` overrides, or full replacement only? - Does a user-defined feed still need a `provider` value, given the NOT NULL column? **Security** — user-supplied extraction rules are executed server-side: - Catastrophic backtracking / ReDoS if regex is user-supplied. - SSRF: a user-defined feed points `HttpFetcher` at an arbitrary URL. Assess what the current fetcher already does or does not guard against. - Whether this is a real concern given the app's trust model (self-hosted, admin-only) — document the assumption either way rather than leaving it implicit. **UX** - Raw JSON editor vs guided field-by-field builder. - A "test these rules against a live URL and show me what was extracted" preview is probably the difference between usable and unusable. How hard is that to build? - What does a rule that matches nothing look like to the user? Ties into #116 "Detect and warn about feeds that fetch successfully but return no articles" (closed). **Migration** - Can VRT and Belga be expressed in the new format, or do they stay as code? Being unable to express the two existing website sources would be a strong signal the format is too weak. ## Non-goals - No implementation — output is a recommendation and follow-up tickets. - No JS execution or headless browsing. See #139 "Add Reuters as a feed provider" for why that is its own can of worms. - No change to the RSS path. ## Relationship to other tickets - **#34 "Custom RSS feeds"** — complementary, not overlapping. #34 is explicitly scoped to *"RSS feeds only (no website scraping for custom feeds)"*. This ticket is the website-scraping half. The two share a likely prerequisite: a system-vs-custom feed distinction and a feed-creation UI that does not depend on a `config/feed.php` provider entry. Worth deciding whether that shared groundwork lands in #34 or in a third ticket. - **#139 "Add Reuters as a feed provider"** — already notes it may not need a dedicated provider if a generic path exists. - **#124 "Investigate multi-language support for the Belga feed"** — the language-parameterisation question overlaps directly. ## Deliverable - [ ] Written recommendation on the ticket: rule format, storage, dependency decision, layering model - [ ] Assessment of whether VRT and Belga can be expressed in the proposed format - [ ] Security assessment (ReDoS, SSRF) with the trust model stated explicitly - [ ] Proposed UI approach, including the test/preview affordance - [ ] Follow-up implementation tickets created, sized and ordered