Custom RSS feeds #34

New issue

Open

opened 2025-07-09 23:34:05 +02:00 by myrmidex · 1 comment

myrmidex commented

2025-07-09 23:34:05 +02:00

(Migrated from codeberg.org)

Summary

Allow users to add custom RSS feeds beyond the pre-configured providers (VRT, Belga, Guardian).

Context

Currently feeds are tied to a provider config key (vrt, belga, guardian) which maps to specific parser classes. Users should be able to add any RSS feed URL without needing a custom parser — the generic RSS parsing infrastructure from #37 handles this.

Scope

RSS feeds only (no website scraping for custom feeds)
Two "system feeds" shipped via seeder (e.g. VRT, Guardian) that can't be deleted
User-added feeds are "custom" type — fully manageable (create, edit, delete)

Tasks

Add is_system flag (or feed_type column: system/custom) to feeds table
Migration to mark existing seeded feeds as system feeds
Seeder updates to set system flag on default feeds
Prevent deletion of system feeds (API + UI)
UI for adding custom RSS feeds: URL input, name, language selection
Validate RSS feed URL on creation (attempt fetch, confirm valid XML/RSS)
Custom feeds use generic RSS parser (no provider-specific parser needed)
Update feed CRUD endpoints to handle custom feeds
Tests for custom feed creation, validation, and deletion protection

## Summary Allow users to add custom RSS feeds beyond the pre-configured providers (VRT, Belga, Guardian). ## Context Currently feeds are tied to a `provider` config key (vrt, belga, guardian) which maps to specific parser classes. Users should be able to add any RSS feed URL without needing a custom parser — the generic RSS parsing infrastructure from #37 handles this. ## Scope - RSS feeds only (no website scraping for custom feeds) - Two "system feeds" shipped via seeder (e.g. VRT, Guardian) that can't be deleted - User-added feeds are "custom" type — fully manageable (create, edit, delete) ## Tasks - [ ] Add `is_system` flag (or `feed_type` column: system/custom) to feeds table - [ ] Migration to mark existing seeded feeds as system feeds - [ ] Seeder updates to set system flag on default feeds - [ ] Prevent deletion of system feeds (API + UI) - [ ] UI for adding custom RSS feeds: URL input, name, language selection - [ ] Validate RSS feed URL on creation (attempt fetch, confirm valid XML/RSS) - [ ] Custom feeds use generic RSS parser (no provider-specific parser needed) - [ ] Update feed CRUD endpoints to handle custom feeds - [ ] Tests for custom feed creation, validation, and deletion protection

myrmidex changed title from ~~Custom feeds~~ to Custom RSS feeds

2026-03-08 17:28:05 +01:00

myrmidex referenced this issue

2026-08-13 20:25:09 +02:00

Add Reuters as a feed provider #139

myrmidex referenced this issue

2026-08-13 21:25:06 +02:00

Research user-definable parsing for non-RSS (website) feeds #143

myrmidex commented

2026-08-15 12:15:03 +02:00

Owner

RSS feeds should not use page parsers at all

Decided 2026-08-15. This changes the ticket's scope beyond the task list above.

Current behaviour

FetchRssArticlesAction is already generic: it fetches any $feed->url, parses
standard RSS, and harvests <link> from each <item>. Nothing about it is
provider-specific, so the premise of this ticket holds.

What is not generic is what happens next. SaveArticleAction stores the link,
and the title, description and thumbnail are filled in later by
FetchArticleDataAction, which dispatches through ArticleParserFactory on
$feed->provider to a per-site page parser. For a custom feed there is no such
parser and getParser() throws No parser found for URL.

Decision

RSS feeds use the RSS item's own data. Title, and the brief description where
one is present, are enough to make a post. No page fetch, no page parser.

This applies to all RSS feeds, not only custom ones, which means The
Guardian stops going through GuardianArticlePageParser and starts using its
RSS content directly.

Consequences to handle

Thumbnails. These are currently extracted by scraping the article page.
RSS sometimes carries one in <enclosure> or <media:thumbnail>, but not
reliably. RSS-sourced posts may have no image where they previously did.
Decide whether to read those elements when present or accept no thumbnail.
Description length. RSS <description> is often a truncated teaser rather
than the full text. Posts from RSS feeds will be shorter than the scraped
equivalent.
GuardianArticlePageParser becomes dead code once Guardian stops using
it, along with its fixtures and tests. Worth removing in the same change
rather than leaving it to rot.
Website feeds are unaffected. VRT and Belga scrape because they have no
usable feed; they keep their parsers.

Upside

Fewer HTTP requests per article, and nothing to break when a publisher changes
their markup. The failure mode that made #115 invisible for weeks only exists
for scraped sources.

## RSS feeds should not use page parsers at all Decided 2026-08-15. This changes the ticket's scope beyond the task list above. ### Current behaviour `FetchRssArticlesAction` is already generic: it fetches any `$feed->url`, parses standard RSS, and harvests `<link>` from each `<item>`. Nothing about it is provider-specific, so the premise of this ticket holds. What is not generic is what happens next. `SaveArticleAction` stores the link, and the title, description and thumbnail are filled in later by `FetchArticleDataAction`, which dispatches through `ArticleParserFactory` on `$feed->provider` to a per-site page parser. For a custom feed there is no such parser and `getParser()` throws `No parser found for URL`. ### Decision RSS feeds use the RSS item's own data. Title, and the brief description where one is present, are enough to make a post. No page fetch, no page parser. This applies to **all** RSS feeds, not only custom ones, which means The Guardian stops going through `GuardianArticlePageParser` and starts using its RSS content directly. ### Consequences to handle - **Thumbnails.** These are currently extracted by scraping the article page. RSS sometimes carries one in `<enclosure>` or `<media:thumbnail>`, but not reliably. RSS-sourced posts may have no image where they previously did. Decide whether to read those elements when present or accept no thumbnail. - **Description length.** RSS `<description>` is often a truncated teaser rather than the full text. Posts from RSS feeds will be shorter than the scraped equivalent. - **`GuardianArticlePageParser` becomes dead code** once Guardian stops using it, along with its fixtures and tests. Worth removing in the same change rather than leaving it to rot. - **Website feeds are unaffected.** VRT and Belga scrape because they have no usable feed; they keep their parsers. ### Upside Fewer HTTP requests per article, and nothing to break when a publisher changes their markup. The failure mode that made #115 invisible for weeks only exists for scraped sources.