Strip the publisher suffix from article titles before publishing #155

New issue

Open

opened 2026-08-15 11:57:52 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-15 11:57:52 +02:00

Owner

Problem

Article titles carry the publisher's site name as a trailing suffix, and it is
published to Lemmy verbatim:

Israëlische kolonisten belegeren al dagenlang huizen van Palestijnse gezinnen:
water en stroom afgesneden | VRT NWS Nieuws

The | VRT NWS Nieuws is noise. It repeats on every post from that feed, eats
into the title length, and tells a reader nothing they cannot see from the link.

Where it comes from

VrtArticlePageParser::extractTitle() reads, in order:

<meta property="og:title">
<h1>
<title>

Publishers routinely append their site name to both og:title and <title> for
search and social previews. The <h1> usually carries the bare headline, but it
is only reached when the meta tag is absent.

BelgaArticlePageParser and GuardianArticlePageParser have the same shape, so
this is not VRT-specific — The Guardian appends | The Guardian by the same
convention.

Approach

Strip a trailing separator-plus-site-name from the extracted title. The
separator is usually | but - and — are also common.

Two ways to decide what to strip:

Per-provider suffix, configured. config/feed.php already holds per-provider
configuration, so each provider could declare the suffix to remove. Explicit and
predictable, but needs updating when a publisher changes their branding.

Generic trailing-segment rule. Strip the last |-delimited segment when it
matches the feed or provider name. Needs no configuration and handles new
providers, but risks eating a real title that happens to end in a pipe.

Leaning toward the configured suffix: there are three providers, the values
rarely change, and a wrong guess silently mangles headlines.

Worth deciding

Where the stripping happens. In each parser's extractTitle(), or once in
a shared place after extraction. A shared helper avoids three copies of the
same logic and gives one place to test.
Whether existing articles are cleaned up. Titles already stored keep their
suffix. A backfill is possible but articles are pruned after 30 days
(CleanupArticlesJob), so doing nothing resolves it within a month.
Whether the suffix should be stripped at extraction or at publish. Storing
the clean title is simpler; storing the original and cleaning at publish keeps
the source text intact. Extraction seems right — nothing else reads the
suffix.

Acceptance criteria

A VRT article publishes without | VRT NWS Nieuws
The same holds for Belga and Guardian articles with their own suffixes
A title that does not carry a suffix is unchanged
A title containing a pipe as part of the actual headline is not truncated
Tests cover each provider, using the existing HTML fixtures

## Problem Article titles carry the publisher's site name as a trailing suffix, and it is published to Lemmy verbatim: ``` Israëlische kolonisten belegeren al dagenlang huizen van Palestijnse gezinnen: water en stroom afgesneden | VRT NWS Nieuws ``` The ` | VRT NWS Nieuws` is noise. It repeats on every post from that feed, eats into the title length, and tells a reader nothing they cannot see from the link. ## Where it comes from `VrtArticlePageParser::extractTitle()` reads, in order: 1. `<meta property="og:title">` 2. `<h1>` 3. `<title>` Publishers routinely append their site name to both `og:title` and `<title>` for search and social previews. The `<h1>` usually carries the bare headline, but it is only reached when the meta tag is absent. `BelgaArticlePageParser` and `GuardianArticlePageParser` have the same shape, so this is not VRT-specific — The Guardian appends ` | The Guardian` by the same convention. ## Approach Strip a trailing separator-plus-site-name from the extracted title. The separator is usually ` | ` but ` - ` and ` — ` are also common. Two ways to decide what to strip: **Per-provider suffix, configured.** `config/feed.php` already holds per-provider configuration, so each provider could declare the suffix to remove. Explicit and predictable, but needs updating when a publisher changes their branding. **Generic trailing-segment rule.** Strip the last ` | `-delimited segment when it matches the feed or provider name. Needs no configuration and handles new providers, but risks eating a real title that happens to end in a pipe. Leaning toward the configured suffix: there are three providers, the values rarely change, and a wrong guess silently mangles headlines. ## Worth deciding - **Where the stripping happens.** In each parser's `extractTitle()`, or once in a shared place after extraction. A shared helper avoids three copies of the same logic and gives one place to test. - **Whether existing articles are cleaned up.** Titles already stored keep their suffix. A backfill is possible but articles are pruned after 30 days (`CleanupArticlesJob`), so doing nothing resolves it within a month. - **Whether the suffix should be stripped at extraction or at publish.** Storing the clean title is simpler; storing the original and cleaning at publish keeps the source text intact. Extraction seems right — nothing else reads the suffix. ## Acceptance criteria - [ ] A VRT article publishes without ` | VRT NWS Nieuws` - [ ] The same holds for Belga and Guardian articles with their own suffixes - [ ] A title that does not carry a suffix is unchanged - [ ] A title containing a pipe as part of the actual headline is not truncated - [ ] Tests cover each provider, using the existing HTML fixtures