lvl0/fedi-feed-router

Fork 0

Title rewriting rules per feed and route #155

New issue

Open

opened 2026-08-15 11:57:52 +02:00 by myrmidex · 1 comment

myrmidex commented

2026-08-15 11:57:52 +02:00

Owner

Problem

Article titles carry the publisher's site name, published to Lemmy verbatim:

Israëlische kolonisten belegeren al dagenlang huizen van Palestijnse gezinnen: water en stroom afgesneden | VRT NWS Nieuws

The | VRT NWS Nieuws repeats on every post, eats title length, and tells the reader nothing. BelgaArticlePageParser and GuardianArticlePageParser have the same shape — publishers append their site name to og:title and <title> by convention, and that is what the parsers read.

A fixed per-provider suffix string was considered and rejected: it only handles suffixes, only handles removal, and only the maintainer can change it. Users want prefixes gone too, and sometimes want a suffix replaced rather than dropped.

Approach

User-defined regex rules that rewrite the title at publish time, with a preview against real titles before saving.

Model

A title_rules table with a polymorphic owner (Feed or Route), one rule per owner, enforced by a unique index on (owner_type, owner_id).

Column	Notes
`owner_type`, `owner_id`	morphTo — `Feed` or `Route`
`pattern`	bare regex, no delimiters
`ignore_case`	boolean → the `i` flag
`replacement`	may be empty (pure strip); supports `$1` backreferences

Plus routes.apply_title_rule, boolean, default true.

This requires routes.id (#170) — routes currently has a composite primary key and cannot be a polymorphic owner.

Precedence, resolved at publish

route.apply_title_rule === false → title unchanged, whatever the feed says
Route has a rule → route rule wins
Feed has a rule → feed rule
Neither → unchanged

Route replaces feed, never merges. One place to look when a title comes out wrong.

Applied at publish, never at extraction

articles.title stays pristine. Two routes from one feed must be able to produce different titles from one stored article, and editing a rule must change future publishes without a backfill. A bad rule is recoverable because the original survives.

PublishRouteArticleAction::resolvePublishData() currently takes an Article; it needs the RouteArticle (or the route) to resolve precedence. That signature change is part of this work.

Regex handling

Delimiters are fixed / — shown in the UI as non-editable affixes, so a user cannot break out or pass a modifier
u (unicode) is always on and never exposed. Feeds are Dutch and French; without it a . can match half of é and emit corrupt UTF-8
One user-facing toggle: Ignore case
m and s are deliberately omitted — titles are single-line

Failure handling

preg_replace returns null on failure (bad pattern, or pcre.backtrack_limit exceeded). Backtracking depends on the input, so a rule can validate at save and still fail later on one pathological title.

On null: publish the original title and record it visibly — never let the title become null. Silent fallback is not acceptable; see #164.

UI

On the Feeds page and the Routes page:

Pattern field with fixed / affixes, Ignore case checkbox, Replacement field
Preview button — deliberately not live-on-keystroke. Each press runs an untrusted regex server-side; a button makes that explicit and bounded
Preview shows before → after for the last 10 titles from that feed
A row whose preg_replace returns null is marked as an error
A pattern that errors on any sample title cannot be saved
A pattern that simply does not match is a no-op, not an error — a suffix rule will not match a title lacking the suffix, and the UI must not present that as broken

Validation runs on save regardless of whether Preview was pressed — the button is a convenience, not a gate.

Commit sequence

Migration, model, factory (title_rules + routes.apply_title_rule)
Precedence resolution + tests
Publish-path wiring, failure handling + tests
Feed and route UI with preview and save-time validation

Open question

pcre.backtrack_limit and pcre.jit in the container are unverified — they bound the worst case for an untrusted pattern. Check before relying on them.

Acceptance criteria

A VRT article publishes without | VRT NWS Nieuws given a feed rule
A route rule overrides its feed's rule
apply_title_rule = false publishes the raw title despite a feed rule
No rule anywhere → title unchanged
A replacement with $1 works; an empty replacement strips
A title with a pipe in the real headline is not truncated by a non-matching rule
preg_replace returning null publishes the original title and is recorded
A pattern failing on a sample title cannot be saved
A non-matching pattern saves cleanly and is not shown as an error
Accented characters survive — regression guard for the missing-u case
articles.title is never modified

#170 — routes.id, required for the polymorphic owner

## Problem Article titles carry the publisher's site name, published to Lemmy verbatim: ``` Israëlische kolonisten belegeren al dagenlang huizen van Palestijnse gezinnen: water en stroom afgesneden | VRT NWS Nieuws ``` The ` | VRT NWS Nieuws` repeats on every post, eats title length, and tells the reader nothing. `BelgaArticlePageParser` and `GuardianArticlePageParser` have the same shape — publishers append their site name to `og:title` and `<title>` by convention, and that is what the parsers read. A fixed per-provider suffix string was considered and rejected: it only handles suffixes, only handles removal, and only the maintainer can change it. Users want prefixes gone too, and sometimes want a suffix **replaced** rather than dropped. ## Approach User-defined regex rules that rewrite the title at publish time, with a preview against real titles before saving. ### Model A `title_rules` table with a **polymorphic owner** (`Feed` or `Route`), one rule per owner, enforced by a unique index on `(owner_type, owner_id)`. | Column | Notes | |---|---| | `owner_type`, `owner_id` | morphTo — `Feed` or `Route` | | `pattern` | bare regex, no delimiters | | `ignore_case` | boolean → the `i` flag | | `replacement` | may be empty (pure strip); supports `$1` backreferences | Plus `routes.apply_title_rule`, boolean, default `true`. **This requires `routes.id` (#170)** — `routes` currently has a composite primary key and cannot be a polymorphic owner. ### Precedence, resolved at publish 1. `route.apply_title_rule === false` → title unchanged, whatever the feed says 2. Route has a rule → route rule wins 3. Feed has a rule → feed rule 4. Neither → unchanged Route **replaces** feed, never merges. One place to look when a title comes out wrong. ### Applied at publish, never at extraction `articles.title` stays pristine. Two routes from one feed must be able to produce different titles from one stored article, and editing a rule must change future publishes without a backfill. A bad rule is recoverable because the original survives. `PublishRouteArticleAction::resolvePublishData()` currently takes an `Article`; it needs the `RouteArticle` (or the route) to resolve precedence. **That signature change is part of this work.** ### Regex handling - Delimiters are fixed `/` — shown in the UI as non-editable affixes, so a user cannot break out or pass a modifier - **`u` (unicode) is always on and never exposed.** Feeds are Dutch and French; without it a `.` can match half of `é` and emit corrupt UTF-8 - One user-facing toggle: **Ignore case** - `m` and `s` are deliberately omitted — titles are single-line ### Failure handling `preg_replace` returns `null` on failure (bad pattern, or `pcre.backtrack_limit` exceeded). Backtracking depends on the **input**, so a rule can validate at save and still fail later on one pathological title. On `null`: **publish the original title and record it visibly** — never let the title become null. Silent fallback is not acceptable; see #164. ### UI On the Feeds page and the Routes page: - Pattern field with fixed `/` affixes, Ignore case checkbox, Replacement field - **Preview button** — deliberately *not* live-on-keystroke. Each press runs an untrusted regex server-side; a button makes that explicit and bounded - Preview shows before → after for the **last 10 titles from that feed** - A row whose `preg_replace` returns `null` is marked as an error - **A pattern that errors on any sample title cannot be saved** - A pattern that simply does not match is a **no-op, not an error** — a suffix rule will not match a title lacking the suffix, and the UI must not present that as broken Validation runs on **save** regardless of whether Preview was pressed — the button is a convenience, not a gate. ## Commit sequence 1. Migration, model, factory (`title_rules` + `routes.apply_title_rule`) 2. Precedence resolution + tests 3. Publish-path wiring, failure handling + tests 4. Feed and route UI with preview and save-time validation ## Open question `pcre.backtrack_limit` and `pcre.jit` in the container are **unverified** — they bound the worst case for an untrusted pattern. Check before relying on them. ## Acceptance criteria - [ ] A VRT article publishes without ` | VRT NWS Nieuws` given a feed rule - [ ] A route rule overrides its feed's rule - [ ] `apply_title_rule = false` publishes the raw title despite a feed rule - [ ] No rule anywhere → title unchanged - [ ] A replacement with `$1` works; an empty replacement strips - [ ] A title with a pipe in the real headline is not truncated by a non-matching rule - [ ] `preg_replace` returning `null` publishes the original title and is recorded - [ ] A pattern failing on a sample title cannot be saved - [ ] A non-matching pattern saves cleanly and is not shown as an error - [ ] Accented characters survive — regression guard for the missing-`u` case - [ ] `articles.title` is never modified ## Related - #170 — `routes.id`, required for the polymorphic owner

myrmidex added this to the v1.5.0 milestone 2026-08-15 11:57:52 +02:00

myrmidex added the

enhancement

label 2026-08-15 11:57:52 +02:00

myrmidex modified the milestone from v1.5.0 to v1.4.3

2026-08-29 18:22:36 +02:00

myrmidex modified the milestone from v1.4.3 to v1.5.0

2026-09-26 14:44:09 +02:00

myrmidex referenced this issue

2026-09-26 16:07:32 +02:00

Give routes a surrogate primary key #170

myrmidex changed title from ~~Strip the publisher suffix from article titles before publishing~~ to Title rewriting rules per feed and route

2026-09-26 17:11:33 +02:00

myrmidex added a new dependency 2026-09-26 17:11:40 +02:00

#170 Give routes a surrogate primary key

myrmidex commented

2026-09-26 17:17:30 +02:00

Author

Owner

Open question resolved: PCRE limits measured

Measured in the dev container (container_tinker), not assumed:

pcre.backtrack_limit  1000000   (PHP default)
pcre.recursion_limit  100000    (PHP default)
pcre.jit              1         (enabled)
PCRE version          10.44 2024-06-07

Cost of a pathological pattern is bounded and small. /(a|a)+$/u against inputs of 20–100 characters hits the limit in ~2.6ms, flat regardless of input length — the backtrack limit is a hard counter, so cost is capped by the limit, not by input size. Ten sample titles in the preview is ~28ms worst case.

Note the textbook catastrophic pattern /(a+)+b/u does not blow up here at all (0.04ms, no error) — PCRE2's JIT and auto-possessification optimise it away. Constructing a genuinely pathological case took deliberate effort, which further bounds the practical risk.

Failure is cleanly detectable:

preg_replace returns null
preg_last_error() returns PREG_BACKTRACK_LIMIT_ERROR (2), distinguishable from a malformed pattern
preg_last_error_msg() gives "Backtrack limit exhausted"

So no timeout guard, no limit tuning, and no separate protection for the preview endpoint are needed. The null check already in the design is sufficient. The preview button remains the right call — not because live-on-keystroke was dangerous, but because it is unnecessary work.

The forced `u` flag is a correctness requirement, demonstrated

Stripping one character from Café:

	Output	Valid UTF-8
without `u`	`CafÃ`	no
with `u`	`Caf`	yes

Without u the pattern operates on bytes, splits the two-byte é, and emits invalid UTF-8 — mojibake published to Lemmy. Confirms u must be always-on and never user-exposed.

No-op vs error in the preview

Three outcomes, unambiguously distinguishable:

Outcome	Detection	UI
Changed	`!== null` and `!== original`	before → after
No match	`!== null` and `=== original`	shown unchanged, not an error
Failed	`=== null`	error row, blocks save

The detection is trivial; the risk is presentational. A suffix rule legitimately will not match titles that lack the suffix, so rendering "no match" as a warning would make a correct rule look half-broken. Only a null return is an error.

## Open question resolved: PCRE limits measured Measured in the dev container (`container_tinker`), not assumed: ``` pcre.backtrack_limit 1000000 (PHP default) pcre.recursion_limit 100000 (PHP default) pcre.jit 1 (enabled) PCRE version 10.44 2024-06-07 ``` **Cost of a pathological pattern is bounded and small.** `/(a|a)+$/u` against inputs of 20–100 characters hits the limit in **~2.6ms, flat regardless of input length** — the backtrack limit is a hard counter, so cost is capped by the limit, not by input size. Ten sample titles in the preview is **~28ms worst case**. Note the textbook catastrophic pattern `/(a+)+b/u` does *not* blow up here at all (0.04ms, no error) — PCRE2's JIT and auto-possessification optimise it away. Constructing a genuinely pathological case took deliberate effort, which further bounds the practical risk. **Failure is cleanly detectable:** - `preg_replace` returns `null` - `preg_last_error()` returns `PREG_BACKTRACK_LIMIT_ERROR` (2), distinguishable from a malformed pattern - `preg_last_error_msg()` gives `"Backtrack limit exhausted"` So no timeout guard, no limit tuning, and no separate protection for the preview endpoint are needed. The `null` check already in the design is sufficient. **The preview button remains the right call** — not because live-on-keystroke was dangerous, but because it is unnecessary work. ## The forced `u` flag is a correctness requirement, demonstrated Stripping one character from `Café`: | | Output | Valid UTF-8 | |---|---|---| | without `u` | `CafÃ` | **no** | | with `u` | `Caf` | yes | Without `u` the pattern operates on bytes, splits the two-byte `é`, and emits invalid UTF-8 — mojibake published to Lemmy. Confirms `u` must be always-on and never user-exposed. ## No-op vs error in the preview Three outcomes, unambiguously distinguishable: | Outcome | Detection | UI | |---|---|---| | **Changed** | `!== null` and `!== original` | before → after | | **No match** | `!== null` and `=== original` | shown unchanged, **not** an error | | **Failed** | `=== null` | error row, blocks save | The detection is trivial; the risk is presentational. A suffix rule legitimately will not match titles that lack the suffix, so rendering "no match" as a warning would make a correct rule look half-broken. Only a `null` return is an error.