Populate Article::image_url instead of re-fetching the thumbnail on every publish #119

New issue

Closed

opened 2026-08-01 17:14:29 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-01 17:14:29 +02:00

Owner

Summary

articles.image_url exists but is never written by production code. The thumbnail is instead re-extracted from the live article page on every publish, which means an avoidable HTTP request per post and no thumbnail available anywhere else in the app.

Current state

Only three references exist:

Location	Reference
`app/Models/Article.php:41`	`'image_url'` in `$fillable`
`database/migrations/2024_01_01_000001_create_articles_and_publications.php:18`	`$table->string('image_url')->nullable();`
`database/factories/ArticleFactory.php:27`	`'image_url' => $this->faker->optional()->imageUrl(),`

Nothing in app/ writes it, nothing in resources/ reads it, and no test asserts it. In the dev database, 0 of 53 articles have a value.

How the thumbnail flows today

PublishNextArticleJob → ArticleFetcher::fetchArticleData($article) → {Belga,Vrt,Guardian}ArticlePageParser::extractThumbnail() → LemmyPublisher::createPost() → LemmyApiService sets custom_thumbnail.

fetchArticleData() performs a live HTTP fetch of the article page each time it runs. Since it is called at publish time, publishing an article to N routes re-fetches the same page N times, purely to recover a value that was already available when the article was first discovered.

Proposed

Persist the thumbnail when the article's content is fetched, and read from the column at publish time.

Write image_url during article fetch/validation, where extractData() is already being called
LemmyPublisher uses $article->image_url when present, falling back to live extraction when null (so existing articles keep working)
Backfill is optional — the fallback covers older rows

Design questions

Where exactly to write it. ValidationService::validate() already fetches article content and sets validated_at, so it is the natural point. Confirm extractData() is called there and that its thumbnail key is available.
Staleness. A persisted URL could rot if the publisher changes their CDN. Given articles are published within days of discovery and cleaned up after 30 days (CleanupArticlesJob), this is likely acceptable — but worth a deliberate decision rather than an accident.
Fallback behaviour. Decide whether a null image_url triggers live extraction (safer, keeps current behaviour) or is treated as "no thumbnail" (simpler, but changes behaviour for existing rows).

Follow-on value

With image_url populated, the Articles page could show thumbnails in the review list — useful when triaging what to approve. Out of scope here, but the reason to populate rather than drop the column.

Alternative considered

Dropping the column entirely. It is genuinely unused and removal would be clean (3 references, no tests, no views, no data). Rejected because the thumbnail is real data the app already extracts and currently discards, and re-fetching it per publish is wasteful — the column is under-used rather than dead.

Acceptance criteria

image_url is populated when an article's content is fetched
Publishing uses the stored value when available
Articles with a null image_url still publish with a thumbnail (fallback path)
Factory and any fixtures remain consistent
Tests cover: value persisted on fetch, publish uses stored value, null falls back to live extraction

#117 — thumbnail URLs were not entity-decoded; found while investigating this same path
#118 — E2E tests; the publish payload shape is one of the things that ticket proposes asserting

## Summary `articles.image_url` exists but is never written by production code. The thumbnail is instead re-extracted from the live article page on **every** publish, which means an avoidable HTTP request per post and no thumbnail available anywhere else in the app. ## Current state Only three references exist: | Location | Reference | |---|---| | `app/Models/Article.php:41` | `'image_url'` in `$fillable` | | `database/migrations/2024_01_01_000001_create_articles_and_publications.php:18` | `$table->string('image_url')->nullable();` | | `database/factories/ArticleFactory.php:27` | `'image_url' => $this->faker->optional()->imageUrl(),` | Nothing in `app/` writes it, nothing in `resources/` reads it, and no test asserts it. In the dev database, **0 of 53 articles** have a value. ## How the thumbnail flows today `PublishNextArticleJob` → `ArticleFetcher::fetchArticleData($article)` → `{Belga,Vrt,Guardian}ArticlePageParser::extractThumbnail()` → `LemmyPublisher::createPost()` → `LemmyApiService` sets `custom_thumbnail`. `fetchArticleData()` performs a **live HTTP fetch of the article page** each time it runs. Since it is called at publish time, publishing an article to N routes re-fetches the same page N times, purely to recover a value that was already available when the article was first discovered. ## Proposed Persist the thumbnail when the article's content is fetched, and read from the column at publish time. - Write `image_url` during article fetch/validation, where `extractData()` is already being called - `LemmyPublisher` uses `$article->image_url` when present, falling back to live extraction when null (so existing articles keep working) - Backfill is optional — the fallback covers older rows ### Design questions - **Where exactly to write it.** `ValidationService::validate()` already fetches article content and sets `validated_at`, so it is the natural point. Confirm `extractData()` is called there and that its `thumbnail` key is available. - **Staleness.** A persisted URL could rot if the publisher changes their CDN. Given articles are published within days of discovery and cleaned up after 30 days (`CleanupArticlesJob`), this is likely acceptable — but worth a deliberate decision rather than an accident. - **Fallback behaviour.** Decide whether a null `image_url` triggers live extraction (safer, keeps current behaviour) or is treated as "no thumbnail" (simpler, but changes behaviour for existing rows). ## Follow-on value With `image_url` populated, the Articles page could show thumbnails in the review list — useful when triaging what to approve. Out of scope here, but the reason to populate rather than drop the column. ## Alternative considered Dropping the column entirely. It is genuinely unused and removal would be clean (3 references, no tests, no views, no data). Rejected because the thumbnail is real data the app already extracts and currently discards, and re-fetching it per publish is wasteful — the column is under-used rather than dead. ## Acceptance criteria - [ ] `image_url` is populated when an article's content is fetched - [ ] Publishing uses the stored value when available - [ ] Articles with a null `image_url` still publish with a thumbnail (fallback path) - [ ] Factory and any fixtures remain consistent - [ ] Tests cover: value persisted on fetch, publish uses stored value, null falls back to live extraction ## Related - #117 — thumbnail URLs were not entity-decoded; found while investigating this same path - #118 — E2E tests; the publish payload shape is one of the things that ticket proposes asserting