Thumbnails too large #138

New issue

Closed

opened 2026-08-13 15:17:42 +02:00 by myrmidex · 1 comment

myrmidex commented

2026-08-13 15:17:42 +02:00

Owner

The slow way of loading the article thumbnails on Lemmy has me suspecting it's using the big images as thumbnails. Verify whether this is correct. If so, we should come up with a way to have a smaller image as thumbnail.

myrmidex added this to the v1.4.0 milestone 2026-08-13 15:17:42 +02:00

myrmidex added the

bug

label 2026-08-13 15:17:42 +02:00

myrmidex referenced this issue

2026-08-13 20:25:09 +02:00

Add Reuters as a feed provider #139

myrmidex commented

2026-08-13 21:21:29 +02:00

Author

Owner

Confirmed, fixed, and verified against a live publish.

The suspicion was correct

Full-size images were being published as thumbnails. Measured from a real stored image_url:


Dimensions	4000 × 2256
Size	3.35 MB
Format	JPEG

All 49 Belga URLs end in the :full CDN selector. The chain was: parsers extract og:image → stored as articles.image_url → passed as custom_thumbnail to Lemmy, with no resizing anywhere.

Why URL rewriting could not fix it

Belga's picturepackcdn serves only full size. Probing :thumbnail, :large, :medium, :small and :preview all returned HTTP 200 with zero bytes — the selectors are valid grammar but no alternate rendition exists. VRT (images.vrt.be) does honour ?width=600 (90 KB → 46 KB; note ?w= is silently ignored), but that would have fixed only 23 of 72 articles.

Dropping custom_thumbnail and letting Lemmy generate its own was considered and rejected: Lemmy then fetches the article's og:image itself, which is the same oversized file. The slowness moves rather than disappears.

The fix

Download the original, resize with GD to max 600px wide, upload to the instance's own pict-rs, and use the returned instance-hosted URL.

Three commits:

6ea3f92 — LemmyRequest::postMultipart(); pict-rs sits outside the /api/v3 prefix the existing methods hardcode
34b78e6 — ThumbnailUploader plus wiring and failure logging
e16368a — hoist the upload out of the stale-token retry

That third commit fixes a regression the finish-phase full-diff review caught and the per-commit reviews structurally could not: the thumbnail work had been added inside createPost(), which publishToChannel() retries on a stale token. Since getToken() returns a cached token, that retry is the ordinary post-expiry path — so a retry meant a second full download, resize, upload, and a duplicate warning log. The upload now happens once per publish and the resolved URL is passed into both attempts.

Verified on a live publish

Lemmy post 2069227 on belgae.social:

thumbnail_url: https://belgae.social/pictrs/image/5b1bc0ed-d024-4033-82a1-d86726b36c59.jpeg

	Source (VRT CDN)	Hosted
Dimensions	1200 × 630	600 × 315
Size	83,227 bytes	30,101 bytes

Instance-hosted, not the source CDN. This also settles the one assumption the test suite could not: the pict-rs contract (multipart field images[], response path files.0.file) came from Lemmy 0.19's documentation rather than an observed upload, and the faked tests would have passed even if it were wrong. It is correct.

Safety

Every failure path returns null and publishing continues without a thumbnail — download failure, non-image body, oversize, GD failure, upload failure, malformed response. LemmyPublisher logs a warning whenever a thumbnail existed but no hosted URL came back, so a future break in the upload path surfaces rather than silently dropping thumbnails.

Memory guards matter here: the worker runs at memory_limit=128M and a 4000×2256 JPEG decodes to ~27 MB in GD, so MAX_SOURCE_BYTES (10 MB) and MAX_SOURCE_PIXELS (50M) are both checked before imagecreatefromstring(). GD is available; no new dependency.

Not covered

The live verification used a VRT article. The Belga path runs the same code at a much larger decode, and while the pixel guard is unit-tested, no real Belga publish has been observed. Worth watching the first one.

Verification: 1188 tests / 2891 assertions green, Pint clean (340 files), PHPStan clean. pr-reviewer run over the combined diff; its one must-fix is e16368a.

Confirmed, fixed, and verified against a live publish. ## The suspicion was correct Full-size images were being published as thumbnails. Measured from a real stored `image_url`: | | | |---|---| | Dimensions | **4000 × 2256** | | Size | **3.35 MB** | | Format | JPEG | All 49 Belga URLs end in the `:full` CDN selector. The chain was: parsers extract `og:image` → stored as `articles.image_url` → passed as `custom_thumbnail` to Lemmy, with no resizing anywhere. ## Why URL rewriting could not fix it Belga's `picturepackcdn` serves **only** full size. Probing `:thumbnail`, `:large`, `:medium`, `:small` and `:preview` all returned **HTTP 200 with zero bytes** — the selectors are valid grammar but no alternate rendition exists. VRT (`images.vrt.be`) does honour `?width=600` (90 KB → 46 KB; note `?w=` is silently ignored), but that would have fixed only 23 of 72 articles. Dropping `custom_thumbnail` and letting Lemmy generate its own was considered and rejected: Lemmy then fetches the article's `og:image` itself, which is the same oversized file. The slowness moves rather than disappears. ## The fix Download the original, resize with GD to max 600px wide, upload to the instance's own pict-rs, and use the returned instance-hosted URL. Three commits: - `6ea3f92` — `LemmyRequest::postMultipart()`; pict-rs sits outside the `/api/v3` prefix the existing methods hardcode - `34b78e6` — `ThumbnailUploader` plus wiring and failure logging - `e16368a` — hoist the upload out of the stale-token retry That third commit fixes a regression the finish-phase full-diff review caught and the per-commit reviews structurally could not: the thumbnail work had been added inside `createPost()`, which `publishToChannel()` retries on a stale token. Since `getToken()` returns a *cached* token, that retry is the ordinary post-expiry path — so a retry meant a second full download, resize, upload, and a duplicate warning log. The upload now happens once per publish and the resolved URL is passed into both attempts. ## Verified on a live publish Lemmy post **2069227** on belgae.social: ``` thumbnail_url: https://belgae.social/pictrs/image/5b1bc0ed-d024-4033-82a1-d86726b36c59.jpeg ``` | | Source (VRT CDN) | Hosted | |---|---|---| | Dimensions | 1200 × 630 | **600 × 315** | | Size | 83,227 bytes | **30,101 bytes** | Instance-hosted, not the source CDN. This also settles the one assumption the test suite could not: the pict-rs contract (multipart field `images[]`, response path `files.0.file`) came from Lemmy 0.19's documentation rather than an observed upload, and the faked tests would have passed even if it were wrong. It is correct. ## Safety Every failure path returns `null` and publishing continues without a thumbnail — download failure, non-image body, oversize, GD failure, upload failure, malformed response. `LemmyPublisher` logs a warning whenever a thumbnail existed but no hosted URL came back, so a future break in the upload path surfaces rather than silently dropping thumbnails. Memory guards matter here: the worker runs at `memory_limit=128M` and a 4000×2256 JPEG decodes to ~27 MB in GD, so `MAX_SOURCE_BYTES` (10 MB) and `MAX_SOURCE_PIXELS` (50M) are both checked **before** `imagecreatefromstring()`. GD is available; no new dependency. ## Not covered The live verification used a **VRT** article. The Belga path runs the same code at a much larger decode, and while the pixel guard is unit-tested, no real Belga publish has been observed. Worth watching the first one. **Verification:** 1188 tests / 2891 assertions green, Pint clean (340 files), PHPStan clean. `pr-reviewer` run over the combined diff; its one must-fix is `e16368a`.

myrmidex closed this issue

2026-08-13 21:21:36 +02:00

myrmidex referenced this issue

2026-08-15 00:02:58 +02:00

Release v1.4.0 #146