lvl0/fedi-feed-router
1
0
Fork 0
Code Issues 27 Pull requests Projects Releases 13 Packages Wiki Activity Actions

Speed up CI by prebuilding a PHP image instead of installing PHP on every run #147

New issue
Closed
opened 2026-08-15 00:05:36 +02:00 by myrmidex · 3 comments
myrmidex commented 2026-08-15 00:05:36 +02:00
Owner
Copy link

Problem

CI runs take around 15 minutes. Recent runs on release/v1.4.0:

Run Duration
#79 15m48s
#80 15m26s

The job runs on catthehacker/ubuntu:act-latest, which ships no PHP. Every run
therefore installs it from scratch via setup-php@v2
(.forgejo/workflows/ci.yml:17-22):

  • PHP 8.3
  • extensions pdo_sqlite, mbstring, xml, dom
  • pcov for coverage, which is compiled rather than downloaded

The Cache Composer dependencies step immediately below only caches
~/.composer/cache. It does nothing for the PHP installation, so that work is
repeated on every run regardless of cache state.

To confirm first

The per-step timings have not been captured. Before changing anything, record
where the 15 minutes actually goes, from the run detail view of a completed run.
Set up PHP is the suspected bulk of it, but that is currently an inference
from what the step does rather than a measurement, and the Tests step runs
the full suite (1231 tests) with coverage enabled, which is not free either.

Fixing the wrong step is the main risk here, so measure before optimising.

Options

A. Prebuilt CI base image. Build an image with PHP 8.3, the four extensions
and pcov already installed, publish it alongside the existing application
image, and use it as the job's container.image. Set up PHP then disappears
from the workflow entirely.

This is an established pattern in this repo rather than new infrastructure: the
repository already builds and pushes images to forge.lvl0.xyz/lvl0, and run #5
("40 - Add base image for faster CI build") solved the same problem for the
Docker build. The cost is a second image to maintain and rebuild when the PHP
version or extension set changes.

B. Cache the PHP installation. Keep setup-php but cache what it produces.
Less invasive, but setup-php's own cache support is the thing to check here,
and a partially warm cache still leaves per-run work.

C. Drop pcov where coverage is not used. Coverage is currently collected
on every run (--coverage-clover --coverage-text) but nothing consumes it: the
coverage PR comment was removed in #127. If coverage is not being read, both
compiling pcov and running the suite under it are unnecessary work on every
run.

These are not mutually exclusive. C is the smallest change and may be worth
doing regardless of A or B.

Acceptance criteria

  • Per-step timings recorded for a run before any change, so the bottleneck
    is identified rather than assumed
  • CI runtime measurably reduced, with before and after figures noted on this
    ticket
  • All three gates still run: Pint, PHPStan, and both test suites
  • Coverage either still collected, or its removal is a deliberate decision
    recorded here
  • If a base image is introduced, its build and update path is documented in
    .claude/PLATFORM.md

Related

  • #127 removed the coverage PR comment, which is why nothing currently consumes
    the coverage output
## Problem CI runs take around 15 minutes. Recent runs on `release/v1.4.0`: | Run | Duration | |-----|----------| | #79 | 15m48s | | #80 | 15m26s | The job runs on `catthehacker/ubuntu:act-latest`, which ships no PHP. Every run therefore installs it from scratch via `setup-php@v2` (`.forgejo/workflows/ci.yml:17-22`): - PHP 8.3 - extensions `pdo_sqlite, mbstring, xml, dom` - `pcov` for coverage, which is compiled rather than downloaded The `Cache Composer dependencies` step immediately below only caches `~/.composer/cache`. It does nothing for the PHP installation, so that work is repeated on every run regardless of cache state. ## To confirm first The per-step timings have not been captured. Before changing anything, record where the 15 minutes actually goes, from the run detail view of a completed run. `Set up PHP` is the suspected bulk of it, but that is currently an inference from what the step does rather than a measurement, and the `Tests` step runs the full suite (1231 tests) with coverage enabled, which is not free either. Fixing the wrong step is the main risk here, so measure before optimising. ## Options **A. Prebuilt CI base image.** Build an image with PHP 8.3, the four extensions and `pcov` already installed, publish it alongside the existing application image, and use it as the job's `container.image`. `Set up PHP` then disappears from the workflow entirely. This is an established pattern in this repo rather than new infrastructure: the repository already builds and pushes images to `forge.lvl0.xyz/lvl0`, and run #5 ("40 - Add base image for faster CI build") solved the same problem for the Docker build. The cost is a second image to maintain and rebuild when the PHP version or extension set changes. **B. Cache the PHP installation.** Keep `setup-php` but cache what it produces. Less invasive, but `setup-php`'s own cache support is the thing to check here, and a partially warm cache still leaves per-run work. **C. Drop `pcov` where coverage is not used.** Coverage is currently collected on every run (`--coverage-clover --coverage-text`) but nothing consumes it: the coverage PR comment was removed in #127. If coverage is not being read, both compiling `pcov` and running the suite under it are unnecessary work on every run. These are not mutually exclusive. C is the smallest change and may be worth doing regardless of A or B. ## Acceptance criteria - [ ] Per-step timings recorded for a run before any change, so the bottleneck is identified rather than assumed - [ ] CI runtime measurably reduced, with before and after figures noted on this ticket - [ ] All three gates still run: Pint, PHPStan, and both test suites - [ ] Coverage either still collected, or its removal is a deliberate decision recorded here - [ ] If a base image is introduced, its build and update path is documented in `.claude/PLATFORM.md` ## Related - #127 removed the coverage PR comment, which is why nothing currently consumes the coverage output
myrmidex added this to the v1.5.0 milestone 2026-08-15 00:05:36 +02:00
myrmidex added the
devops
 label 2026-08-15 00:05:36 +02:00
myrmidex commented 2026-08-15 00:11:57 +02:00
Author
Owner
Copy link

Decisions

Set up PHP confirmed as the slow step. Observed directly on run #81 (the
PR pipeline for #146), which sat on that step for the bulk of its runtime. This
was previously an inference from what the step does; it is now observed.

Coverage is dropped. Option C is taken, and not only as a speed measure.

Coverage output has had no consumer since #127 removed the PR comment, so every
run has been compiling pcov and executing 1231 tests under instrumentation to
produce a file nobody reads.

A README coverage badge was considered as an alternative use and rejected:

  • The usual route is a third-party service (Codecov, Coveralls) that ingests
    coverage.xml and serves the badge. That means sending coverage data
    off-instance and giving an external service access, which does not suit a
    self-hosted Forgejo.
  • Self-generating the SVG in CI works, but needs a write token and a commit on
    every run, which is noisy and risks re-triggering CI.
  • More fundamentally: the PR comment was the more actionable form and was not
    often read. A number on the README is less actionable, not more.

Coverage is worth having as a gate (fail when it drops below a threshold)
rather than a display, because a gate changes what happens. That is a separate
decision and a separate ticket if it is ever wanted; it is not part of this one.

Removing --coverage-clover coverage.xml --coverage-text also removes the need
for pcov in the setup-php extension list, which shortens that step even
before any base image work.

Revised approach

  1. Drop coverage from the Tests step and pcov from setup-php. Measure.
  2. If runtime is still unacceptable, prebuild a CI base image (option A).

Step 1 is a two-line change and may be enough on its own.

## Decisions **`Set up PHP` confirmed as the slow step.** Observed directly on run #81 (the PR pipeline for #146), which sat on that step for the bulk of its runtime. This was previously an inference from what the step does; it is now observed. **Coverage is dropped.** Option C is taken, and not only as a speed measure. Coverage output has had no consumer since #127 removed the PR comment, so every run has been compiling `pcov` and executing 1231 tests under instrumentation to produce a file nobody reads. A README coverage badge was considered as an alternative use and rejected: - The usual route is a third-party service (Codecov, Coveralls) that ingests `coverage.xml` and serves the badge. That means sending coverage data off-instance and giving an external service access, which does not suit a self-hosted Forgejo. - Self-generating the SVG in CI works, but needs a write token and a commit on every run, which is noisy and risks re-triggering CI. - More fundamentally: the PR comment was the more actionable form and was not often read. A number on the README is less actionable, not more. Coverage is worth having as a **gate** (fail when it drops below a threshold) rather than a display, because a gate changes what happens. That is a separate decision and a separate ticket if it is ever wanted; it is not part of this one. Removing `--coverage-clover coverage.xml --coverage-text` also removes the need for `pcov` in the `setup-php` extension list, which shortens that step even before any base image work. ## Revised approach 1. Drop coverage from the `Tests` step and `pcov` from `setup-php`. Measure. 2. If runtime is still unacceptable, prebuild a CI base image (option A). Step 1 is a two-line change and may be enough on its own.
myrmidex commented 2026-08-15 00:37:44 +02:00
Author
Owner
Copy link

Baseline timings

Recorded before any change, for the before/after comparison in the acceptance
criteria. All four ran the same three gates:

Run Trigger Duration
#79 push to release/v1.4.0 15m48s
#80 push to release/v1.4.0 15m26s
#81 pull_request on #146 32m18s

Runs #80 and #81 tested the same commit (4f79aac), one via push and one
via the PR trigger. The identical work took 15m26s and 32m18s, so runtime here
varies by more than a factor of two independent of what changed.

The wider history shows the same spread: #74 took 31m19s and #63 took 1h26m47s,
both eventually succeeding, against a low of ~4m for comparable runs.

So there are two problems, not one:

  1. The floor is high. Even a fast run is ~15 minutes for a three-gate job.
  2. The variance is worse than the floor. A one-line markdown change can cost
    half an hour, which makes CI unusable as a quick feedback loop and is why a
    trivial docs commit held up the v1.4.0 release.

Both point the same way: work that happens on every run and depends on the
network (installing PHP, compiling pcov, fetching packages) is the part
exposed to this variance. Moving it into a prebuilt image removes it from the
per-run path entirely, which addresses the variance and not just the average.

## Baseline timings Recorded before any change, for the before/after comparison in the acceptance criteria. All four ran the same three gates: | Run | Trigger | Duration | |-----|---------|----------| | #79 | push to `release/v1.4.0` | 15m48s | | #80 | push to `release/v1.4.0` | 15m26s | | #81 | `pull_request` on #146 | 32m18s | Runs #80 and #81 tested the **same commit** (`4f79aac`), one via push and one via the PR trigger. The identical work took 15m26s and 32m18s, so runtime here varies by more than a factor of two independent of what changed. The wider history shows the same spread: #74 took 31m19s and #63 took 1h26m47s, both eventually succeeding, against a low of ~4m for comparable runs. So there are two problems, not one: 1. **The floor is high.** Even a fast run is ~15 minutes for a three-gate job. 2. **The variance is worse than the floor.** A one-line markdown change can cost half an hour, which makes CI unusable as a quick feedback loop and is why a trivial docs commit held up the v1.4.0 release. Both point the same way: work that happens on every run and depends on the network (installing PHP, compiling `pcov`, fetching packages) is the part exposed to this variance. Moving it into a prebuilt image removes it from the per-run path entirely, which addresses the variance and not just the average.
myrmidex modified the milestone from v1.5.0 to v1.4.1 2026-08-15 10:59:20 +02:00
myrmidex commented 2026-08-15 11:16:26 +02:00
Author
Owner
Copy link

Result

2m12s, against a baseline of 15m26s / 15m48s / 32m18s.

Per-step, on release/v1.4.1:

Step Duration
Set up job 9s
checkout@v4 11s
Cache Composer dependencies 1s
Install dependencies 15s
Prepare environment 1s
Lint 10s
Static analysis 17s
Tests 1m6s
Total 2m12s

Set up job was the whole problem. It previously carried the setup-php
install of PHP 8.3, four extensions and a compiled pcov on an image with no
PHP at all. Pulling a prebuilt image instead takes 9 seconds.

The suspicion recorded when this ticket was filed turned out to be correct, and
was confirmed directly during the work by watching a run sit on that step.

What was done

  1. Dropped coverage. Nothing had consumed it since #127 removed the coverage
    PR comment, so every run was compiling pcov and running 1231 tests under
    instrumentation to produce a file nobody read. coverage: none and plain
    phpunit.
  2. Prebuilt CI image (docker/build/Dockerfile.ci), Debian-based, carrying
    PHP 8.3 plus the extensions composer.lock requires and gd, which the
    application calls directly without declaring. setup-php removed entirely.
  3. images.yml to build and push both this and the pre-existing base image
    (#148).
  4. Fixed the Composer cache path. ci.yml cached ~/.composer/cache;
    Composer 2 uses ~/.cache/composer, so the cache step had been storing and
    restoring nothing.
  5. Ran PHPUnit directly with an explicit memory limit. artisan test spawns
    a subprocess that ignores -d memory_limit, and the suite needs 172.5MB.
  6. Added release/* to the pull_request trigger, so branches merging into
    a release branch get CI. Previously only PRs into main did.

Things learned along the way

ext-gd was undeclared. The extension list was derived from composer.lock,
which only lists what packages declare. ThumbnailUploader calls gd directly and
nothing required it, so composer install passed and seven tests failed at
runtime instead. ext-gd is now declared in composer.json so the platform
check catches it.

A mutable :latest tag cost two CI runs. The registry held a correct image
and the runner kept using an older cached copy. Consumers now pin an explicit
version tag; see the comment on #148.

Remaining

  • The Composer cache has not yet been observed hitting. This run took 15s to
    install, which is fast enough that it may already be warm, but a second run
    with an unchanged composer.lock would confirm it.
  • #148 still automates the image builds. Until it lands, Dockerfile.ci changes
    need a manual build and push, and the version tag bumped in ci.yml and
    images.yml.
## Result **2m12s**, against a baseline of 15m26s / 15m48s / 32m18s. Per-step, on `release/v1.4.1`: | Step | Duration | |------|----------| | Set up job | 9s | | checkout@v4 | 11s | | Cache Composer dependencies | 1s | | Install dependencies | 15s | | Prepare environment | 1s | | Lint | 10s | | Static analysis | 17s | | Tests | 1m6s | | **Total** | **2m12s** | `Set up job` was the whole problem. It previously carried the `setup-php` install of PHP 8.3, four extensions and a compiled `pcov` on an image with no PHP at all. Pulling a prebuilt image instead takes 9 seconds. The suspicion recorded when this ticket was filed turned out to be correct, and was confirmed directly during the work by watching a run sit on that step. ## What was done 1. **Dropped coverage.** Nothing had consumed it since #127 removed the coverage PR comment, so every run was compiling `pcov` and running 1231 tests under instrumentation to produce a file nobody read. `coverage: none` and plain `phpunit`. 2. **Prebuilt CI image** (`docker/build/Dockerfile.ci`), Debian-based, carrying PHP 8.3 plus the extensions `composer.lock` requires and `gd`, which the application calls directly without declaring. `setup-php` removed entirely. 3. **`images.yml`** to build and push both this and the pre-existing base image (#148). 4. **Fixed the Composer cache path.** `ci.yml` cached `~/.composer/cache`; Composer 2 uses `~/.cache/composer`, so the cache step had been storing and restoring nothing. 5. **Ran PHPUnit directly** with an explicit memory limit. `artisan test` spawns a subprocess that ignores `-d memory_limit`, and the suite needs 172.5MB. 6. **Added `release/*` to the `pull_request` trigger**, so branches merging into a release branch get CI. Previously only PRs into `main` did. ## Things learned along the way **`ext-gd` was undeclared.** The extension list was derived from `composer.lock`, which only lists what packages declare. `ThumbnailUploader` calls gd directly and nothing required it, so `composer install` passed and seven tests failed at runtime instead. `ext-gd` is now declared in `composer.json` so the platform check catches it. **A mutable `:latest` tag cost two CI runs.** The registry held a correct image and the runner kept using an older cached copy. Consumers now pin an explicit version tag; see the comment on #148. ## Remaining - The Composer cache has not yet been observed hitting. This run took 15s to install, which is fast enough that it may already be warm, but a second run with an unchanged `composer.lock` would confirm it. - #148 still automates the image builds. Until it lands, `Dockerfile.ci` changes need a manual build and push, and the version tag bumped in `ci.yml` and `images.yml`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.4.2
v1.4.1
v1.4.0
v1.3.7
v1.3.6
v1.3.5
v1.3.4
v1.3.3
v1.3.2
v1.3.1
v1.3.0
v1.2.0
v1.1.0
v1.0.0
v0.2.0
v0.1.0
Labels
Clear labels   
bug

Something is not working

  
devops

All things Ci/CD

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
good first issue

Interested in contributing? Get started here.

  
layout

UI/UX

  
next major release

Include in next Major Release

  
next minor release

Include in next minor release

  
question

More information is needed

  
research

   
testing

All sorts of testing: UI, automation, ...

No labels
bug
devops
duplicate
enhancement
good first issue
layout
next major release
next minor release
question
research
testing
Milestone
Clear milestone
No items
No milestone
v1.4.1
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lvl0/fedi-feed-router#147
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?