CI image rebuild takes 35min–3h and blocks every lockfile change #61
Labels
No labels
app
backlog
bug
ci-cd
contribution welcome
duplicate
enhancement
good first issue
help wanted
question
testing
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: lvl0/dishplanner#61
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
The
ci-imagejob in.forgejo/workflows/ci.ymltakes between 35 minutes and 3+ hours whenever it has to build. Thecijob (needs: ci-image) is blocked behind it, so a slow or cancelled image build means lint, static analysis, tests, and browser tests never run at all.Evidence
Observed durations from
list_action_tasks:54 - Fix Pint import issuescancelled46 - Tag CI image by composer.lock hashfailure50 - Fix tests failing on missing Vite manifestsuccess50 - Avoid codeload 429 via prefer-sourcefailure50 - Avoid codeload 429 via prefer-sourcefailureRun #20 spent 56m53s in "Build and push CI image" and was cancelled before the push completed. The
cijob showed0s— it never started.Root cause
docker/build/Dockerfile.ciends with:--prefer-sourcemakes Composer perform a fullgit cloneof every package instead of fetching dist zips. It was introduced deliberately in50 - Avoid codeload 429 via prefer-source and baked phpstanto work around codeload.github.com rate limits (429s) hit under--prefer-dist. It solved the 429s and replaced them with a 35min–3h build.This is now the dominant cost in CI, and it is paid in full on every cache miss.
Why it is biting now
The image tag is content-addressed from the lockfile:
dcfa73b(#54, Dusk → Pest) rewrote 3370 lines ofcomposer.lock, producing a new tagphp8.3-80acd7b52304. Thecache-fromregistry ref was built against the old dependency set, so the build starts cold.The tag has still never been pushed. Current registry contents:
php8.3-1…php8.3-4are from the old manual revision scheme, before #46 switched to lockfile hashing. No hash-tagged image has ever landed successfully.Interaction with
1e826e71e826e7 - Skip CI image rebuild when the lockfile tag already existsadds a registry existence check so the build is skipped when the tag is already published. That fix is correct but cannot engage until one build completes, because it skips only when the tag exists.Current state is therefore a loop risk: the next push pays a full cold build; if that build is again cancelled or times out, nothing is pushed, and the following push pays it again.
Suggested directions
--prefer-distwith a Composer auth token (COMPOSER_AUTHwith a GitHub token) to lift the codeload 429 limit that forced--prefer-source. Authenticated requests get a far higher rate limit, which may make the original workaround unnecessary.1e826e7's skip logic take effect.cache-to: mode=maxregistry caching is actually being restored across runs — the observed durations suggest it may not be effective.Acceptance
composer.lockskip the build (verifying1e826e7)cijob runs to completion, including the browser testsNotes
--prefer-sourcecost predates it (introduced under #50). #54's lockfile change is what forced the cold rebuild that exposed it.Investigation — root cause found, supersedes the original analysis
Three compounding causes, not one. The original description's "suggested directions" were partly wrong; corrections noted at the end.
1. The registry cache has never worked — and cannot, as configured
cache-from/cache-topoint atdishplanner-ci:buildcache. That tag returns 404, whilelatest,php8.3-1…php8.3-4all return 200:It has never been written once.
This is a known Forgejo/Gitea registry limitation, not a misconfiguration here. BuildKit represents its registry cache as an OCI Image Index whose manifest array contains layer media types rather than the manifest media types the spec requires. Gitea/Forgejo's registry validates strictly and rejects the push:
Consequence:
cache-tofails every run,cache-frommisses every run, so every build is fully cold. This is the self-sustaining loop — builds too slow to finish → cache never written → next build cold again.2.
--prefer-sourceclones 154 git repositoriescomposer.lockcontains 78 packages + 76 dev packages = 154 total. Under--prefer-sourceeach is a fullgit clonewith complete history, performed sequentially.For comparison, the local
vendor/is 157M containing zero.gitdirectories — a dist install. CI is doing categorically more work than any developer machine does.3. The 429 that forced
--prefer-sourceis an auth problem, not a dist problemThis is the actionable finding. Measured:
Composer makes API calls to resolve dist URLs for all 154 packages, which exhausts the unauthenticated 60/hour and produces the 429s that
50 - Avoid codeload 429 via prefer-sourcewas written to dodge.Meanwhile codeload itself is not rate-limited for these fetches — an unauthenticated download of
laravel/frameworkv12.0.0 returnedHTTP 200, 1974794 bytes. The file host was never the bottleneck; the API metadata calls were.--prefer-sourcetreated the symptom rather than the cause.What the Dockerfile already gets right
Layer order is correct. Extensions, apt packages, and the composer binary all sit above
COPY composer.json composer.lock ./, so only a lockfile change invalidates the expensive install layer:No restructuring needed. The layering simply never gets to pay off, because the cache backend is broken.
Revised recommendations
COMPOSER_AUTHsecret with a GitHub token and revert to--prefer-dist. Addresses the actual cause. Expected to collapse the build from 35min–3h to a few minutes. Requires a PAT with no scopes — public read access is sufficient. This step needs a repo secret to be created manually.cache-from/cache-to, or move the cache off the Forgejo registry. As written they are dead weight: every run pays the export attempt for a cache that can never land. Alternatives aretype=ghaif the runner supports it,type=localwith runner-side storage, or nothing at all — the existence check added in1e826e7already prevents redundant rebuilds once a tag exists.php8.3-80acd7b52304lands in the registry and activates1e826e7's skip logic.Corrections to the original description
"Split the baked dependencies into a separate base image — this is the structural fix"— unnecessary. The layering is already correct; the cache backend is what is broken."Verify— verified: it is not, and cannot be on this registry. Now cause #1 above.cache-to: mode=maxregistry caching is actually being restored"--prefer-dist+ auth token suggestion is confirmed as the right direction by the rate-limit measurements.No code changes were made as part of this investigation.
Resolved
Fixed in
cfad3dcand7f7a945, following the revised recommendations.cfad3dc— Authenticate composer dist downloads and drop the dead registry cacheGH_PATmounted as a build secret and set asgithub-oauthfor the install, then unset. Lifts the API limit from 60 to 5000 requests/hour, which removes the reason--prefer-sourceexisted; the install is back on dist.cache-from/cache-toremoved. They could never land on the Forgejo registry, so every run was paying an export attempt for a cache that never existed.7f7a945— Bake node modules and Chromium into the CI imageBeyond the original scope:
node_modulesand the Playwright Chromium build are now baked at/opt/deps-nodeand/opt/playwrightalongsidevendor/. The image tag hashescomposer.lock+package-lock.jsontogether, so both stay keyed to the image.Acceptance
success1e826e7's skip logic engagedcijob runs to completion, including browser testscineedsci-image, andBrowser testsis its final stepRun durations for the loop this closes: #19
failure3h02m → #20cancelled1h13m → #21cancelled1h11m → #22success22m42s → #23success11m54s.Follow-ups
AGENTS.mdgained a## CIsection recording theGH_PATrequirement and the Forgejo/BuildKit cache limitation, so neither gets rediscovered. Its stale "browser tests currently use Dusk" line was corrected at the same time.