lvl0/fedi-feed-router

Fork 0

CI runner intermittently fails to provision containers and resolve DNS #153

New issue

Closed

opened 2026-08-15 09:39:11 +02:00 by myrmidex · 2 comments

myrmidex commented

2026-08-15 09:39:11 +02:00

Owner

Problem

CI runs fail for reasons unrelated to the code being tested. Two distinct
symptoms, both on the runner rather than in the workflow.

1. Container provisioning fails

Run #151 on fix/150-migration-rate-limit:

Set up job                14m50s
checkout@v4                   0s
Set up PHP                    0s
Cache Composer dependencies   0s
Install dependencies          0s
Prepare environment           0s
Lint                          0s
Static analysis               0s
Tests                         0s
ci                        Failure

Nearly fifteen minutes in Set up job, then every step at 0s. The job never
reached checkout, so nothing in the commit was executed. The runner could not
pull or start the job container.

2. DNS resolution times out mid-install

Two separate runs, different packages, each after roughly 118 packages had
already downloaded successfully:

Failed to download symfony/polyfill-mbstring from dist: curl error 28 while
downloading https://codeload.github.com/... : Resolving timed out after 10000 ms

Failed to download vlucas/phpdotenv from dist: curl error 28 while
downloading https://codeload.github.com/... : Resolving timed out after 10002 ms

Composer's own message points at the resolver:

The following exception probably indicates you have misconfigured DNS resolver(s)

Not the container image

The DNS timeouts were initially attributed to musl's resolver in the alpine
image introduced by #147, and the image was changed to Debian on that basis.

That explanation does not hold: the provisioning failure in run #151 happened on
catthehacker/ubuntu:act-latest, the original Debian-based image, on a branch
that does not contain any of #147's changes. The image is not the common factor.

Both symptoms point at the runner host: name resolution and outbound
connectivity to container registries and to codeload.github.com.

Why this matters beyond the noise

These failures are indistinguishable from code failures in the run list. Effort
has already gone into chasing an image-level explanation for what appears to be
a host-level problem, and a red CI check is currently sitting on #151, the PR
carrying the production migration fix.

Where to look

The runner host's /etc/resolv.conf and which resolver it points at
Whether the runner can reliably pull from forge.lvl0.xyz and Docker Hub
Any rate limiting or filtering between the runner and codeload.github.com
Whether the runner has a network or disk constraint that makes image pulls
take fifteen minutes

Reduces exposure regardless of cause

The Composer cache is not working: every run downloads all 119 packages fresh.
ci.yml caches ~/.composer/cache while Composer 2 actually uses
~/.cache/composer, so the cache step stores and restores nothing. #147
contains a fix for the path. A working cache removes ~119 network fetches per
run and with them most of the exposure to this failure mode.

Acceptance criteria

Cause of the provisioning failure identified
Cause of the DNS timeouts identified
CI runs complete without infrastructure-related failures across several
consecutive runs
Composer cache verified to actually hit, so package downloads stop
happening on every run

## Problem CI runs fail for reasons unrelated to the code being tested. Two distinct symptoms, both on the runner rather than in the workflow. ### 1. Container provisioning fails Run #151 on `fix/150-migration-rate-limit`: ``` Set up job 14m50s checkout@v4 0s Set up PHP 0s Cache Composer dependencies 0s Install dependencies 0s Prepare environment 0s Lint 0s Static analysis 0s Tests 0s ci Failure ``` Nearly fifteen minutes in `Set up job`, then every step at 0s. The job never reached checkout, so nothing in the commit was executed. The runner could not pull or start the job container. ### 2. DNS resolution times out mid-install Two separate runs, different packages, each after roughly 118 packages had already downloaded successfully: ``` Failed to download symfony/polyfill-mbstring from dist: curl error 28 while downloading https://codeload.github.com/... : Resolving timed out after 10000 ms ``` ``` Failed to download vlucas/phpdotenv from dist: curl error 28 while downloading https://codeload.github.com/... : Resolving timed out after 10002 ms ``` Composer's own message points at the resolver: ``` The following exception probably indicates you have misconfigured DNS resolver(s) ``` ## Not the container image The DNS timeouts were initially attributed to musl's resolver in the alpine image introduced by #147, and the image was changed to Debian on that basis. That explanation does not hold: the provisioning failure in run #151 happened on `catthehacker/ubuntu:act-latest`, the original Debian-based image, on a branch that does not contain any of #147's changes. The image is not the common factor. Both symptoms point at the runner host: name resolution and outbound connectivity to container registries and to `codeload.github.com`. ## Why this matters beyond the noise These failures are indistinguishable from code failures in the run list. Effort has already gone into chasing an image-level explanation for what appears to be a host-level problem, and a red CI check is currently sitting on #151, the PR carrying the production migration fix. ## Where to look - The runner host's `/etc/resolv.conf` and which resolver it points at - Whether the runner can reliably pull from `forge.lvl0.xyz` and Docker Hub - Any rate limiting or filtering between the runner and `codeload.github.com` - Whether the runner has a network or disk constraint that makes image pulls take fifteen minutes ## Reduces exposure regardless of cause The Composer cache is not working: every run downloads all 119 packages fresh. `ci.yml` caches `~/.composer/cache` while Composer 2 actually uses `~/.cache/composer`, so the cache step stores and restores nothing. #147 contains a fix for the path. A working cache removes ~119 network fetches per run and with them most of the exposure to this failure mode. ## Acceptance criteria - [ ] Cause of the provisioning failure identified - [ ] Cause of the DNS timeouts identified - [ ] CI runs complete without infrastructure-related failures across several consecutive runs - [ ] Composer cache verified to actually hit, so package downloads stop happening on every run

myrmidex added this to the v1.4.1 milestone 2026-08-15 09:39:11 +02:00

myrmidex added the

devops

label 2026-08-15 09:39:11 +02:00

myrmidex commented

2026-08-15 10:49:54 +02:00

Author

Owner

Investigation on the runner host

The runner is forge-runner, running forgejo-runner.service as a systemd unit
(not a container). It spawns job containers from catthehacker/ubuntu:act-latest.

What was ruled out

The runner logs nothing useful. 30 lines since the previous day, all
task N repo is ... and Cleaning up network for job .... No errors, no
warnings, and nothing at all for the 14m50s provisioning failure. The runner
did not treat it as an error, which suggests it happened inside the image pull
rather than in runner logic.

DNS is healthy at rest and under burst. Sequential lookups of
codeload.github.com returned in 100-230ms. Sixty parallel lookups all
succeeded promptly, which is a heavier burst than composer install produces.
The earlier theory that the router falls over under load does not hold in that
simple form.

What was found

The host resolves through a single nameserver with no fallback:

domain fritz.box
search fritz.box
nameserver 192.168.178.1

Job containers inherited exactly that, confirmed with
docker run --rm catthehacker/ubuntu:act-latest cat /etc/resolv.conf.

With one nameserver and no secondary, any dropped or delayed query has nothing
to fail over to. It simply waits out Composer's 10-second timeout and fails the
step. That matches the observed shape: ~118 packages download fine, then one
lookup gets no answer and the whole install dies.

Mitigation applied

/etc/docker/daemon.json created on forge-runner:

{
  "dns": ["192.168.178.1", "1.1.1.1"]
}

followed by systemctl restart docker. Verified that job containers now receive
both nameservers.

This is a mitigation, not a proven fix. Neither failure could be reproduced
on demand, and both left no trace. What it does is remove a single point of
failure: a transient non-answer from the router is now retried against a second
resolver instead of becoming a hard timeout.

Still open

The 14m50s provisioning failure has no explanation. Nothing in the runner logs
covers it, and it has not recurred. If it happens again, capture
journalctl -u forgejo-runner and journalctl -u docker around the timestamp.
The Composer cache still does not work: ci.yml caches ~/.composer/cache
while Composer 2 uses ~/.cache/composer, so all 119 packages are downloaded
on every run. The path fix is on ci/147-speed-up (#147). Landing it removes
most of the remaining exposure, because a warm cache does not make 119 DNS
lookups.

How to tell whether this worked

There is no positive test. The evidence will be an absence: no further
curl error 28 ... Resolving timed out failures over the next several runs. If
one occurs anyway, the resolver is not the cause and this ticket needs a
different line of investigation.

## Investigation on the runner host The runner is `forge-runner`, running `forgejo-runner.service` as a systemd unit (not a container). It spawns job containers from `catthehacker/ubuntu:act-latest`. ### What was ruled out **The runner logs nothing useful.** 30 lines since the previous day, all `task N repo is ...` and `Cleaning up network for job ...`. No errors, no warnings, and **nothing at all for the 14m50s provisioning failure**. The runner did not treat it as an error, which suggests it happened inside the image pull rather than in runner logic. **DNS is healthy at rest and under burst.** Sequential lookups of `codeload.github.com` returned in 100-230ms. Sixty parallel lookups all succeeded promptly, which is a heavier burst than `composer install` produces. The earlier theory that the router falls over under load does not hold in that simple form. ### What was found The host resolves through a single nameserver with no fallback: ``` domain fritz.box search fritz.box nameserver 192.168.178.1 ``` Job containers inherited exactly that, confirmed with `docker run --rm catthehacker/ubuntu:act-latest cat /etc/resolv.conf`. With one nameserver and no secondary, any dropped or delayed query has nothing to fail over to. It simply waits out Composer's 10-second timeout and fails the step. That matches the observed shape: ~118 packages download fine, then one lookup gets no answer and the whole install dies. ### Mitigation applied `/etc/docker/daemon.json` created on `forge-runner`: ```json { "dns": ["192.168.178.1", "1.1.1.1"] } ``` followed by `systemctl restart docker`. Verified that job containers now receive both nameservers. **This is a mitigation, not a proven fix.** Neither failure could be reproduced on demand, and both left no trace. What it does is remove a single point of failure: a transient non-answer from the router is now retried against a second resolver instead of becoming a hard timeout. ### Still open - The 14m50s provisioning failure has no explanation. Nothing in the runner logs covers it, and it has not recurred. If it happens again, capture `journalctl -u forgejo-runner` and `journalctl -u docker` around the timestamp. - The Composer cache still does not work: `ci.yml` caches `~/.composer/cache` while Composer 2 uses `~/.cache/composer`, so all 119 packages are downloaded on every run. The path fix is on `ci/147-speed-up` (#147). Landing it removes most of the remaining exposure, because a warm cache does not make 119 DNS lookups. ### How to tell whether this worked There is no positive test. The evidence will be an absence: no further `curl error 28 ... Resolving timed out` failures over the next several runs. If one occurs anyway, the resolver is not the cause and this ticket needs a different line of investigation.

myrmidex commented

2026-08-15 11:58:27 +02:00

Author

Owner

Closing as mitigated.

/etc/docker/daemon.json on forge-runner now gives job containers a fallback
resolver, so a dropped query from the router fails over to 1.1.1.1 instead of
waiting out Composer's 10-second timeout. Verified that containers receive both
nameservers.

Neither failure has recurred since. Several runs have passed cleanly, including
the release branch runs and the v1.4.1 PR.

Two things reduce the remaining exposure independently of the resolver change:

#147 landed, so CI no longer installs PHP or compiles pcov on every run.
That removes a large share of the network activity that was exposed to DNS
failures in the first place.
The Composer cache path is fixed, so packages should stop being fetched fresh
on every run once a cache is warm.

The 14m50s provisioning failure still has no explanation and left nothing in
journalctl -u forgejo-runner. It has not recurred. If it happens again, capture
both the runner and docker journals around the timestamp and reopen — closing
this now rather than holding the milestone open against a wait.

Closing as mitigated. `/etc/docker/daemon.json` on `forge-runner` now gives job containers a fallback resolver, so a dropped query from the router fails over to `1.1.1.1` instead of waiting out Composer's 10-second timeout. Verified that containers receive both nameservers. Neither failure has recurred since. Several runs have passed cleanly, including the release branch runs and the v1.4.1 PR. Two things reduce the remaining exposure independently of the resolver change: - #147 landed, so CI no longer installs PHP or compiles `pcov` on every run. That removes a large share of the network activity that was exposed to DNS failures in the first place. - The Composer cache path is fixed, so packages should stop being fetched fresh on every run once a cache is warm. The 14m50s provisioning failure still has no explanation and left nothing in `journalctl -u forgejo-runner`. It has not recurred. If it happens again, capture both the runner and docker journals around the timestamp and reopen — closing this now rather than holding the milestone open against a wait.

myrmidex closed this issue

2026-08-15 11:58:34 +02:00