Intermittent failure in ArticlesTest::test_expanding_a_feed_shows_all_its_articles_beyond_one_page #134

New issue

Open

opened 2026-08-11 00:39:07 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-11 00:39:07 +02:00

Owner

Summary

Tests\Feature\Livewire\ArticlesTest::test_expanding_a_feed_shows_all_its_articles_beyond_one_page errors intermittently during full-suite runs. It passes in isolation.

What was observed

Seen once in four full-suite runs on 2026-08-11, while working on #95. Reported as an error, not a failure:

Tests: 977, Assertions: 2533, Errors: 1.
/app/vendor/laravel/framework/src/Illuminate/Foundation/Testing/Concerns/InteractsWithTime.php:59
/app/tests/Feature/Livewire/ArticlesTest.php:348

Note the assertion count on the failing run (2533) is lower than on clean runs (2553) — consistent with the run aborting partway through that test.

What does not reproduce it

--filter ArticlesTest alone → 38/38 pass
--filter 'ArticlesTest|CleanupArticlesJobTest|RouteArticleRetryTest' → 48/48 pass, 3 runs in a row
Two subsequent full-suite runs → 977/977 pass

The exception message was not captured. It scrolled out of a truncated tail and could not be reproduced afterwards. Capturing it is the first task.

The suspect code

tests/Feature/Livewire/ArticlesTest.php:341-355 travels in time inside a 20-iteration loop:

$base = now()->startOfSecond();

for ($i = 0; $i < 20; $i++) {
    $this->travelTo($base->copy()->addMinutes($i), function () use ($vrt, $i) {
        $this->createRouteArticleForFeed($vrt, "VRT Article {$i}");
    });
}

$this->travelBack();

Three test files manipulate time and could interact through leaked global Carbon test state:

tests/Feature/Livewire/ArticlesTest.php
tests/Unit/Jobs/CleanupArticlesJobTest.php
tests/Unit/Models/RouteArticleRetryTest.php

A full suite takes ~100 seconds, so a second-boundary or ordering effect that only appears under specific interleaving is plausible. startOfSecond() in the test suggests a previous encounter with second-boundary flakiness.

Why this matters

A test that fails roughly one run in four makes every green run less meaningful and will eventually fail CI for no real reason — the worst kind of failure, because the reflex becomes "just re-run it".

Tasks

Reproduce reliably and capture the exception message (e.g. loop the full suite, or run with a fixed random seed / --order-by=random and record the seed)
Determine whether it is leaked Carbon test state between files, a second-boundary effect, or something else
Fix the root cause rather than adding a retry or loosening the assertion
Confirm with repeated full-suite runs

Not caused by

Unrelated to #95, which touches platform accounts only. This test covers feeds and articles and was not modified.

## Summary `Tests\Feature\Livewire\ArticlesTest::test_expanding_a_feed_shows_all_its_articles_beyond_one_page` errors intermittently during full-suite runs. It passes in isolation. ## What was observed Seen **once in four full-suite runs** on 2026-08-11, while working on #95. Reported as an **error**, not a failure: ``` Tests: 977, Assertions: 2533, Errors: 1. /app/vendor/laravel/framework/src/Illuminate/Foundation/Testing/Concerns/InteractsWithTime.php:59 /app/tests/Feature/Livewire/ArticlesTest.php:348 ``` Note the assertion count on the failing run (2533) is lower than on clean runs (2553) — consistent with the run aborting partway through that test. ## What does not reproduce it - `--filter ArticlesTest` alone → 38/38 pass - `--filter 'ArticlesTest|CleanupArticlesJobTest|RouteArticleRetryTest'` → 48/48 pass, 3 runs in a row - Two subsequent full-suite runs → 977/977 pass **The exception message was not captured.** It scrolled out of a truncated tail and could not be reproduced afterwards. Capturing it is the first task. ## The suspect code `tests/Feature/Livewire/ArticlesTest.php:341-355` travels in time inside a 20-iteration loop: ```php $base = now()->startOfSecond(); for ($i = 0; $i < 20; $i++) { $this->travelTo($base->copy()->addMinutes($i), function () use ($vrt, $i) { $this->createRouteArticleForFeed($vrt, "VRT Article {$i}"); }); } $this->travelBack(); ``` Three test files manipulate time and could interact through leaked global `Carbon` test state: - `tests/Feature/Livewire/ArticlesTest.php` - `tests/Unit/Jobs/CleanupArticlesJobTest.php` - `tests/Unit/Models/RouteArticleRetryTest.php` A full suite takes ~100 seconds, so a second-boundary or ordering effect that only appears under specific interleaving is plausible. `startOfSecond()` in the test suggests a previous encounter with second-boundary flakiness. ## Why this matters A test that fails roughly one run in four makes every green run less meaningful and will eventually fail CI for no real reason — the worst kind of failure, because the reflex becomes "just re-run it". ## Tasks - [ ] Reproduce reliably and **capture the exception message** (e.g. loop the full suite, or run with a fixed random seed / `--order-by=random` and record the seed) - [ ] Determine whether it is leaked `Carbon` test state between files, a second-boundary effect, or something else - [ ] Fix the root cause rather than adding a retry or loosening the assertion - [ ] Confirm with repeated full-suite runs ## Not caused by Unrelated to #95, which touches platform accounts only. This test covers feeds and articles and was not modified.