Factories generate colliding URLs, failing tests at random #152

New issue

Closed

opened 2026-08-15 09:29:54 +02:00 by myrmidex · 0 comments

myrmidex commented

2026-08-15 09:29:54 +02:00

Owner

Problem

Several factories generate values with bare faker calls against columns that
carry unique constraints. Faker draws from a small pool, so duplicates occur at
random and fail whichever test happens to hit one.

Factory	Field	Constraint
`FeedFactory`	`url` => `faker->url()`	`unique('url')` on `feeds`
`PlatformInstanceFactory`	`url` => `faker->url()`	`unique(['platform', 'url'])`
`PlatformInstanceFactory`	`lemmy()` state url	same
`PlatformAccountFactory`	`username` => `faker->userName()`	`unique(['username', 'platform', 'is_active'])`

PlatformInstanceFactory is the worst of these because every row it creates has
platform => 'lemmy', so the composite constraint is effectively a constraint
on url alone.

Observed

Two separate failures, hours apart, from the same cause:

UNIQUE constraint failed: platform_instances.platform, platform_instances.url
  at tests/Unit/Jobs/PublishNextArticleJobTest.php:582

UNIQUE constraint failed: feeds.url
  at tests/Unit/Jobs/PublishNextArticleJobTest.php:278
Tests: 1 failed, 1230 passed

Why one collision can fail the whole suite

Under RefreshDatabase, a test that dies mid-transaction leaves the connection
in a state where the next test cannot open a clean one:

PDOException: There is already an active transaction
  at Illuminate/Foundation/Testing/RefreshDatabase.php:129

Every subsequent test then fails during setup rather than on its own logic. One
CI run produced 809 errors from a single root collision. The error count is
not a measure of how much is broken.

Fix, and the complication

$this->faker->unique() on the four fields above resolves the collisions.

Applying it is not sufficient on its own. With unique() on
PlatformInstanceFactory, these tests in PublishNextArticleJobTest start
failing:

test_handle_skips_publishing_when_daily_cap_reached
test_handle_publishes_when_below_daily_cap
test_handle_publishes_when_daily_cap_is_zero

They call ArticlePublication::factory()->count(50)->create(...) and then
assert against a daily publish cap. Today the colliding URLs mean those 50
publications resolve to far fewer distinct instances and channels than 50, and
the assertions depend on that reduced number. Making instances unique changes
the count and the expectations no longer hold.

So those tests currently pass because of a factory bug, not in spite of it.
Their setup needs to state explicitly how many publications should count toward
the cap, rather than inheriting whatever number collisions happen to produce.

Acceptance criteria

The four fields above generate unique values
The daily-cap tests assert against an explicit, stated number of
publications rather than relying on factory collisions
The full suite passes on repeated consecutive runs, not just once
No test's outcome depends on how many distinct records a factory happens
to produce

Notes

The repository has fixed this class of problem before: Fix flaky test from duplicate Article factory URLs and Fix flaky boundary test with freezeTime.
Worth checking whether other factories have the same shape while this is open.

## Problem Several factories generate values with bare faker calls against columns that carry unique constraints. Faker draws from a small pool, so duplicates occur at random and fail whichever test happens to hit one. | Factory | Field | Constraint | |---|---|---| | `FeedFactory` | `url` => `faker->url()` | `unique('url')` on `feeds` | | `PlatformInstanceFactory` | `url` => `faker->url()` | `unique(['platform', 'url'])` | | `PlatformInstanceFactory` | `lemmy()` state url | same | | `PlatformAccountFactory` | `username` => `faker->userName()` | `unique(['username', 'platform', 'is_active'])` | `PlatformInstanceFactory` is the worst of these because every row it creates has `platform => 'lemmy'`, so the composite constraint is effectively a constraint on `url` alone. ## Observed Two separate failures, hours apart, from the same cause: ``` UNIQUE constraint failed: platform_instances.platform, platform_instances.url at tests/Unit/Jobs/PublishNextArticleJobTest.php:582 ``` ``` UNIQUE constraint failed: feeds.url at tests/Unit/Jobs/PublishNextArticleJobTest.php:278 Tests: 1 failed, 1230 passed ``` ## Why one collision can fail the whole suite Under `RefreshDatabase`, a test that dies mid-transaction leaves the connection in a state where the next test cannot open a clean one: ``` PDOException: There is already an active transaction at Illuminate/Foundation/Testing/RefreshDatabase.php:129 ``` Every subsequent test then fails during setup rather than on its own logic. One CI run produced **809 errors from a single root collision**. The error count is not a measure of how much is broken. ## Fix, and the complication `$this->faker->unique()` on the four fields above resolves the collisions. Applying it is not sufficient on its own. With `unique()` on `PlatformInstanceFactory`, these tests in `PublishNextArticleJobTest` start failing: - `test_handle_skips_publishing_when_daily_cap_reached` - `test_handle_publishes_when_below_daily_cap` - `test_handle_publishes_when_daily_cap_is_zero` They call `ArticlePublication::factory()->count(50)->create(...)` and then assert against a daily publish cap. Today the colliding URLs mean those 50 publications resolve to far fewer distinct instances and channels than 50, and the assertions depend on that reduced number. Making instances unique changes the count and the expectations no longer hold. So those tests currently pass because of a factory bug, not in spite of it. Their setup needs to state explicitly how many publications should count toward the cap, rather than inheriting whatever number collisions happen to produce. ## Acceptance criteria - [ ] The four fields above generate unique values - [ ] The daily-cap tests assert against an explicit, stated number of publications rather than relying on factory collisions - [ ] The full suite passes on repeated consecutive runs, not just once - [ ] No test's outcome depends on how many distinct records a factory happens to produce ## Notes The repository has fixed this class of problem before: `Fix flaky test from duplicate Article factory URLs` and `Fix flaky boundary test with freezeTime`. Worth checking whether other factories have the same shape while this is open.