lvl0/trove

Release v0.1.0 #30

Closed

myrmidex wants to merge 0 commits from release/0.1.0 into main

myrmidex commented

2026-04-29 23:27:01 +02:00

Owner

Summary

Crawler stack: schema, models, HTTP fetcher, content extraction (Readability), queue worker, per-domain politeness gate, robots.txt handling, retry logic
Fediverse polling: Mastodon + Lemmy clients, URL discovery via events, structured success logging
Language detection: patrickschur/language-detection with HTML lang attr fallback, sticky across re-crawls
Observability: per-instance admin dashboard, ntfy alerting on repeated poll failures via PollFailed event chain
URL submission: Livewire form with rate limiting
Bot info page (/bot) with crawler identity and robots.txt opt-out instructions
Blade layout refactor: <x-layout> → @extends('layouts.app')

Tickets

Closes all 14 issues in the v0.1 milestone: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14

Test plan

CI passes (Pint + PHPStan + 217 tests / 545 assertions)
Docker image builds on merge to main
/submit form accepts and deduplicates URLs
/bot page renders with correct UA and robots.txt example
/admin/instances shows per-instance last poll, URL count, error count
fedi-discover:poll runs without error against a configured instance
ntfy notification received after NTFY_THRESHOLD consecutive poll failures

## Summary - Crawler stack: schema, models, HTTP fetcher, content extraction (Readability), queue worker, per-domain politeness gate, robots.txt handling, retry logic - Fediverse polling: Mastodon + Lemmy clients, URL discovery via events, structured success logging - Language detection: `patrickschur/language-detection` with HTML `lang` attr fallback, sticky across re-crawls - Observability: per-instance admin dashboard, ntfy alerting on repeated poll failures via `PollFailed` event chain - URL submission: Livewire form with rate limiting - Bot info page (`/bot`) with crawler identity and robots.txt opt-out instructions - Blade layout refactor: `<x-layout>` → `@extends('layouts.app')` ## Tickets Closes all 14 issues in the v0.1 milestone: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 ## Test plan - [ ] CI passes (Pint + PHPStan + 217 tests / 545 assertions) - [ ] Docker image builds on merge to main - [ ] `/submit` form accepts and deduplicates URLs - [ ] `/bot` page renders with correct UA and robots.txt example - [ ] `/admin/instances` shows per-instance last poll, URL count, error count - [ ] `fedi-discover:poll` runs without error against a configured instance - [ ] ntfy notification received after `NTFY_THRESHOLD` consecutive poll failures

myrmidex added this to the v0.1 milestone 2026-04-29 23:27:01 +02:00

myrmidex self-assigned this 2026-04-29 23:27:01 +02:00

myrmidex added 55 commits 2026-04-29 23:27:01 +02:00

1 - Add Nix shell and Docker dev environment 541cc8a3e7

1 - Install Laravel 13 with Livewire 1fe6ae5cff

1 - Add FediDiscover package skeleton 9929a837fe

1 - Fix npm install skipped on first boot due to volume-mounted node_modules 22680d45a7

1 - Add Larastan static analysis + Package test suite 47375e2b4d

1 - Add CI and build workflows for Forgejo Actions fa85decccb

1 - Add production Dockerfile f0a8bdc1de

1 - Add README with deployment docs 6c643373f2

2 - Populate fedi-discover config with http settings and defaults 3706a81d3c

2 - Add fedi_discover_instances migration 00e28c4868

chore - Set Pint concat_space to spaced + reformat 7a2db5a14d

2 - Add InstanceConfig value object and InstanceType enum bdd2b0f2e5

2 - Add Instance Eloquent model with factory fc1c8ba020

2 - Add fedi-discover:validate console command 52d6b493cb

3 - Add UrlDiscovered event 3eff919945

3 - Add PollFediverseAction with FediverseClient interface e5ee0184b5

3 - Add MastodonClient with HTTP-faked tests fea8d48f6e

3 - Add fedi-discover:poll command with failure isolation 1b652752e1

3 - Add LemmyClient with FediverseClientFactory dispatch 1b713e3539

3 - Schedule fedi-discover:poll every minute 2cb86f3337

3 - Harden fediverse polling: timeouts, error handling, payload fields ec2113710a

4 - Add pages and page_links migrations with PageStatusEnum bc535c8c0b

4 - Add Page and PageLink models with factories and unit tests 424ad2ff78

4 - Add UrlDiscoveredListener wiring fediverse polling to pages graph 3ad473f4a1

4 - Drop status promotion in UrlDiscoveredListener; defer to keywords listener

CI / ci (push) Failing after 3h0m0s

Details

5 - Publish Livewire config with class-based components and no emoji 0f14c66c34

5 - Trust forwarded headers behind reverse proxy for real client IP b0a4102637

5 - Add UrlSubmissionForm Livewire component with rate limiting 43837a99db

7 - Add language column to pages for crawler-detected language b1b7adeacd

7 - Add CrawlOutcomeEnum for crawl attempt outcomes 9dd6d84d65

7 - Add page_crawls migration, PageCrawl model, factory, and Page relationships fe8ca7fc10

7 - Add int casts on PageCrawl and tests for cascade-delete + pending scope f2c1fab4e4

8 - Add UrlService with host extraction method 81209125a1

8 - Wire PageObserver to enqueue page_crawls on Page creation de14ae3ad4

8 - Tighten UrlService validation and add observer integration tests 6f75be7328

12 - Add Rejected case to CrawlOutcomeEnum and PageStatusEnum abbcedf2e7

12 - Add crawler config and FetchResult value object a9f2d689ae

12 - Make FetchResult fields nullable and add missing crawler config bb7906e193

12 - Add FetchPageAction with Http::fake-driven outcome paths 1b7fbbfd0c

chore - Add dev-composer command 3e2fd0d2c4

12 - Add HTML content extraction (title, text, links, word count) 35e1147823

12 - Apply pr-reviewer follow-ups: validation, link filters, readonly VO, docs dda5b0f770

chore - Extract RegisterDiscoveredPageAction for shared Page::firstOrCreate logic 649aeb3627

chore - Simplify call-site tests now that RegisterDiscoveredPageAction is unit-tested 6c0e1fe12d

14 - Simplify page_crawls schema (queue moves to Redis) 118de0023a

14 - Add PageCrawlObserver and ProcessCrawlJob skeleton 2a586ecac4

14 - Implement ProcessCrawlJob orchestration with retry logic 720e4bcc1f

14 - Document queue worker container in deployment README e8a935ea31

14 - Fix ProcessCrawlJob outcome write and status mapping bugs 3297c4bb3b

chore - Extract mockFetchPageAction helper in ProcessCrawlJobTest c80be24e6e

10 - Add /bot page with crawler identity and opt-out instructions 69aa5d9d3e

11 - Add PolitenessService and crawler delay config 7171348370

11 - Gate ProcessCrawlJob with per-domain politeness lock 1538ceeb6e

chore - Move outcome → status mapping into CrawlOutcomeEnum methods

CI / ci (push) Failing after 3h0m1s

Details

9 - Add robots.txt handling with cache and politeness integration

CI / ci (push) Failing after 3h0m1s

Details

CI / ci (pull_request) Has been cancelled

Details

myrmidex closed this pull request

2026-04-29 23:58:51 +02:00

CI / ci (push) Failing after 3h0m1s

Details

CI / ci (pull_request) Has been cancelled

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lvl0/trove#30

No description provided.