Commit graph

21 commits

Author SHA1 Message Date
cb83b0df90 13 - Add language and languageConfidence fields to FetchResult 2026-04-28 00:38:46 +02:00
829ce04282 13 - Add LanguageDetectionService wrapping patrickschur/language-detection 2026-04-28 00:32:54 +02:00
a37b1da145 13 - Add language_confidence column and Page model cast 2026-04-28 00:13:14 +02:00
cda1414cd8 9 - Add robots.txt handling with cache and politeness integration
Some checks failed
CI / ci (push) Failing after 3h0m1s
CI / ci (pull_request) Has been cancelled
2026-04-27 23:53:52 +02:00
264180cd36 chore - Move outcome → status mapping into CrawlOutcomeEnum methods
Some checks failed
CI / ci (push) Failing after 3h0m1s
2026-04-27 01:36:37 +02:00
7171348370 11 - Add PolitenessService and crawler delay config 2026-04-27 00:59:45 +02:00
720e4bcc1f 14 - Implement ProcessCrawlJob orchestration with retry logic 2026-04-26 23:50:57 +02:00
118de0023a 14 - Simplify page_crawls schema (queue moves to Redis) 2026-04-26 20:58:07 +02:00
649aeb3627 chore - Extract RegisterDiscoveredPageAction for shared Page::firstOrCreate logic 2026-04-26 20:18:18 +02:00
bb7906e193 12 - Make FetchResult fields nullable and add missing crawler config 2026-04-26 16:50:43 +02:00
a9f2d689ae 12 - Add crawler config and FetchResult value object 2026-04-26 16:45:07 +02:00
abbcedf2e7 12 - Add Rejected case to CrawlOutcomeEnum and PageStatusEnum 2026-04-26 16:35:46 +02:00
6f75be7328 8 - Tighten UrlService validation and add observer integration tests 2026-04-26 16:09:28 +02:00
de14ae3ad4 8 - Wire PageObserver to enqueue page_crawls on Page creation 2026-04-26 15:56:38 +02:00
81209125a1 8 - Add UrlService with host extraction method 2026-04-26 14:52:40 +02:00
f2c1fab4e4 7 - Add int casts on PageCrawl and tests for cascade-delete + pending scope 2026-04-26 14:23:13 +02:00
fe8ca7fc10 7 - Add page_crawls migration, PageCrawl model, factory, and Page relationships 2026-04-26 14:15:49 +02:00
9dd6d84d65 7 - Add CrawlOutcomeEnum for crawl attempt outcomes 2026-04-26 13:06:22 +02:00
b1b7adeacd 7 - Add language column to pages for crawler-detected language 2026-04-26 12:53:21 +02:00
424ad2ff78 4 - Add Page and PageLink models with factories and unit tests 2026-04-26 02:51:49 +02:00
1fe6ae5cff 1 - Install Laravel 13 with Livewire 2026-04-23 03:13:33 +02:00