crawler, branch main Asynchronous webcrawler in Python. https://git.brandonirizarry.xyz/crawler/atom?h=main 2026-04-17T22:33:12Z refactor: wrap the fetching-task with the semaphore 2026-04-17T22:33:12Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-17T22:33:12Z urn:sha1:42022288cebc70e249ae0b979f500172be860d24 That is, do that instead of wrapping the code inside 'fetch'. The current approach conveys more pithily what our intention is - that 'fetch' is the entirety of the asynchronous task we want to "rate limit". chore: untrack description 2026-04-16T21:51:52Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:51:52Z urn:sha1:48a32e67c8061c6d60cc1afe1547574e34885f5c chore: add description for cgit 2026-04-16T21:49:49Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:49:49Z urn:sha1:0f02d1d7f657109acef231339a5e5a808a134325 feat: configure max_concurrency from the command line 2026-04-16T21:42:49Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:42:49Z urn:sha1:0c055dc725f0b8f1b697495884d65dc3ad16537d docs: add docstring to 'fetch' 2026-04-16T21:36:54Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:36:54Z urn:sha1:528f5c261ec6dc2d0090930d4e79a8fef72d3cbd feat: use a semaphore to limit how many tasks make requests 2026-04-16T21:34:36Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:34:36Z urn:sha1:28bb531bd6eec1a9139944fd5b5d1d05a08becd2 feat: use taskgroup + flush printing of '.' + catch timeouterror 2026-04-16T21:25:43Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:25:43Z urn:sha1:a6468fc6a28adc628e64c09ef1e953b8eaf0807a feat: use full concurrency (no semaphores) 2026-04-16T21:04:45Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T21:04:45Z urn:sha1:aa716390836e5e58d9bf3987b24be3f2a4110245 feat: add a timeout for sites that don't respond 2026-04-16T20:36:52Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T20:36:52Z urn:sha1:e771a3ebcee9a7c6c3efd08fa905cbe0591131b1 This should diminish the urge for reaching for Ctrl+C. feat: ping sites sequentially and measure how long it takes 2026-04-16T20:34:00Z Brandon C. Irizarry brandon.irizarry@gmail.com 2026-04-16T20:34:00Z urn:sha1:172bccd05779240d6d6cd087a1c19216f0a5e225