urls/workers.go, branch masterWeb crawler in Go.
https://git.brandonirizarry.xyz/urls/atom?h=master2026-05-27T15:59:40Zrefactor: move packet definitions to their own file2026-05-27T15:59:40Zdemodemo@antix12026-05-27T15:59:08Zurn:sha1:81ee780aca0aa0d9ce6999a23ef94e986307e060
I also decided to make the packet datatype package private.
refactor: move "packet conversion" into a separate function2026-05-27T15:56:57Zdemodemo@antix12026-05-27T15:56:57Zurn:sha1:1190174edda07fea3f956b58a782eaff2d2213b8docs: add extensive comments2026-05-27T02:46:39Zdemodemo@antix12026-05-27T02:32:58Zurn:sha1:26f5b43a82955c77ea4bc1d7a710895e4b36209afeat: measure the depth where each URL is found2026-05-27T02:22:05Zdemodemo@antix12026-05-27T02:22:05Zurn:sha1:99111bcfd5b81ca51f102b35efe8e4be0e6d390efeat: add some prints to prove we need to select on Done()2026-05-27T02:05:21Zdemodemo@antix12026-05-27T02:04:48Zurn:sha1:5898a16cf4b9dc2f0eef2bf81d429b5aca7005fcrefactor: eliminate redundant select statement2026-05-27T02:00:19Zdemodemo@antix12026-05-27T02:00:19Zurn:sha1:dec73ce1d6d0abf31861cae9dd7965d86dc7496ffix: make sure all workers terminate by the end2026-05-27T01:59:10Zdemodemo@antix12026-05-27T01:59:10Zurn:sha1:d7beae73c248e5601e0fce78dca0b65cb6abd066feat: add early termination condition based on maxURLs2026-05-27T01:26:18Zdemodemo@antix12026-05-27T01:24:42Zurn:sha1:00f30b53a746d45586c59773614f4c780880a788feat: add break condition from worklist loop2026-05-27T01:22:14Zdemodemo@antix12026-05-27T01:22:14Zurn:sha1:487a3877000e94c6ac85a198195ae36582dff3c1feat: add the worker-pool-based crawer from TGPL2026-05-27T01:11:01Zdemodemo@antix12026-05-27T01:11:01Zurn:sha1:1e7d34bb14057c79ea1f7fa75e6ecf29951b69aa