urls/classic.go, branch master Web crawler in Go. https://git.brandonirizarry.xyz/urls/atom?h=master 2026-05-28T21:28:59Z feat: implement shortcode feature 2026-05-28T21:28:59Z demo demo@antix1 2026-05-28T21:28:59Z urn:sha1:848481fd1a738225664fa398275fcd1afd61c4ff feat: add comment logging maxDepth and maxURLs inside xml output 2026-05-28T17:20:30Z demo demo@antix1 2026-05-28T17:20:30Z urn:sha1:6b39b2b04e037b809537ec2b6209b62b114dceaf feat: save sitemap to a file 2026-05-28T17:16:12Z demo demo@antix1 2026-05-28T17:16:12Z urn:sha1:2a6cdae8b8ea070000e9870c1df872c2adbb48a3 wip: generate rough draft of sitemap 2026-05-28T16:33:41Z demo demo@antix1 2026-05-28T16:33:41Z urn:sha1:10ddae559ad28e21c7e1db99d29312907ce2c70f refactor: move html document creation to getBatch 2026-05-28T15:45:10Z demo demo@antix1 2026-05-28T15:45:10Z urn:sha1:2e4b6abc07036df7a001b1b8305edc55f27dda9f Also, if there are errors, I log them and simply return a nil slice. fix: make select statement block unless communication takes place 2026-05-27T16:39:56Z demo demo@antix1 2026-05-27T16:39:06Z urn:sha1:29bc8d424f865f6c9187ccda510b0ad91ee228b7 feat: configure maxDepth from the command line 2026-05-27T16:18:56Z demo demo@antix1 2026-05-27T16:18:56Z urn:sha1:a939f6c8ea4e7d493aa282403e1f39a12604484d Similar to maxURLs, a maxDepth of zero means no limit. wip: prototype a max-depth limitation 2026-05-27T16:09:31Z demo demo@antix1 2026-05-27T16:09:31Z urn:sha1:ebb7f3eb80c86ae110aaaaf73381e677c33699c1 It's just bolted on with a constant right now though. feat: update the classic crawler to track depth via packets 2026-05-27T16:05:45Z demo demo@antix1 2026-05-27T16:05:45Z urn:sha1:7852eae1e9653d9a62ff3c82b8fac2954ee1944f fix: release semaphore at the proper time 2026-05-26T22:24:16Z demo demo@antix1 2026-05-26T22:24:16Z urn:sha1:f7330300e912072fe8a9ec9ec372870f9ed83b03