summaryrefslogtreecommitdiff
AgeCommit message (Expand)Author
2026-06-11style: rename findURLs to extractURLsdemo
2026-06-11feat: write XML sitemaps to separate directorydemo
2026-06-11feat: check that GOEXPERIMENT=goroutineleakprofile is setdemo
2026-06-11refactor: delete 'workers' version of webcrawlerdemo
2026-06-06feat: add sqlite tutorial sitedemo
2026-05-31feat: use for-select instead of rangedemo
2026-05-30feat: add code to handle maxDepthdemo
2026-05-30feat: add example.com to shortcodes as "example"demo
2026-05-30feat: add maxDepth argument to workers versiondemo
2026-05-30feat: upgrade hamlet and use newer stuff from theredemo
2026-05-30feat: use hamlet package to simplify command-line argumentsdemo
2026-05-30chore: install hamletdemo
2026-05-28feat: specify whether URL sources are both missing or both presentdemo
2026-05-28feat: implement shortcode featuredemo
2026-05-28refactor: move initial URL parsing into function 'convertToURL'demo
2026-05-28chore: add urls.csv shortcodes filedemo
2026-05-28docs: add some commentsdemo
2026-05-28feat: add a user agent header!demo
2026-05-28feat: add "total entries" as part of XML commentdemo
2026-05-28feat: prettify stats when figure was passed in as 0demo
2026-05-28feat: place comment before URL listingdemo
2026-05-28feat: add comment logging maxDepth and maxURLs inside xml outputdemo
2026-05-28chore: ignore xml outputdemo
2026-05-28feat: save sitemap to a filedemo
2026-05-28feat: check for missing https://demo
2026-05-28feat: add header to xml outputdemo
2026-05-28wip: generate rough draft of sitemapdemo
2026-05-28docs: expand findURLs godocdemo
2026-05-28docs: add comment explaining purpose of log.Lshortfiledemo
2026-05-28refactor: move html document creation to getBatchdemo
2026-05-28chore: include shortfile printout in log invocationsdemo
2026-05-27fix: make select statement block unless communication takes placedemo
2026-05-27feat: configure maxDepth from the command linedemo
2026-05-27wip: prototype a max-depth limitationdemo
2026-05-27feat: update the classic crawler to track depth via packetsdemo
2026-05-27refactor: move packet definitions to their own filedemo
2026-05-27refactor: move "packet conversion" into a separate functiondemo
2026-05-26docs: add extensive commentsdemo
2026-05-26feat: measure the depth where each URL is founddemo
2026-05-26feat: add some prints to prove we need to select on Done()demo
2026-05-26refactor: eliminate redundant select statementdemo
2026-05-26fix: make sure all workers terminate by the enddemo
2026-05-26feat: add early termination condition based on maxURLsdemo
2026-05-26feat: add break condition from worklist loopdemo
2026-05-26feat: add the worker-pool-based crawer from TGPLdemo
2026-05-26docs: add an "Awesome Go" section to the READMEdemo
2026-05-26docs: save websites I usually use with this crawlerdemo
2026-05-26fix: release semaphore at the proper timedemo
2026-05-26feat: implement maxConcurrency using a buffered channel 'sema'demo
2026-05-26feat: add cancellation featuredemo
generated by cgit v1.2.3 (git 2.46.0) at 2026-07-22 04:18:02 +0000