summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2026-06-12docs: use proper Markdown syntaxmasterdemo
2026-06-12feat: guard against non-HTML content typesdemo
2026-06-12chore: add crawler-test.com to shortcodes as 'crawlertest'demo
2026-06-12refactor: return errors instead of using log.Fataldemo
2026-06-12refactor: move sitemap-creation to 'run' and fix missing xml dirdemo
2026-06-12feat: use getLeakProfile to check for goroutine leaks in 'classic'demo
2026-06-12refactor: move main business logic into run.godemo
2026-06-12refactor: move Action code to separate functiondemo
2026-06-12refactor: remove else-branch when deciding which flag to usedemo
2026-06-12refactor: streamline action-logic calling the "classic" crawlerdemo
2026-06-12chore: get rid of URL printdemo
2026-06-11feat: set shortcode filename inside a constantdemo
2026-06-11feat: add remaining flags and reinstate crawling logicdemo
2026-06-11feat: implement URL conversion per casedemo
2026-06-11feat: switch off of which flag is being useddemo
2026-06-11feat: introduce --shortcode and --url as mutually exclusive flagsdemo
2026-06-11feat: make stubs for subcommandsdemo
2026-06-11chore: add urfavedemo
2026-06-11style: rename findURLs to extractURLsdemo
"Find" strongly implies an HTTP GET request, so I decided to change the name.
2026-06-11feat: write XML sitemaps to separate directorydemo
2026-06-11feat: check that GOEXPERIMENT=goroutineleakprofile is setdemo
2026-06-11refactor: delete 'workers' version of webcrawlerdemo
2026-06-06feat: add sqlite tutorial sitedemo
2026-05-31feat: use for-select instead of rangedemo
2026-05-30feat: add code to handle maxDepthdemo
2026-05-30feat: add example.com to shortcodes as "example"demo
2026-05-30feat: add maxDepth argument to workers versiondemo
It's currently unused though.
2026-05-30feat: upgrade hamlet and use newer stuff from theredemo
2026-05-30feat: use hamlet package to simplify command-line argumentsdemo
2026-05-30chore: install hamletdemo
This is a package I wrote to help me painlessly manage my command-line flags.
2026-05-28feat: specify whether URL sources are both missing or both presentdemo
2026-05-28feat: implement shortcode featuredemo
2026-05-28refactor: move initial URL parsing into function 'convertToURL'demo
2026-05-28chore: add urls.csv shortcodes filedemo
This file contains convenient shortcode mappings to eliminate the need to remember URLs I scrape frequently for testing purposes.
2026-05-28docs: add some commentsdemo
2026-05-28feat: add a user agent header!demo
2026-05-28feat: add "total entries" as part of XML commentdemo
This is useful mostly in the context of maxURLs=∞, but perhaps it could also help catch some error cases.
2026-05-28feat: prettify stats when figure was passed in as 0demo
The code understands 0 as "no limit", but I want to convey the no-limit concept to readers of the file who don't have a notion of how the program works. So I convert 0 to ∞ in the string output here.
2026-05-28feat: place comment before URL listingdemo
There could be a lot of URLs, so this should be more user-friendly.
2026-05-28feat: add comment logging maxDepth and maxURLs inside xml outputdemo
2026-05-28chore: ignore xml outputdemo
2026-05-28feat: save sitemap to a filedemo
2026-05-28feat: check for missing https://demo
This isn't as easy as modifying the parsed URL after the fact. This Stack Overflow post has some hints: https://stackoverflow.com/q/46719948/4570292
2026-05-28feat: add header to xml outputdemo
2026-05-28wip: generate rough draft of sitemapdemo
2026-05-28docs: expand findURLs godocdemo
2026-05-28docs: add comment explaining purpose of log.Lshortfiledemo
2026-05-28refactor: move html document creation to getBatchdemo
Also, if there are errors, I log them and simply return a nil slice.
2026-05-28chore: include shortfile printout in log invocationsdemo
2026-05-27fix: make select statement block unless communication takes placedemo