index
:
urls
master
Web crawler in Go.
Brandon Irizarry
summary
refs
log
tree
commit
diff
log msg
author
committer
range
Age
Commit message (
Expand
)
Author
2026-06-12
docs: use proper Markdown syntax
master
demo
2026-06-12
feat: guard against non-HTML content types
demo
2026-06-12
chore: add crawler-test.com to shortcodes as 'crawlertest'
demo
2026-06-12
refactor: return errors instead of using log.Fatal
demo
2026-06-12
refactor: move sitemap-creation to 'run' and fix missing xml dir
demo
2026-06-12
feat: use getLeakProfile to check for goroutine leaks in 'classic'
demo
2026-06-12
refactor: move main business logic into run.go
demo
2026-06-12
refactor: move Action code to separate function
demo
2026-06-12
refactor: remove else-branch when deciding which flag to use
demo
2026-06-12
refactor: streamline action-logic calling the "classic" crawler
demo
2026-06-12
chore: get rid of URL print
demo
2026-06-11
feat: set shortcode filename inside a constant
demo
2026-06-11
feat: add remaining flags and reinstate crawling logic
demo
2026-06-11
feat: implement URL conversion per case
demo
2026-06-11
feat: switch off of which flag is being used
demo
2026-06-11
feat: introduce --shortcode and --url as mutually exclusive flags
demo
2026-06-11
feat: make stubs for subcommands
demo
2026-06-11
chore: add urfave
demo
2026-06-11
style: rename findURLs to extractURLs
demo
2026-06-11
feat: write XML sitemaps to separate directory
demo
2026-06-11
feat: check that GOEXPERIMENT=goroutineleakprofile is set
demo
2026-06-11
refactor: delete 'workers' version of webcrawler
demo
2026-06-06
feat: add sqlite tutorial site
demo
2026-05-31
feat: use for-select instead of range
demo
2026-05-30
feat: add code to handle maxDepth
demo
2026-05-30
feat: add example.com to shortcodes as "example"
demo
2026-05-30
feat: add maxDepth argument to workers version
demo
2026-05-30
feat: upgrade hamlet and use newer stuff from there
demo
2026-05-30
feat: use hamlet package to simplify command-line arguments
demo
2026-05-30
chore: install hamlet
demo
2026-05-28
feat: specify whether URL sources are both missing or both present
demo
2026-05-28
feat: implement shortcode feature
demo
2026-05-28
refactor: move initial URL parsing into function 'convertToURL'
demo
2026-05-28
chore: add urls.csv shortcodes file
demo
2026-05-28
docs: add some comments
demo
2026-05-28
feat: add a user agent header!
demo
2026-05-28
feat: add "total entries" as part of XML comment
demo
2026-05-28
feat: prettify stats when figure was passed in as 0
demo
2026-05-28
feat: place comment before URL listing
demo
2026-05-28
feat: add comment logging maxDepth and maxURLs inside xml output
demo
2026-05-28
chore: ignore xml output
demo
2026-05-28
feat: save sitemap to a file
demo
2026-05-28
feat: check for missing https://
demo
2026-05-28
feat: add header to xml output
demo
2026-05-28
wip: generate rough draft of sitemap
demo
2026-05-28
docs: expand findURLs godoc
demo
2026-05-28
docs: add comment explaining purpose of log.Lshortfile
demo
2026-05-28
refactor: move html document creation to getBatch
demo
2026-05-28
chore: include shortfile printout in log invocations
demo
2026-05-27
fix: make select statement block unless communication takes place
demo
[next]