index
:
urls
master
Web crawler in Go.
Brandon Irizarry
summary
refs
log
tree
commit
diff
log msg
author
committer
range
path:
root
/
classic.go
Age
Commit message (
Collapse
)
Author
2026-06-11
style: rename findURLs to extractURLs
demo
"Find" strongly implies an HTTP GET request, so I decided to change the name.
2026-06-11
feat: write XML sitemaps to separate directory
demo
2026-05-28
feat: implement shortcode feature
demo
2026-05-28
feat: add comment logging maxDepth and maxURLs inside xml output
demo
2026-05-28
feat: save sitemap to a file
demo
2026-05-28
wip: generate rough draft of sitemap
demo
2026-05-28
refactor: move html document creation to getBatch
demo
Also, if there are errors, I log them and simply return a nil slice.
2026-05-27
fix: make select statement block unless communication takes place
demo
2026-05-27
feat: configure maxDepth from the command line
demo
Similar to maxURLs, a maxDepth of zero means no limit.
2026-05-27
wip: prototype a max-depth limitation
demo
It's just bolted on with a constant right now though.
2026-05-27
feat: update the classic crawler to track depth via packets
demo
2026-05-26
fix: release semaphore at the proper time
demo
2026-05-26
feat: implement maxConcurrency using a buffered channel 'sema'
demo
2026-05-26
feat: add cancellation feature
demo
2026-05-26
feat: hit 'em with the classic web crawler
demo
