index
:
urls
master
Web crawler in Go.
Brandon Irizarry
summary
refs
log
tree
commit
diff
log msg
author
committer
range
path:
root
/
main.go
Age
Commit message (
Expand
)
Author
2026-06-12
refactor: streamline action-logic calling the "classic" crawler
demo
2026-06-12
chore: get rid of URL print
demo
2026-06-11
feat: set shortcode filename inside a constant
demo
2026-06-11
feat: add remaining flags and reinstate crawling logic
demo
2026-06-11
feat: implement URL conversion per case
demo
2026-06-11
feat: switch off of which flag is being used
demo
2026-06-11
feat: introduce --shortcode and --url as mutually exclusive flags
demo
2026-06-11
feat: make stubs for subcommands
demo
2026-06-11
feat: check that GOEXPERIMENT=goroutineleakprofile is set
demo
2026-05-30
feat: upgrade hamlet and use newer stuff from there
demo
2026-05-30
feat: use hamlet package to simplify command-line arguments
demo
2026-05-28
feat: specify whether URL sources are both missing or both present
demo
2026-05-28
feat: implement shortcode feature
demo
2026-05-28
refactor: move initial URL parsing into function 'convertToURL'
demo
2026-05-28
docs: add some comments
demo
2026-05-28
feat: check for missing https://
demo
2026-05-28
docs: add comment explaining purpose of log.Lshortfile
demo
2026-05-28
chore: include shortfile printout in log invocations
demo
2026-05-27
feat: configure maxDepth from the command line
demo
2026-05-27
wip: prototype a max-depth limitation
demo
2026-05-27
feat: update the classic crawler to track depth via packets
demo
2026-05-26
feat: add the worker-pool-based crawer from TGPL
demo
2026-05-26
feat: hit 'em with the classic web crawler
demo
2026-05-26
refactor: make deduplication part of main goroutine
demo
2026-05-26
feat: restore original "print 45 and hang" behavior
demo
2026-05-26
feat: change all channel payloads to pointer types
demo
2026-05-26
feat: reveal bug in the channel linkage topology
demo
2026-05-26
feat: add some code to cancel
demo
2026-05-26
refactor: use SplitSeq instead of Split
demo
2026-05-26
feat: add gouroutine-leak profiling
demo
2026-05-26
feat: design worker-pool webcrawler
demo
2026-05-26
docs: remove comment
demo
2026-05-24
refactor: move main logic into separate function
demo
2026-05-23
feat: avoid sending empty URL slices to the worklist
demo
2026-05-23
feat: add diagnostics to prove code is buggy
demo
2026-05-23
docs: add package godoc
demo
2026-05-23
docs: expound help string for -max argument
demo
2026-05-23
fix: check *maxURLs > 0 case
demo
2026-05-23
feat: add maxURLs CLI flag
demo
2026-05-23
feat: implement cancellation
demo
2026-05-23
feat: add semaphore to throttle concurrent GET requests
demo
2026-05-23
refactor: remove intermediate variable
demo
2026-05-23
docs: add comment for clarity
demo
2026-05-23
feat: implement simple BFS webcrawler
demo
2026-05-23
feat: create empty main.go
demo