urls - Web crawler in Go.

Age	Commit message (Collapse)	Author
6 days	feat: upgrade hamlet and use newer stuff from theremaster	demo

6 days	feat: use hamlet package to simplify command-line arguments	demo

8 days	feat: specify whether URL sources are both missing or both present	demo

8 days	feat: implement shortcode feature	demo

8 days	refactor: move initial URL parsing into function 'convertToURL'	demo

8 days	docs: add some comments	demo

8 days	feat: check for missing https://	demo
	This isn't as easy as modifying the parsed URL after the fact. This Stack Overflow post has some hints: https://stackoverflow.com/q/46719948/4570292
8 days	docs: add comment explaining purpose of log.Lshortfile	demo

8 days	chore: include shortfile printout in log invocations	demo

9 days	feat: configure maxDepth from the command line	demo
	Similar to maxURLs, a maxDepth of zero means no limit.
9 days	wip: prototype a max-depth limitation	demo
	It's just bolted on with a constant right now though.
9 days	feat: update the classic crawler to track depth via packets	demo

10 days	feat: add the worker-pool-based crawer from TGPL	demo

10 days	feat: hit 'em with the classic web crawler	demo

10 days	refactor: make deduplication part of main goroutine	demo

10 days	feat: restore original "print 45 and hang" behavior	demo

10 days	feat: change all channel payloads to pointer types	demo

10 days	feat: reveal bug in the channel linkage topology	demo

10 days	feat: add some code to cancel	demo
	However, currently this never gets reached.
10 days	refactor: use SplitSeq instead of Split	demo

10 days	feat: add gouroutine-leak profiling	demo

10 days	feat: design worker-pool webcrawler	demo

10 days	docs: remove comment	demo

12 days	refactor: move main logic into separate function	demo

13 days	feat: avoid sending empty URL slices to the worklist	demo

13 days	feat: add diagnostics to prove code is buggy	demo

13 days	docs: add package godoc	demo

13 days	docs: expound help string for -max argument	demo

13 days	fix: check *maxURLs > 0 case	demo

13 days	feat: add maxURLs CLI flag	demo
	I've also added some more input sanitization.
13 days	feat: implement cancellation	demo
	I've also added 1., 2., etc. to the printed list of URLs.
13 days	feat: add semaphore to throttle concurrent GET requests	demo

13 days	refactor: remove intermediate variable	demo

13 days	docs: add comment for clarity	demo

13 days	feat: implement simple BFS webcrawler	demo

13 days	feat: create empty main.go	demo