| Age | Commit message (Collapse) | Author |
|---|
|
|
|
This is a package I wrote to help me painlessly manage my command-line
flags.
|
|
|
|
|
|
|
|
This file contains convenient shortcode mappings to eliminate the need
to remember URLs I scrape frequently for testing purposes.
|
|
|
|
|
|
This is useful mostly in the context of maxURLs=∞, but perhaps it
could also help catch some error cases.
|
|
The code understands 0 as "no limit", but I want to convey the
no-limit concept to readers of the file who don't have a notion of how
the program works. So I convert 0 to ∞ in the string output here.
|
|
There could be a lot of URLs, so this should be more user-friendly.
|
|
|
|
|
|
|
|
This isn't as easy as modifying the parsed URL after the fact.
This Stack Overflow post has some hints:
https://stackoverflow.com/q/46719948/4570292
|
|
|
|
|
|
|
|
|
|
Also, if there are errors, I log them and simply return a nil slice.
|
|
|
|
|
|
Similar to maxURLs, a maxDepth of zero means no limit.
|
|
It's just bolted on with a constant right now though.
|
|
|
|
I also decided to make the packet datatype package private.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This might expand into a whole journal on what sites I've tried the
crawler with, and what the results were.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
However, currently this never gets reached.
|
|
|
|
|
|
|
|
|
