crawler - Asynchronous webcrawler in Python.

Age	Commit message (Collapse)	Author
40 hours	refactor: wrap the fetching-task with the semaphoreHEAD main	Brandon C. Irizarry
	That is, do that instead of wrapping the code inside 'fetch'. The current approach conveys more pithily what our intention is - that 'fetch' is the entirety of the asynchronous task we want to "rate limit".
3 days	feat: configure max_concurrency from the command line	Brandon C. Irizarry

3 days	docs: add docstring to 'fetch'	Brandon C. Irizarry

3 days	feat: use a semaphore to limit how many tasks make requests	Brandon C. Irizarry

3 days	feat: use taskgroup + flush printing of '.' + catch timeouterror	Brandon C. Irizarry

3 days	feat: use full concurrency (no semaphores)	Brandon C. Irizarry

3 days	feat: add a timeout for sites that don't respond	Brandon C. Irizarry
	This should diminish the urge for reaching for Ctrl+C.
3 days	feat: ping sites sequentially and measure how long it takes	Brandon C. Irizarry

3 days	docs: improve docstring	Brandon C. Irizarry
	That is, make it more accurate and specific.
3 days	feat: read N URLs via command-line parameter from CSV file	Brandon C. Irizarry

3 days	chore: start project	Brandon C. Irizarry