urls, branch master Web crawler in Go. https://git.brandonirizarry.xyz/urls/atom?h=master 2026-05-31T03:04:48Z feat: upgrade hamlet and use newer stuff from there 2026-05-31T03:04:48Z demo demo@antix1 2026-05-31T03:04:48Z urn:sha1:462ca98bd6751ab91a8967a99f98f9553c48e7fd feat: use hamlet package to simplify command-line arguments 2026-05-31T02:43:19Z demo demo@antix1 2026-05-31T02:43:19Z urn:sha1:4f7d3203e44fca492aadb075d03b8b90b99c66c8 chore: install hamlet 2026-05-31T02:23:52Z demo demo@antix1 2026-05-31T02:23:52Z urn:sha1:edfd7022bc801c2eaa1a71a4c639f843c82dd44b This is a package I wrote to help me painlessly manage my command-line flags. feat: specify whether URL sources are both missing or both present 2026-05-28T21:32:07Z demo demo@antix1 2026-05-28T21:32:07Z urn:sha1:73f0ff9f11ff5f75a92adc5cd59f87b6fb019150 feat: implement shortcode feature 2026-05-28T21:28:59Z demo demo@antix1 2026-05-28T21:28:59Z urn:sha1:848481fd1a738225664fa398275fcd1afd61c4ff refactor: move initial URL parsing into function 'convertToURL' 2026-05-28T20:58:21Z demo demo@antix1 2026-05-28T20:58:21Z urn:sha1:0d7056446009f38ea44249f2f74ab70b05410cfb chore: add urls.csv shortcodes file 2026-05-28T20:43:29Z demo demo@antix1 2026-05-28T20:43:29Z urn:sha1:828562584138f27f4b75209dcf665a5033bd0881 This file contains convenient shortcode mappings to eliminate the need to remember URLs I scrape frequently for testing purposes. docs: add some comments 2026-05-28T20:41:53Z demo demo@antix1 2026-05-28T20:41:53Z urn:sha1:0d4e903fef12b5ed7d79cc097f3b18d2869ada29 feat: add a user agent header! 2026-05-28T20:22:41Z demo demo@antix1 2026-05-28T20:22:41Z urn:sha1:d217b6d89d03487bfa1d45c5cb750b27de8a45e5 feat: add "total entries" as part of XML comment 2026-05-28T17:32:59Z demo demo@antix1 2026-05-28T17:31:49Z urn:sha1:138d431695c22927f1b5ab76486390960f5cbb3c This is useful mostly in the context of maxURLs=∞, but perhaps it could also help catch some error cases.