* ConfigFile change to allowlist and blocklist
* revised names and warnings
* consistent file naming in kebab case, and generic use of blocklist and allowlist in cmoments for clarity
* update ci files
* impose maximum delay and document
* live update of servers
* update for source prefixes
* fixup test
* stop registerServers being called twice at startup
* prevent double registration at startup
* tidy function signature for loadSource
Co-authored-by: Ian Bashford <ianbashford@gmail.com>
Previously when the cache was written to disk, the modification time was unspecified.
At the next prefetch, it was possible for the cache to be expiring very soon (on the order of milliseconds) but still deemed valid.
Now the modification time is explicitly set to when the prefetch run began to make this situation much less likely.
This way it matches with the "loading from URL" info message and users with that log level aren't left with the false impression that it loaded fine when it really didn't.
Previously, an expired cache would be read before trying each URL until a download completed.
By moving the download loop, the cache can be read once outside the loop.
This allows a large number of tests to be enabled and pass now that the behaviour is expected.
The main fix here is that a download with an invalid signature will always fall back on using a properly signed cache, no matter how old it is.
Additionally, downloads will never be written to the cache unless they are properly signed (both at startup and prefetching).
Cache TTL is how old the cache can be at startup before trying to download an update immediately.
Prefetch delay is how long the prefetcher should wait between successful downloads of a source.
Previously, the refresh_delay configuration was used at startup as both cache TTL and prefetch delay, with subsequent prefetches using a hard-coded delay.
As discussed, refresh_delay is now only used for cache TTL, prefetch delay always uses the hard-coded delay.
Previously when constructing the signature URL, the `.minisig` suffix was blindly appended to the string version of the source URL.
Now we take the parsed source URL, deep copy it (saves us parsing it twice), and append the `.minisig` suffix to the path component of the URL.
When a list fails to download, there's no point trying to download the signature.
Code duplication moved to where it's easier to refactor away.
Enabled a few more tests.
The proxy shouldn't need to know how prefetching works, just that it needs to do it occasionally. Now the prefetching algorithm can be refactored without having to touch the proxy code.
Signatures in particular were read in from both cache and url as `[]byte`, converted to `string`, then back to `[]byte` to pass through to minisign.
Lists themselves will be converted to `string` by the parsing code anyway.
When comparing times in tests, it's necessary to control the `now` value to ensure slow test runs don't fail incorrectly.
Both cache and download code had been using refreshDelay to set the next prefetch delay, which by default meant the 1st prefetch was 3 days after startup - this has now been corrected to match the 1 day expectation.
Enabling some of the cache tests revealed some other incorrect failures in the test that were also fixed.
- a corrupt source (no names/descriptions) will still abort parsing immediately
- a duplicate, missing or invalid server stamp will not abort parsing
- these errors are logged as warnings, and returned as a single error at the end