Commit Graph

1077 Commits

Author SHA1 Message Date
William Elwood c0e34d1a9e Verify signature immediately after reading from cache or URL
This allows a large number of tests to be enabled and pass now that the behaviour is expected.
The main fix here is that a download with an invalid signature will always fall back on using a properly signed cache, no matter how old it is.
Additionally, downloads will never be written to the cache unless they are properly signed (both at startup and prefetching).
2019-11-08 10:17:12 +01:00
William Elwood 53d5b0f3cd Remove URLToPrefetch struct 2019-11-08 10:17:12 +01:00
William Elwood a83ecf626b Move `when` from URLToPrefetch struct to `refresh` on Source struct 2019-11-08 10:17:12 +01:00
William Elwood b29c70551e Refactor writing download to cache 2019-11-08 10:17:12 +01:00
William Elwood 082a4a5e01 Clarify how refresh_delay is intended to be used internally
Cache TTL is how old the cache can be at startup before trying to download an update immediately.
Prefetch delay is how long the prefetcher should wait between successful downloads of a source.
Previously, the refresh_delay configuration was used at startup as both cache TTL and prefetch delay, with subsequent prefetches using a hard-coded delay.
As discussed, refresh_delay is now only used for cache TTL, prefetch delay always uses the hard-coded delay.
2019-11-08 10:17:12 +01:00
William Elwood 190700e5ba Move cacheFile from URLToPrefetch to Source struct 2019-11-08 10:17:12 +01:00
William Elwood 0991749b19 Allow source URLs to contain query parameters
Previously when constructing the signature URL, the `.minisig` suffix was blindly appended to the string version of the source URL.
Now we take the parsed source URL, deep copy it (saves us parsing it twice), and append the `.minisig` suffix to the path component of the URL.
2019-11-08 10:17:12 +01:00
William Elwood b2ecc45133 Treat each list and signature pair as a single unit
When a list fails to download, there's no point trying to download the signature.
Code duplication moved to where it's easier to refactor away.
Enabled a few more tests.
2019-11-08 10:17:12 +01:00
William Elwood 1e225dbb67 Alter source tests to cover entire prefetch algorithm and make it pass 2019-11-08 10:17:12 +01:00
William Elwood 7e73a26a2f Move most of the prefetching code into sources.go
The proxy shouldn't need to know how prefetching works, just that it needs to do it occasionally. Now the prefetching algorithm can be refactored without having to touch the proxy code.
2019-11-08 10:17:12 +01:00
William Elwood 78f2dead79 Move prefetch URLs onto Source struct
This is mostly in preparation for further refactoring, but does reduce the number of return values from `NewSource()` too.
2019-11-08 10:17:12 +01:00
William Elwood 4a792026eb Refactor cache reading to reduce number of return values 2019-11-08 10:17:12 +01:00
William Elwood fe34d07b68 Refactor away some unnecessary type shuffling
Signatures in particular were read in from both cache and url as `[]byte`, converted to `string`, then back to `[]byte` to pass through to minisign.
Lists themselves will be converted to `string` by the parsing code anyway.
2019-11-08 10:17:12 +01:00
William Elwood ad92be5b9c Refactor saving downloads to cache
Moved writing to happen immediately after the download to reduce duplicated code and number of return values from the download function.
2019-11-08 10:17:12 +01:00
William Elwood 4c156784c8 Refactor calculation of update delay when reading cache
Set the default delay once at the top instead of before every early return.
2019-11-08 10:17:12 +01:00
William Elwood e818eeb800 Refactor reading a URL's content to own function
No longer shadows `url` package with variable of the same name.
2019-11-08 10:17:12 +01:00
William Elwood d851c9eeb6 Refactor signature verification to own function
The cache is no longer destroyed whenever any signature verification fails.
The public key is stored on the Source struct for future use.
2019-11-08 10:17:12 +01:00
William Elwood da0d7fe841 Fix various timing inconsistencies
When comparing times in tests, it's necessary to control the `now` value to ensure slow test runs don't fail incorrectly.
Both cache and download code had been using refreshDelay to set the next prefetch delay, which by default meant the 1st prefetch was 3 days after startup - this has now been corrected to match the 1 day expectation.
Enabling some of the cache tests revealed some other incorrect failures in the test that were also fixed.
2019-11-08 10:17:12 +01:00
William Elwood 03dea47130 Remove dead code paths
These paths were unreachable because XTransport.Get already checks the same conditions.
2019-11-08 10:17:12 +01:00
William Elwood af0629856c Add unit tests for sources.go
Tests cover most of the cache and download related code paths and specify the expected result of various starting states and external failure modes.
Where the current code's behaviour doesn't match a test's expectations, the test is disabled and annotated with a TODO until it can be fixed.
Added dependency on `github.com/powerman/check` and ran `go mod vendor`.
2019-11-08 10:17:12 +01:00
William Elwood 503bfb877b go mod tidy
```console
$ go mod tidy -v
(snip)
unused github.com/agl/ed25519
```
Also add base .gitattributes file to normalize line endings in the repository across differing developer environments.
2019-11-08 10:17:12 +01:00
Frank Denis e6a4a4ffda Update deps 2019-11-05 01:32:38 +01:00
Frank Denis b76db70a6c Revert "cleanup: estimators: simplify blindAdjust"
This reverts commit c699e7bec4.
2019-11-05 01:16:22 +01:00
Frank Denis 17a675021e No one ever completes // TODO things 2019-11-05 01:10:57 +01:00
Frank Denis da3f30871f Revert "fix: proxy: Trigger query logging plugins using defer"
This reverts commit fc9509a8c8.
2019-11-05 00:54:03 +01:00
Frank Denis 14862c2fc7 defer is slow and not worth it here 2019-11-05 00:37:46 +01:00
Frank Denis 316c5ca6b1 Don't return immediately on non-Windows system if netprobe_timout is -1
Fixes #1016
2019-11-04 17:14:31 +01:00
Frank Denis 9852a289f8 Increase the default cache size and minimum TTL 2019-11-03 17:31:41 +01:00
Frank Denis e0c37f92fc Add a comment about why DoH addresses from stamps don't expire 2019-11-03 00:33:17 +01:00
Frank Denis 0f332c644d Set a minimum TTL when caching resolver IPs
Comcast having a 30 sec TTL is silly
2019-11-02 02:01:03 +01:00
Frank Denis 63ed3b4fef Update comment 2019-11-02 01:52:51 +01:00
Frank Denis a84a789a8a Keep resolving if needed 2019-11-02 01:50:35 +01:00
Frank Denis d932d5fdfc Inverse test 2019-11-02 01:20:28 +01:00
Frank Denis 6032c3b79b Add a grace TTL for expired cached IPs
And some comments to make the code more readable
2019-11-01 23:19:07 +01:00
Frank Denis 0dc69eacd5 resolveHostWithCache -> resolveWithCache 2019-11-01 23:10:36 +01:00
Frank Denis b30904f20b lowercase 2019-11-01 23:06:42 +01:00
Frank Denis 8d191cdcf1 Rename CheckResolver to IsIPAndPort for clarity 2019-11-01 23:05:17 +01:00
Frank Denis 3cef651b07 Rename resolveHost() to resolveHostWithCache() for clarity
(but to be honest, I don't understand anything to that code any more)
2019-11-01 23:00:39 +01:00
Frank Denis e028f4d483 Don't delete cached server IP addresses
If we can't update an entry, keep the previous one.
2019-11-01 22:55:06 +01:00
Frank Denis 3db3de0a91 Use SystemResolverTTL as a minimum timeout for cached resolver IPs 2019-11-01 21:47:13 +01:00
Alison Winters 97e4c44223 remove err return values that are never set 2019-11-01 17:13:14 +01:00
Alison Winters 36808cdec7 remove unused patternType return 2019-11-01 17:13:14 +01:00
Alison Winters a0d9412a25 run goimports 2019-11-01 17:13:14 +01:00
Will Elwood cd675913b2
Fix copy-paste oversight
I think these variables are always both nil or both not nil, but maybe in the future they might not be.
2019-11-01 09:07:57 +00:00
Alison Winters 816acb9d8d move ConfigLoad into AppMain 2019-10-31 18:55:44 +01:00
Alison Winters 2f7e057996 move flags parsing into main() 2019-10-31 18:55:44 +01:00
Frank Denis 116f985b96 Bump 2019-10-31 18:12:13 +01:00
Frank Denis c5bda9e2ae Update deps 2019-10-31 18:00:44 +01:00
Frank Denis 59c3d5121d Add brackets around cached IPv6 IP addresses
Fixes #1005
2019-10-31 17:55:54 +01:00
Eric Lagergren 1c9924e055 check error that was being erroneously shadowed 2019-10-31 17:55:26 +01:00