Without artifact streaming, a chart was fetched with no expected
digest, committed to the cache, and only then compared with the
index's digest by serveChart, which cleared it again. Until then the
unverified chart was a cache entry like any other.
The index digest is now always passed to the fetch, so a mismatching
chart is discarded before it is committed, as it already was when
streaming. serveChart's check stays for charts that were cached before.
Manifests and digest-addressed artifacts were only checked against
sha256 digests. A sha512 digest, which the OCI spec also allows, or a
digest with any other algorithm was fetched and cached unchecked: a
manifest requested as sha512:<hex>, or a tag whose Docker-Content-Digest
header named one, was stored under that digest whatever the body was.
Manifests are now checked against any digest in the reference or the
Docker-Content-Digest header with the digest's own algorithm, and one
the proxy can't compute is answered with DIGEST_INVALID and not cached.
GetOrFetchArtifactFromURLWithDigest verifies sha512 (and sha384)
digests through the declared-integrity check, so a mismatch is never
committed, and refuses other algorithms with ErrUnsupportedDigest
instead of fetching unverified. Blob routes still only accept sha256.
Cached metadata was stored at _metadata/<ecosystem>/<key>/metadata. Keys
are case-sensitive, but on a case-insensitive filesystem such as APFS or
NTFS, file:// storage put keys that differ only in case in one file:
github.com/Sirupsen/logrus and github.com/sirupsen/logrus each had a
row, and whichever was cached last was served for both. Keys may also
hold names Windows reserves or end in a dot.
The path now carries a SHA-256 of the exact key. Keys that already are
SHA-256 hex digests, as most hashed keys are, keep their path.
A row still pointing at an unhashed path may hold another key's bytes,
so it is neither served nor revalidated; the document is fetched again
and stored at the new path, and the old blob is queued for deletion.
An uncached tarball is checked against the integrity its packument
declares. When the packument couldn't be fetched or was refused, for
example because it is larger than metadata_max_size or the registry
answered 503, the download went ahead with no integrity at all, and
the unverified tarball was cached and served to every later client.
Without the packument the download now fails like the Composer one
does: 502, or 404 when the registry says the package doesn't exist.
Nothing is fetched or cached. Tarballs already in the cache are still
served, since they were verified when they were stored, and a packument
that lists the version without a usable checksum still falls back to
an unverified fetch as before.
With cooldown on, /info/{name} is fetched from upstream and versions still in cooldown are removed line by line. The fetch passed on the client's Accept-Encoding, and setting that header turns off the transport's transparent decompression, so for any client that accepts gzip, Bundler included, the filter scanned gzip bytes, matched no versions and sent the compressed index on with upstream's Content-Encoding. Cooldown filtered nothing for those clients.
The fetch now passes on only Accept, so the transport decompresses the index before it is filtered. If-None-Match and If-Modified-Since are no longer passed on either: the client's validators came from a filtered copy, not from upstream's file.
The filtered index no longer carries upstream's ETag, Last-Modified, Digest, Repr-Digest or range headers, which describe the unfiltered file; Bundler checks the digest against the body it gets.
The index is now read through ReadMetadata before anything is written. bufio.Scanner stopped at a line over 64 KB and the handler sent what it had so far as a complete 200; a read error or an index over the metadata size limit now answers 502.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
The transport remembered a challenge for every /v2/<repository> it was
ever challenged on and never forgot one, so a long-running proxy, or a
metadata document naming many registries, grew the map without limit.
Tokens were only pruned of expired entries, and the expiry came from
the token service.
Challenges are now kept per registry host, holding only the realm and
service, and the pull scope for a request is built from its path, so
pulling many repositories from one registry keeps one entry. Both
caches have a size limit, evicting an arbitrary challenge or the token
that expires soonest, and a token is reused for at most an hour,
whatever lifetime the token service claims.
After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
parseGemFilename tries every "-<digit>" in the filename as the start of the version and matches the rest against two patterns each time, so its cost grows with the square of the filename's length. A path of a few hundred kilobytes kept a request busy for minutes, and the parse doesn't watch the request context.
Gem filenames longer than 512 bytes are now answered with 400 before parsing. Real ones are far shorter.
Without X-Frame-Options or a frame-ancestors policy another site could
frame a version page and trick a signed-in user into clicking Delete.
Every /ui response now sends both; browsed archive files keep their
sandbox policy alongside frame-ancestors.
A page on another site could make a visitor's browser POST a text/plain
body that parses as JSON and start mirror jobs. Browsers only send
application/json cross-origin after a CORS preflight, which the proxy
never answers, so HandleCreate now refuses other content types with 415.
- Name the proxy in the User-Agent of publish-time HEAD requests, as its
artifact downloads do, rather than Go's default, which Maven Central
asks tools not to send.
- Take a version's time from its Gradle .module as well as its .pom, so
either one fetched first carries it.
- Serve filtered metadata through the shared rewrite path, with gzip and
the stale warning like the other ecosystems.
- Say in the docs that a refused .pom or .module is still cached.
A build fetching pinned versions never reads maven-metadata.xml, so each
version's time was read with a HEAD before its download, roughly
doubling the requests to Maven Central, which answered a fresh build
with 429 and a 30-minute Retry-After. The .pom's own response now gives
the time; Maven fetches the .pom before a version's other files, so
their check reads the stored time. A HEAD is only sent for a .pom served
from the cache or a file requested before its .pom.
A 429 on that HEAD now answers 503 with the upstream's Retry-After, as
other held upstream requests do, instead of 502.
A version's publish time is the Last-Modified of its .pom, read with a
HEAD request once per version and stored in the versions table. Named
repositories store theirs under a repository_url qualifier.
Versions in cooldown are left out of maven-metadata.xml, <latest> and
<release> are moved to the last versions left, and the metadata's
checksum files are computed from what is served. Files of a version in
cooldown get 403. Only versions that can still be in cooldown are
checked: none when <lastUpdated> is older than the window, otherwise
from the end of the list back to the first old enough version.
Part of #132.
TestSlowFetchAfterPushIsWaitedFor relied on the ref check finishing
within two seconds, which a slow Windows runner missed. The test now
holds the fetch and ends RefreshWait itself. When the check and the wait
end together, ensureFresh now waits for the fetch rather than picking
either at random. Failing gitcache tests print the cache's log.
A response that ended early for another reason than the size limit was
passed on as complete. TestRelayStopsAtLimit failed that way in CI. The
test upstream now reads the request body as a git server does, and the
test prints the relay's log when it fails.
Shutdown cancelled the eviction, retention, reclaim, backfill and stats
loops but closed storage and the database without waiting for them, and
the stats loop never stopped. A pass still running kept the database
open, which on Windows left proxy.db locked after a server test. Shutdown
now waits up to 30 seconds for them, and for mirror jobs.
The job store tests left their jobs running after the test, and on
Windows the test's directories could not be removed while a job was
still writing to them. JobStore.Wait now waits for jobs to return, and
the tests cancel their jobs and wait for them.
A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Clients resolve base_url and shards_base_url against the proxy's URL. A
scheme or host sends them around the proxy, and a leading slash leaves
its mount, so neither layout is served.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
Each request for a General registry tarball whose hash differed from
the loaded name map started a goroutine that downloaded the whole
registry from upstream again, with no limit on concurrent loads and no
deadline. Alternating between two valid snapshots made every request
cost a full download and parse, and left the map at whichever snapshot
came last, older ones included.
The refresh now only loads the hash /registries currently lists, read
through the metadata cache, runs one load at a time and skips while one
is running, and stops after five minutes. Registry.toml is read up to
64 MB.
A failed .info fetch counted as an unknown time, so a rate-limited or
flaky upstream let versions in cooldown through. It now fails the list
or download with 502; only a version upstream doesn't know, or whose
.info has no time, is let through.
Times are stored under the escaped module path, which keeps modules
that differ only in case apart. Modules with uppercase letters were not
stored before and fetched every .info again on each list request. Only
canonical versions are stored, since a query such as a branch name can
resolve to a later commit.
A .mod or .zip whose version doesn't unescape is answered with 404
instead of being passed upstream unchecked.
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
