A version's publish time is the Last-Modified of its .pom, read with a
HEAD request once per version and stored in the versions table. Named
repositories store theirs under a repository_url qualifier.
Versions in cooldown are left out of maven-metadata.xml, <latest> and
<release> are moved to the last versions left, and the metadata's
checksum files are computed from what is served. Files of a version in
cooldown get 403. Only versions that can still be in cooldown are
checked: none when <lastUpdated> is older than the window, otherwise
from the end of the list back to the first old enough version.
Part of #132.
TestSlowFetchAfterPushIsWaitedFor relied on the ref check finishing
within two seconds, which a slow Windows runner missed. The test now
holds the fetch and ends RefreshWait itself. When the check and the wait
end together, ensureFresh now waits for the fetch rather than picking
either at random. Failing gitcache tests print the cache's log.
A response that ended early for another reason than the size limit was
passed on as complete. TestRelayStopsAtLimit failed that way in CI. The
test upstream now reads the request body as a git server does, and the
test prints the relay's log when it fails.
Shutdown cancelled the eviction, retention, reclaim, backfill and stats
loops but closed storage and the database without waiting for them, and
the stats loop never stopped. A pass still running kept the database
open, which on Windows left proxy.db locked after a server test. Shutdown
now waits up to 30 seconds for them, and for mirror jobs.
The job store tests left their jobs running after the test, and on
Windows the test's directories could not be removed while a job was
still writing to them. JobStore.Wait now waits for jobs to return, and
the tests cancel their jobs and wait for them.
A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Clients resolve base_url and shards_base_url against the proxy's URL. A
scheme or host sends them around the proxy, and a leading slash leaves
its mount, so neither layout is served.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
Each request for a General registry tarball whose hash differed from
the loaded name map started a goroutine that downloaded the whole
registry from upstream again, with no limit on concurrent loads and no
deadline. Alternating between two valid snapshots made every request
cost a full download and parse, and left the map at whichever snapshot
came last, older ones included.
The refresh now only loads the hash /registries currently lists, read
through the metadata cache, runs one load at a time and skips while one
is running, and stops after five minutes. Registry.toml is read up to
64 MB.
A failed .info fetch counted as an unknown time, so a rate-limited or
flaky upstream let versions in cooldown through. It now fails the list
or download with 502; only a version upstream doesn't know, or whose
.info has no time, is let through.
Times are stored under the escaped module path, which keeps modules
that differ only in case apart. Modules with uppercase letters were not
stored before and fetched every .info again on each list request. Only
canonical versions are stored, since a query such as a branch name can
resolve to a later commit.
A .mod or .zip whose version doesn't unescape is answered with 404
instead of being passed upstream unchecked.
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
Denylist entries now also match Composer versions with Packagist's
leading v, OCI repositories written as a path, Swift registry packages
in either form (also when streaming), Go versions with escaped uppercase
letters, and gems whose names contain a dash followed by a digit. OCI
manifests are checked by tag and digest, and denied blobs and Swift
archives answer 403 instead of 500 or 502.
Package files outside Hashed/, such as All/{name}-{version}.pkg from
repositories built without pkg repo --hash and the bootstrap
Latest/pkg.pkg, were relayed without any checks. They are still passed
through uncached, since their names can be reused for rebuilds, but a
versioned name is now checked against the denylist first, and while a
scanner covers freebsd they are refused, because only cached artifacts
can be scanned.
CI, the release binaries and swagger take their Go version from go.mod,
which pinned 1.26.7 and so missed the stdlib fixes in 1.26.8 to 1.26.10.
A toolchain line now sets 1.27.3 for them, and the Docker builder uses
the same version. The go line stays at 1.26.7, so source builds with an
older local Go still work.
golangci-lint 2.13.1 can't read Go 1.27.2+ export data, so CI moves to
2.15.0. Its errcheck flags two test cleanups that dropped the server's
exit error; they now check it.
Tomcat and Jetty drop ";params" from a path segment before resolving it,
so "..;" passed the dot-segment check and still got the credentials
configured for the upstream path. The check now ignores ";params", and
package routes refuse ';' like '?' and '#', which also stops "1.0;x"
from passing the denylist as a different version. Composer archives
stay exempt, since git branch names may contain ';'.
Denylist entries were compared as plain PURL strings, but handlers build
PURLs from download URLs that spell packages differently, so some denied
versions were still served. Entries and checked PURLs now go through one
per-ecosystem comparison form: PyPI names follow PEP 503 (case and runs of
-, _ and . are equivalent), NuGet ids and versions ignore case, a gem
version also matches its platform builds, and a conda entry matches the
package in every channel unless it carries a channel qualifier. Cache keys
are unchanged.
Routers clean literal dot segments, but encoded ones such as %2e%2e
reach the handlers. Maven, conda, cran, go and gem append the decoded
path to the upstream URL, so the upstream resolved it outside the
configured base path. Package routes now answer 400 for a '.' or '..'
segment, also when encoded twice or written with a backslash.
Upstream credentials matched such a URL by its path prefix, so
/scope/../other still got the credentials configured for /scope. URLs
with dot segments now get no credentials.
Conda download caching keyed artifacts on channel, name, version and
filename only, but the same filename can hold different bytes in each
subdir (python_abi-3.11-4_cp311.conda in linux-64 and osx-arm64, for
example). A client on one platform could be served another platform's
package. The subdir is now part of the cached filename, so each one gets
its own entry. Entries cached under the old key are no longer used and
are re-fetched on first request.
Only files with a known artifact or checksum extension went through the
download path; anything else, such as the .exe binaries protoc and gRPC
plugins publish or .hpi and .so files, was relayed from upstream as is.
Those files skipped the denylist, cooldown, scanning and the cache. Any
file in a version directory named {artifact}-{version}..., the way Maven
names what it publishes (timestamped for SNAPSHOTs), now takes the
download path. Directory listings and catalogs are still proxied.
proxy mirror and mirror API jobs resolved download URLs with a bare
resolver, so npm, cargo, gem, golang, hex, pub, maven and nuget packages
came from the public registries whatever upstream.* says. The resolver
now takes the configured upstreams as base URLs.
npm packages in upstream.npm_routes are fetched from their route and
stored under the route's filename prefix, the way the npm handler does,
so a public package can't stand in for a routed one and clients are
served the mirrored tarball.
Refs #414
proxy mirror filed pkg:gem and pkg:composer packages under rubygems and
packagist, the names purl gives their registries, while the handlers
store them under gem and composer. Since the resolver accepts both
spellings, mirrored gems were downloaded and filed under rubygems, so
the packages list filtered by gem didn't show them.
The mirror now uses the handler names. The package list, count and
search match every spelling of the requested ecosystem, so rows already
written as rubygems or packagist, by the mirror or by git-pkgs, show up
too without a migration.
Fixes#415
proxy mirror and mirror API jobs listed the versions of unversioned
PURLs from the public registries, ignoring upstream.*. They now list
them from the configured upstream through the proxy's upstream client,
and npm packages from their npm_routes entry. Where the upstream has no
listing API, or the ecosystem has no listing at all, the error asks
for a version.
Refs #524
Catalogues (meta.conf, packagesite and data) carry their signature
inside the archive and go through the metadata cache byte for byte.
Packages under Hashed/ have the content hash in their name, so they go
to the artifact cache as pkg:bsd/freebsd/{name}@{version}. Other files,
including the bootstrap files under Latest/, are passed through.
pkg.freebsd.org is the built-in freebsd repository; more mirrors go
under upstream.freebsd.
Closes#536
Serving cached metadata when the upstream refuses is documented for
generic upstreams, and GitHub rate limits are 403s, so the fallback
stays as it is. #503 handles npm's definite answers. What is left here
is the stale warning for Composer, Chef and hosted npm metadata, which
went through the rewrite cache without it.
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Composer names are case-insensitive, so acme/private and acme/Private
share the version record and its integrity but get different filenames.
A second push under the other case was accepted and broke the integrity
check of the first archive. For ecosystems with one file per version,
any pushed file of the version now refuses the push, both before the
upload and atomically when recording it.
- Record a fetched artifact with its package and version in one
transaction that refuses to replace a pushed artifact, so a fetch that
finishes after a push keeps both the pushed file and its version's
integrity. Pushes and cache fills share this write, including the
compare-and-swap on the storage path that queues every replaced object.
- Don't open the stored file for ecosystems without a manifest.
- Derive multi-file versions from the file being named in the path.
- Validate DELETE paths like PUT paths, answering 400 for malformed ones.
- Parse ?sha256= with integrity.ParseHex.
