After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
A page on another site could make a visitor's browser POST a text/plain
body that parses as JSON and start mirror jobs. Browsers only send
application/json cross-origin after a CORS preflight, which the proxy
never answers, so HandleCreate now refuses other content types with 415.
- Name the proxy in the User-Agent of publish-time HEAD requests, as its
artifact downloads do, rather than Go's default, which Maven Central
asks tools not to send.
- Take a version's time from its Gradle .module as well as its .pom, so
either one fetched first carries it.
- Serve filtered metadata through the shared rewrite path, with gzip and
the stale warning like the other ecosystems.
- Say in the docs that a refused .pom or .module is still cached.
A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
A failed .info fetch counted as an unknown time, so a rate-limited or
flaky upstream let versions in cooldown through. It now fails the list
or download with 502; only a version upstream doesn't know, or whose
.info has no time, is let through.
Times are stored under the escaped module path, which keeps modules
that differ only in case apart. Modules with uppercase letters were not
stored before and fetched every .info again on each list request. Only
canonical versions are stored, since a query such as a branch name can
resolve to a later commit.
A .mod or .zip whose version doesn't unescape is answered with 404
instead of being passed upstream unchecked.
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
Denylist entries now also match Composer versions with Packagist's
leading v, OCI repositories written as a path, Swift registry packages
in either form (also when streaming), Go versions with escaped uppercase
letters, and gems whose names contain a dash followed by a digit. OCI
manifests are checked by tag and digest, and denied blobs and Swift
archives answer 403 instead of 500 or 502.
Package files outside Hashed/, such as All/{name}-{version}.pkg from
repositories built without pkg repo --hash and the bootstrap
Latest/pkg.pkg, were relayed without any checks. They are still passed
through uncached, since their names can be reused for rebuilds, but a
versioned name is now checked against the denylist first, and while a
scanner covers freebsd they are refused, because only cached artifacts
can be scanned.
Denylist entries were compared as plain PURL strings, but handlers build
PURLs from download URLs that spell packages differently, so some denied
versions were still served. Entries and checked PURLs now go through one
per-ecosystem comparison form: PyPI names follow PEP 503 (case and runs of
-, _ and . are equivalent), NuGet ids and versions ignore case, a gem
version also matches its platform builds, and a conda entry matches the
package in every channel unless it carries a channel qualifier. Cache keys
are unchanged.
proxy mirror and mirror API jobs resolved download URLs with a bare
resolver, so npm, cargo, gem, golang, hex, pub, maven and nuget packages
came from the public registries whatever upstream.* says. The resolver
now takes the configured upstreams as base URLs.
npm packages in upstream.npm_routes are fetched from their route and
stored under the route's filename prefix, the way the npm handler does,
so a public package can't stand in for a routed one and clients are
served the mirrored tarball.
Refs #414
proxy mirror and mirror API jobs listed the versions of unversioned
PURLs from the public registries, ignoring upstream.*. They now list
them from the configured upstream through the proxy's upstream client,
and npm packages from their npm_routes entry. Where the upstream has no
listing API, or the ecosystem has no listing at all, the error asks
for a version.
Refs #524
Catalogues (meta.conf, packagesite and data) carry their signature
inside the archive and go through the metadata cache byte for byte.
Packages under Hashed/ have the content hash in their name, so they go
to the artifact cache as pkg:bsd/freebsd/{name}@{version}. Other files,
including the bootstrap files under Latest/, are passed through.
pkg.freebsd.org is the built-in freebsd repository; more mirrors go
under upstream.freebsd.
Closes#536
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Composer names are case-insensitive, so acme/private and acme/Private
share the version record and its integrity but get different filenames.
A second push under the other case was accepted and broke the integrity
check of the first archive. For ecosystems with one file per version,
any pushed file of the version now refuses the push, both before the
upload and atomically when recording it.
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes#295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
proxy list and proxy delete sat next to serve, stats and mirror and read as cache operations. The commands are now proxy hosted push, proxy hosted list and proxy hosted delete, before a release makes the names hard to change. proxy hosted without a known subcommand prints the three.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
