* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes#295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
Module zips are stored under the module's PURL, which lowercases golang
names, and under the filename {last element}@{version}.zip. So
github.com/Sirupsen/logrus and github.com/sirupsen/logrus shared one
cache entry although upstream serves different zips, and the second one
requested failed go's checksum check.
A module path with capitals now gets a hash of its exact spelling in the
filename. All-lowercase paths keep the plain name, so existing cache
entries stay valid.
A capitalised module cached before this change sits under its lowercase
twin's plain filename. When a cached zip's upstream URL names a different
module path, it is discarded and fetched again. Only the module path is
compared, so a cache filled from another upstream.go host is kept.
Fixes#466
With cache_metadata on, a failed upstream metadata fetch fell back to the
cached copy for any error. A 404 (package removed) and a 401/403 (the
proxy's credentials rejected) are definite answers, though, and serving
the old version list in their place kept them from reaching the client.
They now pass through: 404 as 404, 401/403 as 502 like other upstream
failures. Timeouts, connection errors, 5xx and 429 still fall back.
The npm, Composer and Chef handlers serve metadata through the rewrite
cache, which dropped the stale flag, so a fallback looked like a fresh
200. They now set the same Warning: 110 header as other metadata.
Fixes#499
* Use shared archives browser for cached artifacts
* Pin archives to combined browser and Hex changes
* Use archives v0.9.0 release
* Return 404 when browsing archive directories as files
* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* Abort the response when a cached artifact fails its integrity check
* Read the seeded artifact's storage path from the database in the integrity test
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Pub downloads without .tar.gz fetched the HTML version page from pub.dev and cached it as the archive, so one request broke that version for everyone after; they now get a 404, and the upstream URL is built from the escaped name and version so an encoded ? or # can't become a query or fragment.
* Pub archives that older versions cached from a path without .tar.gz are now discarded and fetched again, so versions that were already broken heal on the next download.
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
When upstream answered a revalidation with 304, the proxy read the cached
document back from storage and then stored it again: a full upload and a
SHA-256 over identical bytes, for every popular package once per TTL.
A 304 now only moves the cache row's fetched_at on, keeping its digest.
The update is conditional on the row still recording the digest of the
document that was read, so a concurrent fetch that replaced it is not
overwritten; in that case, and for rows written before digests were
recorded, the document is stored as before.
The metadata route caches a package's packument in the abbreviated
install format, unless cooldown or npm_full_metadata needs the full one.
Tarball downloads read the packument from the same cache row but asked
for application/json. Once the row was due for revalidation, a tarball
the proxy had not cached yet fetched the full packument, which did not
match the abbreviated document's ETag, and stored it in its place. Until
the next refresh every client was then served the full document,
typically several times the size.
Both now use packumentAccept. Downloads read only dist.tarball,
dist.integrity and dist.shasum, which the abbreviated format carries; the
publish times the cooldown needs come with the full format it selects.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Checksum and .asc files beside a versioned artifact are as immutable as the
artifact, but were routed through the metadata path: uncached until a
first successful fetch, with no retry or circuit breaker. A build pulling
hundreds of artifacts sent as many checksum requests straight to Maven
Central, which rate-limited the proxy, after which every further checksum
request still went upstream and failed.
Only maven-metadata.xml and its own checksums stay on the metadata path,
since the version listing changes with each release.
TestServerKeepsFetchAfterClientHangsUp gave the upstream fetch 5s to start
and the follow-up request 5s to be answered. On a Windows CI runner, where
the whole test took 13s, the follow-up timed out awaiting headers with
nothing wrong. The test checks the outcome, not the speed, so allow 30s
for both. A real hang still fails.
A version's integrity is a single value shared by every archive cached
for it, but a dev branch caches one archive per commit. Recording the
listed head's shasum made a cache hit for an older pinned commit fail its
check, so its archive was discarded and fetched again, which cleared the
head's recorded shasum in turn. The listed shasum only ever describes the
current head, and Composer checks dist.shasum itself.
- Skip the shasum for dev versions
- Test a pinned commit, then the head, then both again from the cache
npm versions published before dist.integrity existed carry only a SHA-1
dist.shasum, and Composer metadata lists only a SHA-1 dist.shasum, so
neither could be checked.
- Accept native integrity given as an algorithm-prefixed hex checksum,
verified with integrity's exact-digest check, so a recorded "sha1-<hex>"
is also checked on cache hits
- npm: fall back to dist.shasum when dist.integrity is missing or cannot
be parsed; a usable dist.integrity still takes precedence
- Composer: verify an archive against its dist.shasum, except another
commit of a branch fetched in place of the listed one, which the listed
shasum does not describe
- Fetch unverified as before when no shasum is listed, as Packagist does
for GitHub archives, or it is not a SHA-1 digest
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
Package routes share an origin with /ui, and serve bodies whose
Content-Type comes from upstreams or from whoever pushed a hosted package.
An artifact served as text/html or image/svg+xml ran its scripts with
access to the UI.
- Send X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox
on every package-route response: cached and streamed artifacts, cached
metadata, relayed upstream responses and errors
- Archive files browsed from /ui already get the same two headers
The proxy cached the first tarball upstream sent for a version without
checking it against the packument, so a tampered tarball was served to
every client after it.
- Pass the packument's dist.integrity to the existing integrity checks, so
a fresh tarball is verified as it is stored or streamed; a mismatch is
discarded and returns 502, or aborts the streamed response
- Record it with the cached version, so cache hits are verified against it
through the same check as before
- Verify with integrity.Result.Verify, which accepts any digest of the
strongest algorithm listed
- Fetch unverified as before when dist.integrity is missing or cannot be
parsed, or the packument cannot be fetched
Docker sends one Accept header per media type. Reading only the first
forwarded just that type upstream and ruled out cached manifests of the
others, so pulls failed when the upstream was down.
- GetLocalArtifact went through checkCache, which reports a miss when StreamArtifacts is set, so npm and Composer downloads fetched the upstream archive instead of the pushed one
- Split readStored out of lookupCache and read pushed artifacts through it regardless of StreamArtifacts
- Keep the denylist check and the 503/404 answers for unreadable or missing pushed artifacts
- Test npm and Composer downloads of pushed archives with StreamArtifacts set
Refs #133
Pushing and downloading refuse denied versions, but npm and Composer metadata listed every pushed version. A version denied after it was pushed stayed listed, so resolvers picked it and then got 403 on the download. localVersions now leaves denied versions out, which covers both ecosystems and the hosted-only namespaces; npm latest falls back to the highest allowed version. /api/hosted still lists them, so operators can see and delete them.
Refs #134
The manifest lookup took the first composer.json at the root or inside a top directory, in zip order. Dist archives keep composer.json files for tests and examples, so tests/composer.json stored before the root file refused the push with a name mismatch. The root composer.json now always wins. Without one, the manifest inside a top directory is used only when there is exactly one; more than one refuses the archive as ambiguous.
Refs #134
