* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* Abort the response when a cached artifact fails its integrity check
* Read the seeded artifact's storage path from the database in the integrity test
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Pub downloads without .tar.gz fetched the HTML version page from pub.dev and cached it as the archive, so one request broke that version for everyone after; they now get a 404, and the upstream URL is built from the escaped name and version so an encoded ? or # can't become a query or fragment.
* Pub archives that older versions cached from a path without .tar.gz are now discarded and fetched again, so versions that were already broken heal on the next download.
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
When upstream answered a revalidation with 304, the proxy read the cached
document back from storage and then stored it again: a full upload and a
SHA-256 over identical bytes, for every popular package once per TTL.
A 304 now only moves the cache row's fetched_at on, keeping its digest.
The update is conditional on the row still recording the digest of the
document that was read, so a concurrent fetch that replaced it is not
overwritten; in that case, and for rows written before digests were
recorded, the document is stored as before.
The metadata route caches a package's packument in the abbreviated
install format, unless cooldown or npm_full_metadata needs the full one.
Tarball downloads read the packument from the same cache row but asked
for application/json. Once the row was due for revalidation, a tarball
the proxy had not cached yet fetched the full packument, which did not
match the abbreviated document's ETag, and stored it in its place. Until
the next refresh every client was then served the full document,
typically several times the size.
Both now use packumentAccept. Downloads read only dist.tarball,
dist.integrity and dist.shasum, which the abbreviated format carries; the
publish times the cooldown needs come with the full format it selects.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Checksum and .asc files beside a versioned artifact are as immutable as the
artifact, but were routed through the metadata path: uncached until a
first successful fetch, with no retry or circuit breaker. A build pulling
hundreds of artifacts sent as many checksum requests straight to Maven
Central, which rate-limited the proxy, after which every further checksum
request still went upstream and failed.
Only maven-metadata.xml and its own checksums stay on the metadata path,
since the version listing changes with each release.
A version's integrity is a single value shared by every archive cached
for it, but a dev branch caches one archive per commit. Recording the
listed head's shasum made a cache hit for an older pinned commit fail its
check, so its archive was discarded and fetched again, which cleared the
head's recorded shasum in turn. The listed shasum only ever describes the
current head, and Composer checks dist.shasum itself.
- Skip the shasum for dev versions
- Test a pinned commit, then the head, then both again from the cache
npm versions published before dist.integrity existed carry only a SHA-1
dist.shasum, and Composer metadata lists only a SHA-1 dist.shasum, so
neither could be checked.
- Accept native integrity given as an algorithm-prefixed hex checksum,
verified with integrity's exact-digest check, so a recorded "sha1-<hex>"
is also checked on cache hits
- npm: fall back to dist.shasum when dist.integrity is missing or cannot
be parsed; a usable dist.integrity still takes precedence
- Composer: verify an archive against its dist.shasum, except another
commit of a branch fetched in place of the listed one, which the listed
shasum does not describe
- Fetch unverified as before when no shasum is listed, as Packagist does
for GitHub archives, or it is not a SHA-1 digest
The proxy cached the first tarball upstream sent for a version without
checking it against the packument, so a tampered tarball was served to
every client after it.
- Pass the packument's dist.integrity to the existing integrity checks, so
a fresh tarball is verified as it is stored or streamed; a mismatch is
discarded and returns 502, or aborts the streamed response
- Record it with the cached version, so cache hits are verified against it
through the same check as before
- Verify with integrity.Result.Verify, which accepts any digest of the
strongest algorithm listed
- Fetch unverified as before when dist.integrity is missing or cannot be
parsed, or the packument cannot be fetched
Docker sends one Accept header per media type. Reading only the first
forwarded just that type upstream and ruled out cached manifests of the
others, so pulls failed when the upstream was down.
- GetLocalArtifact went through checkCache, which reports a miss when StreamArtifacts is set, so npm and Composer downloads fetched the upstream archive instead of the pushed one
- Split readStored out of lookupCache and read pushed artifacts through it regardless of StreamArtifacts
- Keep the denylist check and the 503/404 answers for unreadable or missing pushed artifacts
- Test npm and Composer downloads of pushed archives with StreamArtifacts set
Refs #133
Pushing and downloading refuse denied versions, but npm and Composer metadata listed every pushed version. A version denied after it was pushed stayed listed, so resolvers picked it and then got 403 on the download. localVersions now leaves denied versions out, which covers both ecosystems and the hosted-only namespaces; npm latest falls back to the highest allowed version. /api/hosted still lists them, so operators can see and delete them.
Refs #134
The manifest lookup took the first composer.json at the root or inside a top directory, in zip order. Dist archives keep composer.json files for tests and examples, so tests/composer.json stored before the root file refused the push with a name mismatch. The root composer.json now always wins. Without one, the manifest inside a top directory is used only when there is exactly one; more than one refuses the archive as ambiguous.
Refs #134
checkCache answered nil both when a stored object couldn't be opened and after an integrity failure had cleared the record. Download handlers then fetched the upstream archive for the same version and filename, and recording it turned the pushed row into an upstream one without its manifest and queued the pushed object for deletion. One storage error while downloading a patched version served the unpatched upstream archive and dropped the push.
checkCache now looks at the record on a miss. For a pushed artifact it returns ErrHostedArtifactUnavailable when the object is recorded but couldn't be served, and ErrHostedArtifactMissing when no object is recorded any more. Every path that would fetch upstream goes through it, so none of them can replace the pushed artifact. Downloads answer 503 and 404 for the two.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
The packument lists pushed versions without cooldown, but the tarball download checked cooldown against the upstream's publish times before looking at the cache. A pushed override of a version the upstream published recently was listed and then withheld with 404. Pushed tarballs are now served right after the denylist check.
Refs #134
With hosted packages enabled, a package's p2 metadata lists its pushed versions, built from composer.json without the root-only keys, with a zip dist at the proxy and the time they were pushed. dist.reference is the archive's sha256 so a re-push isn't served from Composer's cache, and shasum is left empty. Dev versions go to the ~dev file and tagged versions to the other.
A pushed version takes the place of an upstream entry with the same version, with or without a leading v. Pushed entries follow the upstream ones and go through the same minified writer, so fields an earlier upstream entry set and the pushed one lacks are written as __unset rather than inherited.
When the upstream doesn't know the package, the document holds only the pushed versions. Like npm, documents with pushed versions bypass the rewrite cache.
Refs #134
With hosted packages enabled, the packument of a package with pushed versions lists each of them, built from its package.json with dist pointing at the proxy and carrying the archive's sha512, and the time it was pushed. A pushed version takes the place of an upstream entry with the same version. Cooldown and the denylist still filter only upstream versions, and dist-tags stay as upstream has them.
When the upstream doesn't know the package, the packument holds only the pushed versions, with latest set to the highest of them. Requests for a single pushed version are answered from its entry.
Rewrites that include pushed versions bypass the rewrite cache, which is keyed by the upstream document alone, so a push or delete shows on the next request and changes the ETag. Packages without pushed versions take the cached path as before, and with hosted packages disabled there is no extra lookup.
Refs #134
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest).
The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage.
The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity.
Refs #134
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
Handlers set the configured upstream auth header on the request itself
before handing it to the client. When upstream redirected to another host,
Go's client copied that header to the next hop: it strips only
Authorization and Cookie, so a "header" type credential such as
X-JFrog-Art-Api reached the redirect target. The auth transport already
applies credentials per hop, scoped to each hop's URL.
- Remove Proxy.applyUpstreamAuth and Proxy.AuthForURL; auth comes only
from the transport
- Add a server test that redirects a header-authenticated upstream to
another host and asserts the header stays with the upstream
- Wire handler tests' auth through the transport, as the server does
Cooldown only filtered the metadata. A composer.lock names the proxy's
/files URL directly, so `composer install` fetched versions still inside
the cooldown window without ever seeing the filtered metadata. npm already
checks the publish time on tarball downloads.
- Return the version's publish time with its dist URL, following the
minified format's inheritance
- Answer 404 for a version in cooldown before fetching its archive
- Read the download lookup's metadata through ReadMetadata, so it has the
same size limit as other metadata reads
Packages are stored as pkg:deb/{name}, without a vendor and whichever configured archive they came from. Package rules may name a vendor the way PURLs usually do (pkg:deb/debian/curl, pkg:deb/ubuntu/curl); it's dropped, so a rule covers the package from every archive. The docs row warns that Debian and Ubuntu archives only keep the current version per suite.
The mirror can't fetch deb packages, so there's no mirror test.
Refs #306
