With cache_metadata on, a failed upstream metadata fetch fell back to the
cached copy for any error. A 404 (package removed) and a 401/403 (the
proxy's credentials rejected) are definite answers, though, and serving
the old version list in their place kept them from reaching the client.
They now pass through: 404 as 404, 401/403 as 502 like other upstream
failures. Timeouts, connection errors, 5xx and 429 still fall back.
The npm, Composer and Chef handlers serve metadata through the rewrite
cache, which dropped the stale flag, so a fallback looked like a fresh
200. They now set the same Warning: 110 header as other metadata.
Fixes#499
* Use shared archives browser for cached artifacts
* Pin archives to combined browser and Hex changes
* Use archives v0.9.0 release
* Return 404 when browsing archive directories as files
* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* Abort the response when a cached artifact fails its integrity check
* Read the seeded artifact's storage path from the database in the integrity test
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Pub downloads without .tar.gz fetched the HTML version page from pub.dev and cached it as the archive, so one request broke that version for everyone after; they now get a 404, and the upstream URL is built from the escaped name and version so an encoded ? or # can't become a query or fragment.
* Pub archives that older versions cached from a path without .tar.gz are now discarded and fetched again, so versions that were already broken heal on the next download.
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
When upstream answered a revalidation with 304, the proxy read the cached
document back from storage and then stored it again: a full upload and a
SHA-256 over identical bytes, for every popular package once per TTL.
A 304 now only moves the cache row's fetched_at on, keeping its digest.
The update is conditional on the row still recording the digest of the
document that was read, so a concurrent fetch that replaced it is not
overwritten; in that case, and for rows written before digests were
recorded, the document is stored as before.
The metadata route caches a package's packument in the abbreviated
install format, unless cooldown or npm_full_metadata needs the full one.
Tarball downloads read the packument from the same cache row but asked
for application/json. Once the row was due for revalidation, a tarball
the proxy had not cached yet fetched the full packument, which did not
match the abbreviated document's ETag, and stored it in its place. Until
the next refresh every client was then served the full document,
typically several times the size.
Both now use packumentAccept. Downloads read only dist.tarball,
dist.integrity and dist.shasum, which the abbreviated format carries; the
publish times the cooldown needs come with the full format it selects.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Checksum and .asc files beside a versioned artifact are as immutable as the
artifact, but were routed through the metadata path: uncached until a
first successful fetch, with no retry or circuit breaker. A build pulling
hundreds of artifacts sent as many checksum requests straight to Maven
Central, which rate-limited the proxy, after which every further checksum
request still went upstream and failed.
Only maven-metadata.xml and its own checksums stay on the metadata path,
since the version listing changes with each release.
TestServerKeepsFetchAfterClientHangsUp gave the upstream fetch 5s to start
and the follow-up request 5s to be answered. On a Windows CI runner, where
the whole test took 13s, the follow-up timed out awaiting headers with
nothing wrong. The test checks the outcome, not the speed, so allow 30s
for both. A real hang still fails.
A version's integrity is a single value shared by every archive cached
for it, but a dev branch caches one archive per commit. Recording the
listed head's shasum made a cache hit for an older pinned commit fail its
check, so its archive was discarded and fetched again, which cleared the
head's recorded shasum in turn. The listed shasum only ever describes the
current head, and Composer checks dist.shasum itself.
- Skip the shasum for dev versions
- Test a pinned commit, then the head, then both again from the cache
npm versions published before dist.integrity existed carry only a SHA-1
dist.shasum, and Composer metadata lists only a SHA-1 dist.shasum, so
neither could be checked.
- Accept native integrity given as an algorithm-prefixed hex checksum,
verified with integrity's exact-digest check, so a recorded "sha1-<hex>"
is also checked on cache hits
- npm: fall back to dist.shasum when dist.integrity is missing or cannot
be parsed; a usable dist.integrity still takes precedence
- Composer: verify an archive against its dist.shasum, except another
commit of a branch fetched in place of the listed one, which the listed
shasum does not describe
- Fetch unverified as before when no shasum is listed, as Packagist does
for GitHub archives, or it is not a SHA-1 digest
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
Package routes share an origin with /ui, and serve bodies whose
Content-Type comes from upstreams or from whoever pushed a hosted package.
An artifact served as text/html or image/svg+xml ran its scripts with
access to the UI.
- Send X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox
on every package-route response: cached and streamed artifacts, cached
metadata, relayed upstream responses and errors
- Archive files browsed from /ui already get the same two headers
