The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
Downloads are routed as /{channel}/{arch}/{filename}, so the stored name is always {channel}/{name}. A key like pkg:conda/conda-forge/label/rc/numpy passed validation and matched nothing; it's now refused.
Refs #306
parseTarballFilename split at the last hyphen before a digit, so amqp_client-4.1.5-1.tar was stored as package amqp_client-4.1.5. Hex names can't contain a hyphen, so it now splits at the first one. The docs caveat is gone.
Refs #306
The handler stores packages as {channel}/{name}, which purl writes as pkg:conda/conda-forge%2Fnumpy. Package rules name the channel as the namespace (pkg:conda/conda-forge/numpy) and get mapped to that form, with the package name lowercased the way conda treats it. A key without a channel or with an empty part could never match and is refused.
The mirror can't fetch conda packages today, so there's no mirror test.
Refs #306
Tarballs are stored under the name from the tarball filename, and purl keeps hex names as they are, so package rules use the default mapping with the name as published.
The filename parser splits versions like 4.1.5-1 at the wrong hyphen and files them under a package like amqp_client-4.1.5, so package rules don't reach them; the docs row says so until the parser is fixed.
Refs #306
The mirror passed ids through as written, so pkg:nuget/Newtonsoft.Json@13.0.3 was stored under a copy no client is served while the handler kept serving its lowercased one. GetOrFetchArtifact now lowercases nuget names, and the docs no longer ask for lowercase ids in mirror lists.
Refs #306
parseGemFilename split at the last hyphen before a digit, so sorbet-static-0.5.11002-universal-darwin-22.gem was stored as package sorbet-static-0.5.11002-universal-darwin. It now splits at the first one, which only misreads gem names with a -<digit> segment. The docs caveat is gone.
Refs #306
The history is process-wide, and tests cleared it only after running, and
not all of them. Records from earlier tests or an earlier -count repetition
reached later assertions: TestRecordUpstreamOperations left a record for
TestRecentFailuresNewestFirst, and the server tests never cleared it.
- Export ResetRecentFailures so tests outside the metrics package can use it
- Clear the history before and after every test that records or reads
failures directly, in metrics, server and handler
- Assert exactly one failure in the handler tests instead of looking only
at the newest
Source archives are stored under the registry identity, pkg:generic/swift-registry/{scope}.{name}, lowercased. Package rules write the scope as the namespace (pkg:swift/apple/swift-nio) or use the stored form, in any case. A source repository key like pkg:swift/github.com/apple/swift-nio doesn't name a registry identity and is refused.
The mirror can't fetch swift packages, so there's no mirror test.
Refs #306
Since #431 the handler stores Conan archives under pkg:conan/{name} with the version as the version and the revisions and any user/channel in the filename. purl keeps conan names as they are, so package rules use the default mapping, and pkg:conan/zlib covers every version, revision and user/channel of the recipe.
The mirror can't fetch conan packages, so there's no mirror test.
Refs #306
Archives are stored under pkg:composer/{vendor}/{package}, which purl lowercases, so case in a package rule doesn't matter. A key without a vendor could never match and is refused. Packages that git-pkgs wrote as "packagist" fall under the composer rules too.
The mirror can't fetch composer packages today (its resolver has no composer support, and #415 would file them under packagist), so there's no mirror test.
Refs #306
Crates are stored under the name they were published with, and purl keeps cargo names as they are. Package rules have to match that name exactly, so pkg:cargo/inflector doesn't cover Inflector.
Refs #306
Generic upstreams pointed at archive.apache.org, nodejs.org or
dl.google.com/go now cache their versioned downloads as artifacts without
any generic_artifacts patterns. Configured patterns add to the built-in ones.
- Start from the schema before migration 011 on SQLite and PostgreSQL, with
npm and Composer metadata cached, and check the rows and stored bytes
survive the migration unchanged
- After the upgrade, check two replicas serve the cached document with
matching ETag and Last-Modified and answer conditional requests with 304
- Change the upstream document and check neither replica answers the old
validators with 304, and both agree on the new ones
- Give the PostgreSQL handler test a database of its own, so it shares no
tables with the database package tests on the same server
The handler stores NuGet ids lowercased, but purl keeps nuget names as written, so package rules get lowercased before they're matched. Case in a rule doesn't matter.
The mirror stores ids as given, so a mixed-case PURL ends up as a copy clients never get. Until that's fixed, the docs ask for lowercase ids in mirror lists.
Refs #306
Packages are stored as pkg:rpm/{name}, without a vendor and with the name's case kept. Package rules may name a vendor the way PURLs usually do (pkg:rpm/fedora/curl); it's dropped. The docs row warns that Fedora and EPEL update repositories only keep the latest build.
The mirror can't fetch rpm packages, so there's no mirror test.
Refs #306
Packages are stored under the name from the download path, and purl keeps pub names as they are, so package rules use the default mapping with the name as published.
Refs #306
Gems are stored under the name from the gem filename, and purl keeps gem names as they are, so package rules use the default mapping and match the name as published, case included. Packages that git-pkgs wrote as "rubygems" fall under the gem rules too.
The filename parser splits platforms that end in a number (universal-darwin-22) at the wrong hyphen and files those gems under a package like sorbet-static-0.5.11002-universal-darwin, so package rules don't reach them; the docs row says so until the parser is fixed.
The mirror can't fetch gems today (it asks the resolver for "rubygems", see #415), so there's no mirror test.
Refs #306
Julia takes no package rules, since its PURLs need a uuid qualifier the cache doesn't store; configuring one already fails validation. The julia rule and the default cover package tarballs, artifacts and registry snapshots alike. Everything there is content-addressed, so an evicted tarball comes back unchanged.
Refs #306
Helm takes no package rules: charts are cached under the configured repository's name and the chart digest, not under the chart name, so a key like pkg:helm/nginx would name a repository. The helm rule and the default cover charts from every HTTP repository; charts pulled over OCI go through the container handler and fall under oci.
Refs #306
Dates were kept in each process's rewrite cache, so replicas sharing a
database and storage dated independently: one dated a document, upstream
changed in that second, and another gave the new document the same date,
so a client holding it got 304 and kept the old document.
- Date from metadata_dates instead of process memory: a package's date is
the second it started serving its current representation, recorded once
for every replica
- Switch a package to another representation only in a second later than
its recorded date, and serve it undated until then
- Date nothing in the second the process started, so dates clients hold
from before this change, or from the process it replaced, stay older
- Serve no date ahead of the local clock, recorded by a replica whose clock
is ahead
- Date rewrites without the rewrite cache too; the cache keeps only ETags
- Add two-replica, pre-fix date and clock skew regression tests
Replicas sharing a database need one place to agree on the Last-Modified
of rewritten metadata, which process memory cannot give them.
- Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the
ETag being dated and its date in Unix seconds, so no zone is involved
- Insert a package's first date only if no replica recorded one first
- Swap a date only if it is still the one read, so of two replicas
switching a package at once only one succeeds
- Create it in both schemas and in migration 011
Two false 304s remained. A rewrite kept the date it was made, so when
upstream went from A to B and back to A, A's cached rewrite came back with
a date earlier than B's and a client holding B got 304. And a second
rewrite in the same second was dated a second ahead of the clock, a date
only that process remembered: after a restart a changed document could be
dated earlier than one the client held.
- Record which rewrite each package is serving and since when, and date a
rewrite by the second its package switched to it, including back to one
still cached
- Never date ahead of the clock: a rewrite switched to in a second that
already dated another for the package goes undated until the next second
- Date nothing in the second the cache starts, the last second the previous
process may have dated a rewrite in
- Drop a package's record when its current rewrite is evicted, so records
stay bounded by the cache; forgetting one dated in the current second
holds back dates for new packages until the next second
Last-Modified was upstream's date clamped to the process start, so
different documents could share a date: after a restart, a cached document
and the newer one a refresh fetched were both dated at startup, and a client
holding the first got 304 for the second.
- Date each rewrite by when the rewrite cache first made it, and keep that
date while the entry stays cached
- Give every new rewrite of a package a date later than the last one it got,
even within the same second or after its entry was evicted, so no two
representations share a date; dates are remembered only until the clock
passes them
- Send no Last-Modified when the rewrite cache is off, since nothing records
when an uncached rewrite was made; cooldown still sends none
- Drop the upstream Last-Modified from metadata lookups, which nothing uses
now
- With cooldown on every rewrite runs, and hashing each rewritten body for
its ETag made every npm metadata request about 20% more expensive
- Hash what fixes the output instead: the upstream digest, proxy URL,
package name, the versions cooldown and the denylist kept, and a
per-process nonce standing for the startup configuration
- Carry the stored document's digest and upstream Last-Modified with the
metadata lookup result, so neither is read from a second cache row lookup
that a concurrent refresh could have changed
- Fall back to hashing the body when the digest is unknown, and keep the
body hash with cooldown off, where it is computed once per cached rewrite
- Rewritten metadata went out with only Content-Type, so pnpm, npm and
Composer could never revalidate and downloaded every document in full
- ETag is the SHA-256 of the rewritten body, computed once per rewrite and
kept with the rewrite cache entry; exact under cooldown as well
- Last-Modified is the upstream Last-Modified, never earlier than the process
start, since a restart can change the rewrite; omitted with cooldown on
- Answer If-None-Match, and If-Modified-Since when there is no If-None-Match,
with 304 on the rewrite fast path and on the fetch path
* npm: version lookup returns 502
Looking up a specific version will trigger a 502 instead of return the version metadata.
* scope only not handled explicitly
* honor cooldown in version lookups
tested (set default cooldown to 7d and 8.0.7 is released four days ago or something):
```
$ curl -sS 'http://localhost:8080/npm/@babel%2Fcore/8.0.7' | jq
{
"error": "package not found"
}
```
* package not found -> version not found
* gocognit
* Enable storage retention for npm
npm stores packages under the PURL purl builds from the full name, so package rules use the default mapping. Both pkg:npm/@babel/core and pkg:npm/%40babel/core work.
Refs #306
* Tighten npm retention tests and docs
The docs now say npm package rules match the name exactly, so pkg:npm/jsonstream doesn't cover JSONStream. The refetch, package rule and mirror tests were weaker than their names and now check what they claim.
Refs #306
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Enable storage retention for pypi
PyPI names are stored the way purl normalizes them, lowercased and with _ turned into -, so Typing_Extensions and typing-extensions hit the same package. The mirror can't fetch PyPI files today, so there's no mirror test.
Refs #306
* Correct the pypi package rule note
Dots in PyPI names aren't normalized, so zope.interface and zope_interface files end up under two packages. The docs now say to add a rule for each.
Refs #306
* Mention repeated separators in the pypi rule note
Runs like "__" aren't collapsed either, so a__b is stored as a--b.
Refs #306
Module paths are stored lowercased, so package rules match whatever case they're written in. A major version like /v2 is its own module and needs its own rule. Keys in the !-escaped form from proxy URLs and the module cache are refused, since they could never match.
Refs #306
Blobs are stored under the repository name of the pull, which purl writes as one name (pkg:oci/library%2Fnginx). Package rules write it as a path, pkg:oci/library/nginx, and get mapped to that form; keys with an empty segment, a tag or digest, or a bare upstream/{name} prefix are refused. Named upstreams keep their prefix, also when containerd reaches them through ns. Homebrew bottles and Helm charts pulled over OCI fall under oci too, bottles under Homebrew's image name (openssl@3 is homebrew/core/openssl/3). Manifests live in the metadata cache and aren't touched.
Refs #306
Package rules use pkg:maven/{group}/{artifact} with the case as published. Keys without a group or with the group written as a path can never match and are refused. Files from the Gradle Plugin Portal fallback fall under the maven rule; the Gradle build cache isn't affected.
The mirror can't fetch Maven packages today (it passes group/artifact where the resolver wants group:artifact), so there's no mirror test.
Refs #306
- Add ui_failure_details / PROXY_UI_FAILURE_DETAILS, off by default
- Each failure row expands to its latest failures, newest first
- Show a note naming the config key when failures exist and the flag is off
- Document the flag in README, configuration reference and example config
- Failure recorders take a FailureDetail: PURL, filename, URL, storage path, error
- Keep the last 20 failures per counter series in memory, reset on restart
- Strip userinfo, query and fragment from URLs, including those inside error text
- Pass the detail from upstream, storage, integrity, health probe and scan call sites
- Package names and URLs stay out of Prometheus labels
Source, archived and binary packages are stored under pkg:cran/{name}, the name from the filename with its case kept, and purl keeps cran names as they are. Package rules use the default mapping and match the name as published, so pkg:cran/rcpp doesn't cover Rcpp.
The mirror can't fetch cran packages, so there's no mirror test.
Refs #306
Packages are stored as pkg:apk/alpine/{name} whichever configured repository they came from, so a package rule covers the package in all of them. Rules may leave out the alpine namespace (pkg:apk/curl); another namespace could never match and is refused. The docs row warns that Alpine mirrors only keep the current build, so an evicted older one can't come back.
The mirror can't fetch apk packages, so there's no mirror test.
Refs #306
