* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
The dashboard shows the default and the ecosystem overrides, and counts
package overrides and patterns without naming them, since the UI has no
auth. A package page shows the cooldown that applies to the package and
the kind of rule it comes from.
Also add FillVersionPublishedAt, which stores a publish time only where
none is stored, for recording the versions cooldown holds back.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
On a cache miss the proxy stored the whole artifact before sending any of
it, so clients with an idle timeout gave up on large artifacts: git-lfs
after 30 seconds, pacman after 10. The caller that starts a shared fetch
now gets the bytes as they arrive while they are stored; requests joining
the fetch, HEAD and range requests still wait for the stored copy.
The client's copy is checked against the upstream's digest as it is read,
and a mismatch aborts the response before it completes. A client going
away leaves the copy to storage running. A storage error leaves the
client's copy running: the client gets the artifact, and the failure is
logged at error level and counted in proxy_artifacts_served_uncached_total.
A block-mode scanner has to see the whole artifact first, so with one
configured for the ecosystem a miss is handled as before.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
A HEAD on a cache hit opened the stored object and closed it unread.
On S3 and Azure that is a GET whose body is thrown away, which can also
drop the pooled connection; on GCS it is a GET as well.
The server now marks HEAD requests in the request context, and readStored
answers a marked cache hit from the database record. Size, content type
and ETag already came from the record. Accept-Ranges depended on the
opened reader being seekable, so backends now say whether their readers
are (Blob does, GCS does not) and the HEAD response matches the GET.
A HEAD still counts as a cache hit, as before. DirectServe redirects are
unchanged since they are decided before the object would be opened.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
Mirroring a PyPI version passed each file's sha256 as the integrity
GetOrFetchArtifactFromURLWithIntegrity records with the version. Wheels of
one version share the name and version they are cached under, so the last
file's checksum replaced the others', and serving the other wheels from
the cache failed the integrity check.
The mirror now passes the sha256 as a digest, which is checked against
that file's content hash alone.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
Without X-Frame-Options or a frame-ancestors policy another site could
frame a version page and trick a signed-in user into clicking Delete.
Every /ui response now sends both; browsed archive files keep their
sandbox policy alongside frame-ancestors.
A page on another site could make a visitor's browser POST a text/plain
body that parses as JSON and start mirror jobs. Browsers only send
application/json cross-origin after a CORS preflight, which the proxy
never answers, so HandleCreate now refuses other content types with 415.
