Source, archived and binary packages are stored under pkg:cran/{name}, the name from the filename with its case kept, and purl keeps cran names as they are. Package rules use the default mapping and match the name as published, so pkg:cran/rcpp doesn't cover Rcpp.
The mirror can't fetch cran packages, so there's no mirror test.
Refs #306
Packages are stored as pkg:apk/alpine/{name} whichever configured repository they came from, so a package rule covers the package in all of them. Rules may leave out the alpine namespace (pkg:apk/curl); another namespace could never match and is refused. The docs row warns that Alpine mirrors only keep the current build, so an evicted older one can't come back.
The mirror can't fetch apk packages, so there's no mirror test.
Refs #306
* Cache configured version-pinned paths of generic upstreams
The generic handler only knew GitHub release asset paths were immutable, so
any other download, such as Maven from archive.apache.org, went through the
metadata route: streamed uncached by default, or kept as metadata and never
listed as a package.
upstream.generic_artifacts maps a generic upstream to regular expressions
for its version-pinned paths. A pattern must match the whole path after
/generic/{name}/ and has version and file named groups, plus an optional
name group. Matching downloads take the release asset route: cached in the
artifact cache, served while the upstream is down, and listed under the
generic ecosystem. Other paths are unchanged.
* Fix generic artifact cache identity and query forwarding
* Serve generic release assets cached under the unhashed key
Release assets cached before hashed keys used {repository}/{asset} and
became unreachable after an upgrade, so an offline upstream returned 502.
Serve such an entry when its recorded upstream URL matches the request.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Proxy the OCI referrers API in the container handler
GET /v2/{name}/referrers/{digest} used to fall through to a plain 404,
so clients only ever saw referrers stored under the sha256-<hex> tag
schema. Registries that keep SBOMs and signatures behind the referrers
API looked empty through the proxy.
The index is now fetched from the upstream that the repository routes
to and cached like the tag list: fresh within the metadata TTL,
revalidated with If-None-Match, and served stale when the upstream is
down. artifactType is not forwarded, so one cached row covers every
filter and clients filter the full list themselves, which they do when
OCI-Filters-Applied is absent. An upstream 404 is passed through
untouched, so clients still fall back to the tag schema for registries
like GHCR. When the upstream cannot answer and nothing is cached, the
proxy keeps answering 404 as before rather than failing the request.
Closes#144
* Cover absolute upstream links in the referrers link test
The second page now answers with an absolute Link pointing back at the
upstream host, and the test checks that it comes back rewritten to the
proxy. That was missing so far, and it also gives the upstream variable
a reason to be declared before the handler closure, which staticcheck
complained about.
* Read cached referrers through a small shared loader
loadContainerReferrers was a line-for-line copy of loadContainerTags with
the types renamed, which dupl flags in CI. The part that opens the cache
row and reads the body now lives in loadContainerMetadata next to
storeContainerMetadata, and the referrers loader only maps the row onto
its struct. The tag and manifest loaders are left as they are.
* Make the manifest routing test notice which handler answered
For foo/manifests/referrers/<digest> the manifest handler and the
referrers handler send the very same upstream path, so the test passed
even with the referrers case moved in front of the manifest case. The
fake upstream now rejects requests that only accept an image index,
which is what the referrers handler sends.
* Check that a cached referrers row links back to whoever reads it
A repository can be reached as upstream/<name>/<repo> and, when the
default registry is the same host, as plain <repo>. Both end up in the
same cache row, so the pagination Link has to be rewritten when the row
is served, not when it is stored. The link test now reads the row
through both paths; with a store-time rewrite the second path would get
the first one's link.
* Make the referrers ETag test prove the revalidation
Without If-None-Match the fake upstream just sends the full index again,
so the old assertions held either way. The test now counts the 304s the
upstream sent and expects exactly one.
* Reword which tools use the referrers API in the README
The cosign mention was too broad. cosign reaches the referrers API for
its newer signature bundles, while the old .sig tags keep going through
the normal manifest path. Also rewraps the paragraph, one line had run
past the others.
* Check that a 304 restarts the referrers cache TTL
The ETag test now backdates the cached row instead of running with a
zero TTL. After the upstream answers 304, a third request has to come
from the cache. Before, the test would also have passed if the 304
branch forgot to bump fetchedAt and every request kept revalidating.
* Test the referrers Content-Type fallback
None of the fake upstreams left out Content-Type, so the default to the
image index type was never exercised. oras-go rejects anything else, so
it is worth pinning, both for the first fetch and for the cached copy.
* Test referrers digests with extra characters around them
The invalid-request table only had digests that were too short or used
an unknown algorithm, so it would not notice if the digest pattern lost
its anchors. Two cases now put junk before and after an otherwise valid
sha256 digest.
* Reuse the manifest cache key for referrers
The referrers handler had its own key function, but it did exactly what
containerManifestCacheKey already does with four strings. Referrers rows
sit in their own oci-referrers ecosystem anyway, so they can't collide
with manifests.
* Drop the separate struct for cached referrers
A cached referrers response carries the same fields as a cached tag
list, and the freshness check was a straight copy too. So the referrers
code just uses cachedContainerTags and containerTagsFresh now. Loading,
storing and writing stay on their own, since the ecosystem and the Link
handling are different.
* Route referrers requests by ns like the other OCI endpoints
Now that containerd ns support is on main, referrers requests have to
go through registryForRequest too, otherwise a _default mirror would
look every subject up on Docker Hub. ns only picks the registry, so it
is dropped from the forwarded query and the cache key, and the next-page
Link keeps it, the same way tag lists do. This also follows the new
rewriteContainerTagsLink signature, which takes the namespace.
* Without a cached index, referrers now pass upstream errors through and answer 502 when the upstream can't be reached or sends garbage, so clients no longer read an outage as a registry without the referrers API. Stale cache is still served when there is one.
* The tag list and manifest routes now only match paths that end in their endpoint, so a repository with tags/list or manifests in its name can still reach the referrers endpoint.
* GitLab Composer archives are now cached per commit and fetched for the commit the lock file pins, and branch names with a slash are escaped in dist URLs so the download route matches them.
* Composer branch archives that an older proxy cached from another commit's URL are now discarded and fetched again, and a refetched artifact records the URL it actually came from.
* When a GitLab commit is fetched under its own name, the archive older proxies cached under the shared archive.zip name is dropped. A test now moves the branch from A to B and checks that A keeps coming from the cache.
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
* Fetch the pinned commit for Composer branch downloads
The download handler resolved a version's dist URL from the current metadata and cached the archive under whatever file name the client asked for. A lock file pinning an older commit of a branch like dev-master got the branch head, stored and served as that older commit.
Downloads now fetch the commit the file name stands for when the dist URL ends in a commit hash (GitHub zipballs, Bitbucket archives), and return 404 for other names the metadata no longer lists.
* Accept bare commit names from older Composer lock files
Proxy versions before the .zip suffix handed out GitHub zipball URLs ending in the bare commit hash. Lock files written back then still ask for those names, and the new file name check turned them into 404s.
* Rebuild Composer commit URLs only for branch versions
A tagged release lists one commit and lock files ask for that one. Rebuilding the URL for any requested commit let a client store another commit, even one from a fork, as the release, and the browse view could then show it.
* Note that the Composer cache key leaves out the query
* Conan 2 downloads recipe and package files from /v2/conans/.../revisions/{rrev}/files/{file} and .../packages/{id}/revisions/{prev}/files/{file}, but the handler only cached the invalid /v1/files and /v2/files paths, so archives were never stored. This adds the real routes and drops the old ones; manifests and conaninfo.txt stay in the metadata cache.
* Conan 2.25 can also compress archives with xz or zstd (conan_package.txz, conan_package.tzst and so on), so cache those too.
* Store Conan archives under plain name and version, like pkg:conan/zlib@1.3.1, with the revisions and any user/channel in the filename. The old name/version@user/channel with one version per revision never matched a denylist entry or what scanners look up.
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
- Walk version lists in place instead of decoding the document into
map[string]any and expanding the minified format
- Keep minified metadata minified: each written version carries what differs
from the client's expanded state, so changes made by versions dropped by
cooldown fold into the next kept one; every version gets its own dist
- Copy everything else byte for byte; only dist values are written fresh
- Resolve download URLs by walking the list in place, tracking inherited
fields, instead of decoding and expanding the whole document
- Add forEachJSONElement, rewriteJSONMembers and jsonStringIs to the scanner
The Maven handler cached only .jar, .war, .ear, .pom, .aar, .klib and
.module files, so distributions published as archives, such as
apache-maven-{version}-bin.tar.gz, were proxied uncached on every request.
.zip, .tar.gz, .tgz, .tar.bz2 and .tar.xz files now go through the artifact
cache like other artifacts.
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
Rewriting a packument decoded the whole document into map[string]any and
encoded it again. For a full packument of tens of megabytes that costs many
times the document size per request, and with cooldown enabled (which skips
the rewrite cache and forces full packuments) it happens on every request.
- Rewrite npm metadata in place: walk the raw JSON for member offsets, decode
only the version names, time and dist-tags for the existing filters, and
assemble the response from slices of the original bytes with only tarball
URLs (and filtered time/dist-tags) written fresh. Untouched values now reach
clients byte for byte instead of round-tripping through encoding/json.
- Look up versions[v].dist.tarball and time[v] directly on the tarball
download path instead of decoding every version.
- Record the stored metadata's SHA-256 in metadata_cache.content_digest and
key the rewrite cache by it, so a repeat npm or Composer request inside the
TTL is served from the rewrite cache without reading or hashing the stored
document.
On a synthetic 25 MB packument with 5000 versions, rewriteMetadata drops from
~150-174 MB / 706k allocations per call to ~31 MB / 90k, and runs in about
two thirds of the time. npmVersionTarball drops from ~1 MB / 10k allocations
to one allocation.
* Coalesce concurrent metadata fetches
Concurrent cache misses for one artifact already share a single upstream
fetch (#329), but metadata misses did not: every request for a package's
metadata went to the registry, even when the same document was already
being fetched. In CI that is the common case. Composer resolves versions
on every job, since MediaWiki core and its extensions commit no
composer.lock, and each package is one /p2/ document, so a burst of jobs
fetched each document once per job.
Metadata misses now follow the same pattern as artifacts: the first caller
fetches, the rest wait for its result, and the caller that takes the key
rechecks the cache first so a fetch that just finished is not repeated.
The key includes the Accept and Accept-Encoding headers, which change the
bytes upstream returns (npm's abbreviated and full documents), and whether
the caller validates the response.
One difference from artifacts: the shared fetch is detached from the first
caller's cancellation. Artifact fetches keep it for scanning and mirroring,
neither of which applies to metadata, and otherwise one job disconnecting
would fail every job waiting on the same document. runScan detaches the
same way, and the HTTP client's timeout still bounds the fetch.
This covers every ecosystem that goes through fetchOrCacheMetadata
(Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with
metadata caching on or off. Streamed metadata with caching off is not
coalesced.
On an 8 vCPU test instance with caching off, 10 concurrent clients
requesting the npm metadata of 1,263 packages went from 12,630 upstream
requests to 1,898. With cache_metadata on, 20 clients requesting 1,538
packages made exactly 1,538.
* Run each caller's validate on a shared metadata fetch
A caller that joined another's metadata fetch got the shared bytes
without running its own validate. NuGet's cooldown path uses validate to
decode the document for the request, so a joined download found no
publication date and was allowed, bypassing the cooldown.
coalesceMetadata now reports whether the caller joined, and
coalescedMetadataMiss runs that caller's validate on the shared bytes.
Validation before caching is unchanged: the first caller still runs it
inside the fetch.
TestNuGetCooldownConcurrentDownloads reproduces the bypass through
NuGetHandler.Routes() with two overlapping downloads of a version inside
its cooldown.
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
expandMinifiedVersions deep-copied every inherited field into every
version (b68184c) so that rewriting one version's dist URL in place
could not change the versions that inherited it. For a package with a
long history that means recursively copying require, autoload and the
rest hundreds of times per document, on every metadata request.
dist is the only field the proxy changes after expansion, so copy just
that: rewriteDistURL now gives the version its own dist map before
setting the URL, and expansion shares the other inherited values.
TestComposerExpandMinifiedSharedDistReferences still guards the original
bug, and fails if the copy in rewriteDistURL is removed.
On the symfony/console metadata from Packagist the rewritten output is
byte-identical, and the rewrite drops from about 20.7 ms to 13.8 ms,
with allocations down from 13.6 MB to 9 MB.
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
TestRelayRoutes fails intermittently on CI with "write: connection reset
by peer" at the upstream handler's final Flush, on /hex/packages/demo and
/gem/info/demo.
Those two routes are the only ones in the table that fan out a second
upstream request: with cooldown enabled, the hex and gem handlers fetch
version timestamps concurrently with the artifact. The test server answers
that sidecar request with the same 64 KiB chunked 502, but the handler
returns as soon as it sees a non-200 and closes the body unread, so the
proxy drops the connection while the test server is still writing to it.
Whether the reset lands before or after the Flush is a scheduling race,
which is why it only surfaces on loaded runners.
The flush error says nothing about the response under test, which is
asserted on the downstream side, so stop reporting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eviction deleted an object before checking its record still pointed at
it. If a refetch had moved the record, that object was queued and a
request that read the record earlier could still be opening it. Eviction
now clears first, skips the delete if the record moved, and queues the
object when the delete fails after a clear.
Reclaim queues a failed delete again, which moves it behind the rest, so
objects the backend keeps refusing cannot fill every batch.
Also drop the clearArtifactCache passthrough and ClearCachedArtifact's
unused context, note that only tests call ArtifactPath, and document that
max_size does not count objects waiting to be reclaimed.
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
The denylist tests call NewDebianHandler with three arguments, but the
additional APT archives change gave it a fourth. Each passed CI on its
own branch, and together they leave internal/handler without a test
binary that compiles, so go vet and go test fail on main.
A request to /debian/{name}/ for a name that is not configured fell through
to the main archive, so a typo surfaced as the upstream's own HTML 404 with
no mention of the repository. The reply now names the unknown repository and
lists the configured ones.
Scoped to {name}/dists/ and {name}/pool/. The main archive serves README,
indices/ and project/ at its root, so an unknown first segment stays a
main-archive path everywhere else.
apt prints only the status line, so this is visible via curl or the proxy
log rather than in apt output.
The motivation for separate archives was repeated across six files and the
reserved-name rationale across five. Keep each in one place -- the reference
docs -- and leave the code comments to what an informed reader cannot get
from the code: why an unknown first segment is not a 404 as it is for APK,
and why the main archive keeps its legacy cache identities.
Comments and prose only; no behaviour change.
The digest-aware cache check discarded a stale entry before its caller
took the key. A slow caller could delete an entry another caller's
fetch had just committed, and everyone sharing that fetch then failed
to open it.
The check now only reports the miss. The caller running the shared
fetch discards the entry under the key, after the recheck, so one a
previous fetch refreshed is served, not deleted. The mismatch warning
fires once per refresh instead of once per request.
Swift HEAD no longer discards either, having no fetch to do it under.
It probes upstream as before, and the next GET replaces the entry.
Different digests or URLs, or no digest, use different keys and can
still collide on the storage path. That is the storage layout follow-up.
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
