* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
Tomcat and Jetty drop ";params" from a path segment before resolving it,
so "..;" passed the dot-segment check and still got the credentials
configured for the upstream path. The check now ignores ";params", and
package routes refuse ';' like '?' and '#', which also stops "1.0;x"
from passing the denylist as a different version. Composer archives
stay exempt, since git branch names may contain ';'.
Routers clean literal dot segments, but encoded ones such as %2e%2e
reach the handlers. Maven, conda, cran, go and gem append the decoded
path to the upstream URL, so the upstream resolved it outside the
configured base path. Package routes now answer 400 for a '.' or '..'
segment, also when encoded twice or written with a backslash.
Upstream credentials matched such a URL by its path prefix, so
/scope/../other still got the credentials configured for /scope. URLs
with dot segments now get no credentials.
Catalogues (meta.conf, packagesite and data) carry their signature
inside the archive and go through the metadata cache byte for byte.
Packages under Hashed/ have the content hash in their name, so they go
to the artifact cache as pkg:bsd/freebsd/{name}@{version}. Other files,
including the bootstrap files under Latest/, are passed through.
pkg.freebsd.org is the built-in freebsd repository; more mirrors go
under upstream.freebsd.
Closes#536
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes#295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
hosted.enabled turns on pushing packages to the proxy and is off by default, like mirror_api. A pushed version is served in place of the upstream version with the same number, so anyone who can write could swap a package for every client. Enabling it therefore requires hosted.token, which supports ${VAR_NAME} expansion and can be set with PROXY_HOSTED_TOKEN. hosted.max_upload_size caps one archive and defaults to 100MB.
requireBearerToken compares SHA-256 digests of the presented and configured tokens in constant time and answers 401 with a WWW-Authenticate challenge otherwise.
Refs #133
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
Everything under /go/sumdb/ was passed on to the module upstream. But
proxy.golang.org doesn't serve the checksum database, so those requests
always came back 404 and every client had to talk to sum.golang.org itself.
The Go handler now serves /sumdb/{name}/ for the databases in
upstream.go_sumdbs, which defaults to sum.golang.org. Lookups, latest and
tiles come from the database through the metadata cache. Complete tiles
never change, so once they're cached we don't ask again. Unknown names and
paths that aren't part of the protocol get a 404 without going upstream.
If the database answers 404 we pass on up to 4 KiB of its text, because the
go command prints it and it tells you why a version is missing. For that the
not-found error from the metadata fetch now carries the start of the body.
Other handlers still just see ErrUpstreamNotFound.
- Add ui_failure_details / PROXY_UI_FAILURE_DETAILS, off by default
- Each failure row expands to its latest failures, newest first
- Show a note naming the config key when failures exist and the flag is off
- Document the flag in README, configuration reference and example config
* Cache configured version-pinned paths of generic upstreams
The generic handler only knew GitHub release asset paths were immutable, so
any other download, such as Maven from archive.apache.org, went through the
metadata route: streamed uncached by default, or kept as metadata and never
listed as a package.
upstream.generic_artifacts maps a generic upstream to regular expressions
for its version-pinned paths. A pattern must match the whole path after
/generic/{name}/ and has version and file named groups, plus an optional
name group. Matching downloads take the release asset route: cached in the
artifact cache, served while the upstream is down, and listed under the
generic ecosystem. Other paths are unchanged.
* Fix generic artifact cache identity and query forwarding
* Serve generic release assets cached under the unhashed key
Release assets cached before hashed keys used {repository}/{asset} and
became unreachable after an upgrade, so an offline upstream returned 502.
Serve such an entry when its recorded upstream URL matches the request.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- Track per-caller request and byte counts in a bounded table
- Export proxy_client_requests_total and
proxy_client_response_bytes_total over a closed label set; addresses
are never labels
- Gate the sources card behind ui_request_sources, off by default,
since /ui carries no authentication of its own
- Move access_log.trust_forwarded_for to a top-level key
- Keep remote as the TCP peer and add remote_ip alongside it
Part 6 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cached download wrote its hit in its own UPDATE. SQLite runs on
one connection, so each download was a write transaction the rest
queued behind, holding a warm npm cache near 1,000 requests a second
with the CPU two thirds idle.
Hits are now counted in memory and written in one transaction every
database.hit_flush_interval, "1s" by default, "0" for a write per hit.
A failed write keeps the hits for the next flush, and Close writes
what is pending. The same test then served about 2,200 requests a
second.
Closes#322.
The motivation for separate archives was repeated across six files and the
reserved-name rationale across five. Keep each in one place -- the reference
docs -- and leave the code comments to what an informed reader cannot get
from the code: why an unknown first segment is not a 404 as it is for APK,
and why the main archive keeps its legacy cache identities.
Comments and prose only; no behaviour change.
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes#327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
