The 250ms client is meant for the readiness poll, where a timeout is
retried. Reusing it for the OCI manifest request makes the test flake
under -race on Windows CI when fetch and cache I/O take longer, as
seen on #328. The request checks upstream routing, not latency.
The digest-aware cache check discarded a stale entry before its caller
took the key. A slow caller could delete an entry another caller's
fetch had just committed, and everyone sharing that fetch then failed
to open it.
The check now only reports the miss. The caller running the shared
fetch discards the entry under the key, after the recheck, so one a
previous fetch refreshed is served, not deleted. The mismatch warning
fires once per refresh instead of once per request.
Swift HEAD no longer discards either, having no fetch to do it under.
It probes upstream as before, and the next GET replaces the entry.
Different digests or URLs, or no digest, use different keys and can
still collide on the storage path. That is the storage layout follow-up.
createTestPostgresDB dropped artifacts, versions, packages and
schema_info before calling CreateSchema, but not the migrations table.
On a database that has seen one test the migration records survive, so
the next CreateSchema fails while recording 001_add_packages_enrichment_
columns with a duplicate key on migrations_pkey. Running the package
against one Postgres therefore failed from the second Postgres-backed
test on.
Drop vulnerabilities, metadata_cache and migrations as well, so the
fixture clears every table CreateSchema creates. The package now passes
repeatedly against the same database.
* fix(database): set connection pool limits for Postgres
OpenPostgres returned sqlx.Open's handle with database/sql's defaults:
no cap on open connections and two idle ones. Under load nearly every
request opened a new Postgres session, ran its few statements and closed
it again, paying a backend fork and SCRAM authentication each time.
Set the pool limits the issue suggests: 32 open and 32 idle connections,
idle connections closed after 5 minutes and every connection recycled
after 30 minutes. The values live in named constants because the mnd
linter rejects the literals inline.
The test (skipped without PROXY_DATABASE_URL, like the other Postgres
tests) takes 16 connections from the pool, releases them and checks that
all 16 stay idle; with the default pool only two survive.
Fixes#323
* test(database): pin the pool properties instead of the wiring
Review pointed out that the pool test compared MaxOpenConnections to the
constant it was set from, so the assertion followed the constant and
would have accepted postgresMaxOpenConns = 0 (unlimited), and that its
burst of 16 only proved MaxIdleConns >= 16.
The burst now takes postgresMaxIdleConns connections, so every
configured idle slot has to survive the release, and the open cap is
checked to be finite and large enough for that burst before any
connection is taken, so a cap below the idle count fails fast instead
of blocking in db.Conn. The doc comment now says which settings the
test covers; the idle-time and lifetime settings only show up in
DBStats.MaxIdleTimeClosed and MaxLifetimeClosed after minutes of
wall-clock time and stay unexercised.
* test(database): guard the pool test against a vacuous burst
Review showed that with the burst tied to postgresMaxIdleConns the test
also passed for a constant of 2, database/sql's default, or of 0, where
it took no connections at all. It now fails outright unless the
configured idle count exceeds the default, and every connection it
takes is released in a cleanup, so an assertion failure mid-burst no
longer leaves sessions open for the rest of the test binary.
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes#327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
* Stop writing fileblob's .attrs sidecar
fileblob stores blob metadata in an ".attrs" file per object and rewrites
it with os.Create, truncating in place outside the atomic rename that
protects the blob. A read overlapping a write decodes a partial file and
fails with "opening reader: EOF", served as a 502. One writer against
four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob
is most exposed to it, rewriting a key on every refresh while readers
are served from it.
Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is
imported only by internal/storage, Store sets no ContentType, and
Attributes is used only for Size, which comes from os.Stat. A missing
sidecar already defaults cleanly, so "metadata=skip" removes the hazard
rather than locking around it, and saves a write per store.
* Clear .attrs sidecars left by earlier versions
metadata=skip stops fileblob rewriting sidecars but does not delete ones
already on disk, so a sidecar left partial by an interrupted write now
fails every read of its key for good. Before, a later store repaired it
by rewriting.
Store therefore removes the sidecar for the key it writes. Removal is
atomic where the rewrite was not, so a concurrent reader gets the whole
old file or nothing. Delete already removes sidecars, so the two paths
drain a cache between them.
Deriving that path is necessary because fileblob's key escaping is
unexported. It is the identity for a plain key and parts from one only
for keys that are not valid local paths, which is what filepath.Localize
rejects. That also keeps the removal inside the cache directory: without
it a key holding ".." resolves outside.
The clearing test runs one key per storage path the proxy builds, seeded
through a bucket that still writes sidecars so the path under test is
fileblob's own.
* Explain why failed sidecar cleanup does not fail the store
Move the removal into clearLegacySidecar and say why its error is
dropped rather than returned. A failed removal leaves exactly the state
this change inherited, while failing the write would turn a cleanup miss
into a failed request.
Windows makes that concrete: Go opens files with FILE_SHARE_READ and
FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the
sidecar open blocks deletion, and that reader is the workload this
change exists to protect. Propagating would fail stores during exactly
the overlap being fixed. The next store of the key retries.
A test pins it, using a non-empty directory at the sidecar path to make
os.Remove fail with something other than not-exist on any platform.
* Fail the concurrency test if its writer stops
The writer returned silently when Store failed, so the test could pass
with no concurrent writes at all. Its error is now reported, and the
test also checks that at least one write completed.
Reporting it showed the writer had been dying on Windows at its first
collision: Go opens files without FILE_SHARE_DELETE, so a reader holding
the file open makes the writer's rename fail with access denied. The
test now skips there, since it cannot contend a writer with readers on
that platform.
* Clear legacy sidecars for keys fileblob escapes
legacySidecarPath declined any key filepath.Localize rejects, which on
Windows is every key with a colon: OCI digests and Debian epochs. Their
sidecars were never cleared there, and a truncated one kept failing
reads, since fileblob still reads a sidecar it finds under metadata=skip.
fileblob hex-escapes such characters on the way to disk. The path is now
derived the same way, so the sidecar is looked for where fileblob wrote
it. Localize still validates the escaped form, which keeps the removal
inside the cache directory.
* Drop stale comment about declining colon keys on Windows
623ff3e made legacySidecarPath escape keys the way fileblob does, so
colon-bearing keys are now cleared on Windows and the OCI and Debian
rows in TestStoreClearsLegacyAttrsSidecar prove it. The comment
described the behaviour before that commit.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Return the stored artifact from storeArtifact, not a reader
storeArtifact returned a CacheResult holding an open file handle. A
handle has one read position, so it can only ever serve a single caller,
which is what blocks sharing one fetch between concurrent requests.
Return the artifact and its storage path instead, and let each caller
open its own reader through openStoredArtifact. Threading that type
through fetchAndCache, fetchAndCacheFromURL and their error paths is
mechanical; behaviour is unchanged.
* Coalesce concurrent cache misses
A cache miss went from checkCache straight to an upstream fetch with
nothing tracking in-flight work, so N concurrent requests for one
uncached artifact produced N upstream fetches and N stores to the same
key. That is the CI shape: parallel jobs installing overlapping
dependencies against a cold cache. The duplicate stores also fail
requests, racing fileblob's per-key ".attrs" sidecar into a partial read
served as a 502. Over 12 runs of 8 simultaneous requests for one
uncached tarball, against bb2205a: before, 8 fetches per run and 12 of
96 responses were 502; after, 1 fetch per run and none failed.
Route both miss paths through a shared in-flight map keyed on the
artifact, including the download URL and upstream-declared hash so
callers expecting different bytes never share a fetch.
singleflight does not fit: Do gives waiters no way to leave, while
DoChan lets the caller running the fetch abandon it, breaking
storeArtifact's scan-on-disconnect contract. Deciding roles under a
mutex gives both behaviours. The fetch runs on the first caller's
context and is seen through; waiters leave when their own clients do.
This removes the sidecar trigger on this path. The race is in fileblob
and three writers bypass this path entirely, so it is fixed separately.
Fewer failures now reach the circuit breaker, so it trips later.
Sixteen concurrent callers against real file:// storage fail 10 of 10
runs on main and pass 10 of 10 here. Other tests pin key discrimination,
failure propagation, resolver-path coalescing, per-caller readers,
waiter cancellation, key release and panic safety. allocs/op is
unchanged. mockStorage gains a mutex so concurrent tests can use it.
* Normalize digest case in the coalescing key
artifactHashMatches compares digests with strings.EqualFold, but the
coalescing key used the hash verbatim. The same digest in two casings
produced two keys, so two callers for one artifact each ran their own
upstream fetch and store, which is what the coalescing is meant to
prevent.
* Make the panic coalescing test deterministic
The test timed the second caller's arrival with a sleep, so which caller
became the leader was left to the scheduler. When it lost that race the
second caller ran the fetch itself, and its panic was not recovered, so
the test binary died instead of the test failing.
Whether a caller has reached the wait is not observable from outside:
it runs a cache lookup against the database first, so releasing the
leader on a timer races that query. Drive coalesceFetch directly and
hold the shared entry instead, which removes the timing entirely. The
panicking fetcher is no longer needed.
* Recheck the cache before running a shared fetch
A caller checks the cache before it reaches coalesceFetch, so a fetch
that commits in that gap is invisible to it. Arriving after the sharing
entry is gone, it became a new leader and fetched, stored and scanned an
artifact the cache already held.
The leader now rechecks the committed record first. It serves that
record only if its bytes still open, because a record can outlive them,
and refetching is the recovery the cache lookup already makes for that
case. Waiters are unaffected: the record fills the same shared value a
fetch would, and each caller opens its own reader from it.
The recheck is the leader's alone. A waiter has a fetch in flight to
wait on, and rechecking would race it for no gain.
* Lock the mock fetcher's bookkeeping
Coalescing tests call the handler from many goroutines. The key keeps
the fetch itself serialized, but the mock should not lean on that: it
now locks the fields it records, so any concurrency the handler applies
is safe under the race detector.
* Wait for the leader's fetch instead of sleeping
The canceled-waiter test slept 200ms and assumed the leader had taken
the key by then. On a slow scheduler the canceled call could become the
leader and the test would no longer cover waiter cancellation. The
fetcher now signals when its first fetch begins, which happens only once
the key is held.
* fix(handler): fetch conda repodata gzip-compressed on both hops
#304 made the ProxyCached path request Accept-Encoding: identity so the
metadata cache stores upstream bytes verbatim. That is required for the
signed / hash-pinned index ecosystems, but conda's repodata.json is large
plain JSON: linux-64 repodata.json is ~441 MB uncompressed (over the
metadata_max_size cap, so it 502s today) versus ~34 MB gzip.
Replace the ProxyCached path's verbatim bool with an explicit
acceptEncoding string ('' = leave unset / transparent, 'identity', or
'gzip'), reusing #304's existing store-and-replay of Content-Encoding
unchanged. ProxyCached keeps its exported signature and continues to send
identity, so the nine other ecosystems and helm/maven are untouched; only
conda's repodata.json / current_repodata.json now request gzip. Setting
Accept-Encoding explicitly disables Go's transparent decompression, so the
compressed bytes and the Content-Encoding: gzip header are cached and
replayed exactly as identity bytes are. conda, mamba and pixi solicit and
decode gzip on .json URLs; repodata.json.bz2 stays identity.
Fixes#305
* fix(handler): pass metadata content-encoding with the body it describes
The adversarial review of the conda gzip route found a reachable
regression: writeMetadataCachedResponse took Content-Encoding from a
fresh cache-row read while cacheMetadataBlob skips the row write when
Storage.Store fails. Under identity that was benign (the body was plain
anyway), but on the new gzip route a disk-full or object-store outage
served raw gzip bytes as Content-Type: application/json with no
Content-Encoding and HTTP 200 -- conda, mamba and pixi fail to parse
them, with no HTTP signal and only a Warn log, on every request until a
cache write succeeds.
fetchOrCacheMetadata now returns the encoding of the body it hands back
(the upstream value on a fetch, the stored row's value on a TTL hit or
stale fallback) and proxyCachedWithEncoding passes it to
writeMetadataCachedResponse, so the header always describes the bytes
actually written. cachedMeta drops its now-unused content_encoding
field. helm and maven pass "" -- both fetch transparently, so their
stored encoding was always empty and behaviour is unchanged.
Also fixes a vacuous assertion in the new conda test: the upstream
request counter incremented behind the availability gate, so the
cached-replay block could never observe a refetch.
* fix(handler): pin the stale-fallback content-encoding and drop a dead guard
Follow-ups from the adversarial review of the #305 branch, limited to
code this branch introduced:
- proxyMetadataStream is only ever reached with an explicit
Accept-Encoding (ProxyCached passes identity, conda passes gzip or
identity), so the guard around the header set was unreachable; replace
it with the plain one-token substitution of the former literal, which
is the smallest change from main.
- The stale-fallback return of fetchOrCacheMetadata (encoding taken from
the cache row) was the one #305 return site no test pinned: replacing
it with an empty encoding survived the whole suite. Add a conda test
that expires the entry, fails the upstream, and asserts the stored
gzip blob is served with Content-Encoding: gzip.
Not changed, by scope: cacheMetadataBlob still discards the
UpsertMetadataCache error (pre-existing on main). If Storage.Store
succeeds and the row write fails, a later stale fallback or TTL hit can
serve the gzip blob with the row's stale encoding; that needs a DB write
failure plus a second event and is tracked separately.
* fix(handler): restore the pre-existing cachedMeta content-encoding field
The third adversarial review classified deleting cachedMeta.contentEncoding
and its lookupCachedMeta populate as elective: neither line was created by
this branch nor forced by the fix (writeMetadataCachedResponse now reads
the encoding from its parameter and ignores the row value). Under the rule
that pre-existing code this branch did not have to touch stays untouched,
restore both as they are on main. No behaviour change.
Residuals the review documented, unchanged by scope (both share one root
cause: the encoding lives in the cache row and the bytes in the blob, and
neither is written or read atomically):
- cacheMetadataBlob discards the UpsertMetadataCache error, so after a
successful gzip Store and a failed row write a later stale fallback or
TTL hit can serve the gzip blob with the row's stale encoding.
- During the one-time identity->gzip rollout, a request that read a
pre-branch identity row, lost the upstream race to a request that stored
the gzip blob, and then failed upstream serves the gzip bytes with no
Content-Encoding for that one response; later requests self-heal.
- helm and maven now pass an empty encoding; on main a spec-violating
upstream that answered a transparent gzip request with an encoding Go
does not decode (e.g. br) would have had that header replayed from the
row. Degenerate; documented rather than changed.
* fix(handler): keep conda's proxyCached and .bz2 route as on main
Threading acceptEncoding through CondaHandler.proxyCached changed the
form of two pieces of original code the fix did not need to touch: the
repodata.json.bz2 route (method value rewritten as a closure) and
proxyCached itself (new parameter, new call). Restore both exactly as on
main; ProxyCached still sends identity, so the .bz2 route is unchanged in
behaviour. handleRepodata's non-cooldown branch now derives the cache key
inline and calls proxyCachedWithEncoding with gzip directly, so the only
original conda.go line that changes is that one call.
* fix(handler): keep writeMetadataCachedResponse and its callers as on main
Adding a contentEncoding parameter to writeMetadataCachedResponse changed
a signature that predates #304 and dragged its two pre-#304 callers
(helm.go, maven.go) into the diff, even though #304 only ever added the
cm.contentEncoding block inside the function body.
Restore writeMetadataCachedResponse's doc and signature exactly as on
main and make it a delegate that passes an empty encoding to a new
unexported writeMetadataCachedResponseWithEncoding, which carries the
original body with #304's block reading the parameter instead of the
cache row. proxyCachedWithEncoding calls the sibling with the encoding
returned alongside the body. helm.go and maven.go drop out of the diff;
their behaviour is unchanged (both fetch transparently, so their stored
encoding was always empty). Same split pattern as ProxyCached ->
proxyCachedWithEncoding.
* fix(handler): move the conda gzip change to its own branch
The conda call site in handleRepodata predates #304 and #304 never
touched it, so under the rule that this PR only corrects code and
behaviour #304 introduced it does not belong here. Restore conda.go and
conda_test.go as on main; the conda change continues on a stacked branch
against its own issue.
Replace the conda-route tests with tests that exercise
proxyCachedWithEncoding directly, so this PR still pins its own plumbing:
gzip is requested and the compressed bytes plus Content-Encoding are
cached and replayed (cached and streaming paths), the header survives a
metadata cache write failure, and the stale fallback keeps the stored
encoding.
* fix(homebrew): fetch the JSON API gzip-compressed on both hops
Homebrew (#254) routes every API path through ProxyCached and so, since
#304, fetches formula.jws.json (~33 MB plain, ~5 MB gzip) uncompressed on
every refresh -- the case that motivated #305.
Request gzip for the JSON API via proxyCachedWithEncoding: brew fetches
every API download with curl --compressed and decodes Content-Encoding
itself, so the compressed bytes and header are cached and served as-is
and both hops stay compressed. The analytics endpoints are the one brew
consumer fetched without --compressed; they stay on identity.
* fix(handler): leave Accept-Encoding unset in proxyMetadataStream for an empty value
fetchUpstreamMetadata treats an empty acceptEncoding as 'do not set the
header'; proxyMetadataStream set it unconditionally, which would send an
empty Accept-Encoding line if a caller ever passed . Guard it the same
way so both paths agree. No caller passes today.
* fix(handler): keep the metadata row and blob from describing different bytes
Two ways the cache row could stop describing the stored blob once a
caller requests gzip, both raised by the review of #324:
- cacheMetadataBlob stored the blob and then discarded the
UpsertMetadataCache error. After a successful gzip store and a failed
row write, a later TTL hit or stale fallback served the gzip blob with
the previous row's encoding. On a row-write failure, log it and delete
the blob just written, so the next request refetches instead.
- fetchOrCacheMetadata read the row once up front and reused it for the
stale fallback. A request that read an identity row, lost the upstream
race to a request that stored the gzip blob, and then failed upstream
labelled the new blob with the old row. Re-read the row before falling
back so the encoding matches the blob as it is now.
Both only become harmful with an encoding change, which this branch
introduces; the pre-existing validator-from-row read is tracked
separately.
* Drop unused cachedMeta.contentEncoding and fix stale doc reference
The field was added by #304 and its only reader is replaced in this
branch by the encoding parameter passed alongside the body. The
proxyCachedWithEncoding comment named conda repodata, which was moved
out of this branch in 5991d95; Homebrew is the caller that ships here.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Use shared artifacts at cache boundaries
* Address artifact cache review
* Defer cached artifact validation to checkCache
Construct artifacts.Artifact from the row without validation so a
malformed content hash reaches newIntegrityChecks in checkCache, which
clears the row and treats the request as a miss. Erroring at the DB
boundary instead surfaced a 500 and left the bad row in place.
* Build stored Artifact after digest and scan checks
Construct the shared artifact struct from trusted storage output as a
literal, after the hash-mismatch and scanner paths that delete failed
downloads, so no path between Store and updateCacheDB can leave bytes
in storage without a database row.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes#183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Runs fetched artifacts through pluggable external scanners after they're
staged in storage but before they're committed to the cache DB, so a
block verdict deletes the object instead of ever exposing it to a
client. Scanners pull the staged bytes themselves via a short-lived
HMAC-signed internal route rather than the proxy pushing bytes to them,
keeping the mechanism storage-backend-agnostic and avoiding uploading
potentially huge artifacts through the proxy's own egress.
Hardening baked in from the start: the internal scan-fetch route is
gated both at router-mount time and in the handler so it's inert
whenever scanning is disabled or unsigned; the signing key is mandatory
whenever scanning is enabled, enforced directly in scanner.NewGroup
rather than relying on callers to invoke config validation; the scan
call and the delete-on-block cleanup both run on a context detached
from the client's, so a client disconnecting mid-scan can't be mistaken
for a scanner failure, doesn't cause a legitimate artifact to be
deleted, and doesn't leave a genuinely blocked artifact's bytes
orphaned in storage; and scanner infrastructure errors (connection
failures, internal hostnames) are never forwarded verbatim to anonymous
clients, only a generic message. The scan-error metric also correctly
distinguishes a scanner's own timeout from being cancelled because a
sibling scanner already decided the verdict.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(handler): preserve upstream content-encoding for cached metadata
Signed and hash-pinned index files (debian Release/Packages.gz, rpm
repomd.xml, helm index.yaml, conda repodata.json, apk APKINDEX.tar.gz)
were cached after Go's transport transparently decompressed any
Content-Encoding: gzip response, so the proxy served bytes that differ
from what the upstream signed and broke client verification.
Request the identity encoding on the shared metadata fetch so Go no
longer auto-decompresses, and persist the upstream Content-Encoding in
a new metadata_cache column so both the cached and offline responses
replay the exact bytes and header. The uncached streaming path now
forwards Content-Encoding too.
Fixes#300
* fix(handler): scope verbatim metadata fetch to the ProxyCached path
The first cut forced Accept-Encoding: identity in the shared
fetchUpstreamMetadata, which an adversarial review showed both missed
the bug and regressed unrelated ecosystems:
- proxyMetadataStream (the default path, since cache_metadata is off)
never forced identity, so a client sending no Accept-Encoding still
triggered Go's transparent gzip decompression and served altered bytes
for apk/debian/rpm/conda indexes.
- Direct FetchOrCacheMetadata callers that parse or rewrite the body
(npm, pypi, cargo, composer, pub, nuget, swift, maven, helm) were
forced to identity too, losing wire compression and 502-ing against
upstreams that ignore identity and gzip anyway (helm rewriteIndex).
- Rows cached before the fix kept serving decompressed bytes via ETag
304 revalidation.
Scope the verbatim behavior to the ProxyCached code path, which serves
upstream bytes through unchanged (apk, debian, rpm, go, hex, conda,
cran, gem, conan, julia). That path now requests identity on both the
cached fetch and the uncached stream branch and replays Content-Encoding;
direct callers keep transparent compression, matching main. Migration
008 clears etag/fetched_at so legacy rows refetch once with identity.
Tests now exercise the stream path with no client Accept-Encoding and
pin that direct callers are not forced to identity.
goconst tripped on main after #259 landed on top of #280: five
composite-literal occurrences each of "Content-Length" and
"Content-Type" across container.go, container_manifest.go,
container_tags.go, handler.go, and swift.go crossed the
min-occurrences: 5 threshold. Neither PR hit it alone.
Add headerContentType and headerContentLength beside
headerAcceptEncoding and use them throughout the package rather than
only at the flagged sites, so the next handler that adds one does not
re-trip the check.
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
* Add ECR auto-refreshing upstream authentication
- Add "ecr" auth type to upstream.auth config with optional region
- Cache ecr:GetAuthorizationToken results per region and refresh
shortly before expiry via the AWS SDK default credential chain
- Route type: ecr through the token cache in Server.authForURL
- Document the new type in config.example.yaml and docs/configuration.md
Fixes#276
* Collapse ecrTokens.header to a single return path
Drops the internal/server package below the goconst min-occurrences
threshold for the Authorization literal.
* Coalesce concurrent ECR token fetches with singleflight
Concurrent cache misses for the same region now share a single
GetAuthorizationToken call instead of each issuing their own, avoiding
a request burst against the ECR API at cold start and at each 12-hour
refresh. golang.org/x/sync is already a direct dependency.
* Improve ECR token refresh and region inference
* Back off failed ECR token refreshes
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Consult versions.published_at before fetching the packument, and persist
the parsed time after the packument fallback, so each version's metadata
is fetched and parsed at most once
- Add DB.SetVersionPublishedAt, an upsert that writes only the publish time
- Preserve a stored published_at in UpsertVersion when the incoming value
is NULL, so the artifact-cache upsert cannot erase it
- Add handler tests for stored-time downloads and single-fetch behavior,
and a database test for preserve-on-NULL in both dialects
- Request application/json from the npm upstream when the option is set,
independent of cooldown, so served packuments carry the "time" map
- Wire the option through the shared Proxy struct and the
PROXY_UPSTREAM_NPM_FULL_METADATA environment override
- Document it in config.example.yaml and docs/configuration.md
- Test that the option forces full metadata with cooldown disabled
* Add GCS storage backend with Workload Identity support
Register gocloud.dev/blob/gcsblob so gs:// URLs are accepted as a storage
backend. Authentication uses Application Default Credentials, which makes
GKE Workload Identity work out of the box; signed URLs (direct_serve)
fall back to the IAM Credentials signBlob API when no private key is
available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Replace gcsblob with lightweight GCS backend
* Extract GCS client into standalone module
---------
Co-authored-by: Anthony A. <github@anthony-arnaud.fr>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
* Bump go tool golangci-lint to v2.13.1
The .golangci.yml goconst.ignore-tests setting was added in v2.12.0
(golangci/golangci-lint#6480). On the previously pinned v2.10.1,
config verify fails with "additional properties 'ignore-tests' not
allowed" and the setting is silently ignored at run time, so goconst
counts test-file literals toward min-occurrences.
* Apply gofmt and CutSuffix simplification
- gofmt -w internal/server/health_test.go
- Replace HasSuffix+TrimSuffix with CutSuffix in ParseSize
* Remove dead code and migrate tests off legacy Filesystem storage
Migrate the three test call sites of storage.NewFilesystem to
storage.OpenBucket("file://...") and drop the deprecated
StorageConfig.Path field from test configs, then delete code that
deadcode reports as unreachable from cmd/proxy:
- internal/storage/filesystem.go and its tests
- storage.HashingReader
- enrichment.Service.BulkCheckVulnerabilities and NormalizeLicense
- server.ActiveRequestsMiddleware (no-op body; the real tracking
is the inline r.Use at server.go:226)
- mirror.RegistrySource (unimplemented stub)
metrics.UpdateCircuitBreakerState and RecordCircuitBreakerTrip are
kept because #275 wires them.
Update the CONTRIBUTING.md storage section to reflect blob.go.
* Show build information in web UI
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
* Fix footer build info shadowed by page Version fields
Shared footer templates were reading .Version and .Commit, which resolve
to package data on VersionShowData and BrowseSourceData. Point the footer
at Layout.BuildInfo and cover both pages so the proxy version stays visible.
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Layout build info field promotion
---------
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
MakePURL/MakePURLString/New now apply the same per-type normalization as
Parse (git-pkgs/purl#30), so canonicalPackagePURL no longer needs its own
Normalize call and DB writes/lookups produce canonical keys.
Existing rows written under a non-canonical purl (mixed-case pypi,
composer, etc) become cache misses on lookup and re-populate under the
canonical key on the next fetch; the old rows are left in place.
Closes#207
Cooldown filtering only ran when rewriting metadata, so a version could
be missing from the npm packument and the PyPI simple index while its
tarball stayed reachable. Lockfiles record artifact URLs verbatim, so
npm ci and pinned pip requirements reach handleDownload without ever
requesting metadata.
The shared artifact path has no publish time to check against, since
updateCacheDB upserts versions without PublishedAt and the column is
only set by enrichment. Each handler now resolves the publish time from
metadata it already fetches and returns 404 while a version is inside
the window. Versions with no usable publish time are still served, as
they are when filtering metadata.
When cooldown is disabled, send:
Accept: application/vnd.npm.install-v1+json;q=1.0, application/json;q=0.8
This allows upstreams like JFrog Artifactory that return 406 for the
abbreviated packument type to fall back to full JSON metadata, while
letting the public npm registry continue to serve the smaller
abbreviated format it prefers.
When cooldown is enabled, keep sending only application/json because
the abbreviated format omits the "time" map required for version age
filtering.
Fixes#228
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Refactor LoadFromEnv to use helpers
Avoid triggering the linter about the cyclomatic complexity of the LoadFromEnv
function in later changes by refactoring it to use setEnvString/setEnvBool
helpers.
No functional change, just collapse ~29 repetitive if-blocks into single-line
calls to two small helpers.
* Make Debian upstream repository configurable
Support overriding the Debian handler's upstream (e.g. Ubuntu archives)
via PROXY_UPSTREAM_DEBIAN or upstream.debian in the config file.
Tested with PROXY_UPSTREAM_DEBIAN=http://archive.ubuntu.com/ubuntu
to get Ubuntu Resolute packages.
* resolve name and version for PEP 658 metadata sidecars
* fix(pypi): parse Windows installer and egg filenames separately
The bdist_wininst and bdist_msi layout joins the platform to the version
with a '.' rather than a '-', so treating .exe/.msi like a wheel folded
the platform into the version: foo-1.0.win32-py2.0.exe resolved to
version "1.0.win32". Eggs shared the problem, as setuptools' hyphen
escaping is not universal: aws-sdk-1.0.0-py3.11.egg resolved to name
"aws", version "sdk".
Give each format its own parser. Wheels keep the PEP 427
spec-guaranteed field positions, eggs locate the version relative to the
py{X.Y} interpreter field, and Windows installers strip the platform and
interpreter fields before splitting name from version.
A PEP 658 sidecar resolves to the same name and version as the
distribution it describes, so it is cached under that version. Browse and
compare took the first cached artifact without checking its extension,
handing openArchive plain text: a version pip had only fetched metadata
for reported hasCached and then 500'd.
Add firstBrowsableArtifact, replacing five duplicated selection loops,
and export PyPIMetadataSuffix so the suffix has a single definition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
