Eviction deleted an object before checking its record still pointed at
it. If a refetch had moved the record, that object was queued and a
request that read the record earlier could still be opening it. Eviction
now clears first, skips the delete if the record moved, and queues the
object when the delete fails after a clear.
Reclaim queues a failed delete again, which moves it behind the rest, so
objects the backend keeps refusing cannot fill every batch.
Also drop the clearArtifactCache passthrough and ClearCachedArtifact's
unused context, note that only tests call ArtifactPath, and document that
max_size does not count objects waiting to be reclaimed.
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes#327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes#183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Runs fetched artifacts through pluggable external scanners after they're
staged in storage but before they're committed to the cache DB, so a
block verdict deletes the object instead of ever exposing it to a
client. Scanners pull the staged bytes themselves via a short-lived
HMAC-signed internal route rather than the proxy pushing bytes to them,
keeping the mechanism storage-backend-agnostic and avoiding uploading
potentially huge artifacts through the proxy's own egress.
Hardening baked in from the start: the internal scan-fetch route is
gated both at router-mount time and in the handler so it's inert
whenever scanning is disabled or unsigned; the signing key is mandatory
whenever scanning is enabled, enforced directly in scanner.NewGroup
rather than relying on callers to invoke config validation; the scan
call and the delete-on-block cleanup both run on a context detached
from the client's, so a client disconnecting mid-scan can't be mistaken
for a scanner failure, doesn't cause a legitimate artifact to be
deleted, and doesn't leave a genuinely blocked artifact's bytes
orphaned in storage; and scanner infrastructure errors (connection
failures, internal hostnames) are never forwarded verbatim to anonymous
clients, only a generic message. The scan-error metric also correctly
distinguishes a scanner's own timeout from being cancelled because a
sibling scanner already decided the verdict.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
* Add ECR auto-refreshing upstream authentication
- Add "ecr" auth type to upstream.auth config with optional region
- Cache ecr:GetAuthorizationToken results per region and refresh
shortly before expiry via the AWS SDK default credential chain
- Route type: ecr through the token cache in Server.authForURL
- Document the new type in config.example.yaml and docs/configuration.md
Fixes#276
* Collapse ecrTokens.header to a single return path
Drops the internal/server package below the goconst min-occurrences
threshold for the Authorization literal.
* Coalesce concurrent ECR token fetches with singleflight
Concurrent cache misses for the same region now share a single
GetAuthorizationToken call instead of each issuing their own, avoiding
a request burst against the ECR API at cold start and at each 12-hour
refresh. golang.org/x/sync is already a direct dependency.
* Improve ECR token refresh and region inference
* Back off failed ECR token refreshes
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Request application/json from the npm upstream when the option is set,
independent of cooldown, so served packuments carry the "time" map
- Wire the option through the shared Proxy struct and the
PROXY_UPSTREAM_NPM_FULL_METADATA environment override
- Document it in config.example.yaml and docs/configuration.md
- Test that the option forces full metadata with cooldown disabled
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
NormalizedPackages runs config keys through purl.Parse().String(), which
applies per-type rules like lowercasing pypi names. The handler side was
building lookup keys with MakePURLString, which does not, so a config
entry for pkg:pypi/Django would be normalized to pkg:pypi/django and
never match the runtime key pkg:pypi/Django.
Route both sides through the same canonical form: a new
canonicalPackagePURL helper calls Normalize() on the constructed PURL
before stringifying, and all cooldown IsAllowed call sites use it.
Add http_timeout config option (and PROXY_HTTP_TIMEOUT env var) to set
the per-request timeout on the shared HTTP client used by protocol
handlers for upstream metadata fetches and pass-through file requests.
Defaults to the previous hardcoded 30s; "0" disables the timeout.
Fixes#187
Adds UIBaseURL (env PROXY_UI_URL), advertised separately from BaseURL
for deployments where the UI is reached on a public domain while build
machines hit a Docker network alias for the package endpoints. Defaults
to BaseURL when unset.
Consumers:
- Logged alongside base_url at startup.
- <link rel="canonical"> and og:url / og:title / og:site_name in every
UI page, omitted when UIBaseURL is empty.
- Banner on the install guide when UIBaseURL differs from BaseURL,
clarifying that the URLs in the snippets are the package endpoint and
that the UI itself lives elsewhere.
Docs call out that the proxy serves UI and package endpoints on the same
listener, so reverse proxies fronting the UI publicly must restrict the
public route to PathPrefix(/ui) to avoid exposing /npm, /pypi, etc.
nginx and Traefik examples both show the path-split pattern.
* add Gradle Build Cache support with handler and tests
* linting issue
* MR Suggestions: Add Gradle HTTP Build Cache configuration to README
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
Co-authored-by: Copilot <copilot@github.com>
* Add Gradle build cache configuration and eviction support
- Introduced configuration options for Gradle build cache in config files and documentation.
- Implemented read-only mode and upload size limits for the Gradle build cache.
- Added cache eviction logic based on age and size, with corresponding tests.
- Enhanced storage interfaces to support listing objects by prefix.
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
* last finding fix
* fix tests and implement PR suggestions
Co-authored-by: Copilot <copilot@github.com>
* unify path
---------
Co-authored-by: Mateusz (Mati) Kepa <m.kepa@sportradar.com>
Co-authored-by: Copilot <copilot@github.com>
Cached metadata is now served directly within a configurable TTL window
(default 5m) without contacting upstream, reducing latency and upstream
load. When upstream is unreachable and the cache is past its TTL, stale
content is served with a Warning: 110 header per RFC 7234.
New config: `metadata_ttl` (YAML) / `PROXY_METADATA_TTL` (env).
Set to "0" to always revalidate with upstream.
- ProxyCached now stores upstream Last-Modified in the cache and uses it
(along with ETag) for conditional request handling, returning 304 when
client validators match. Adds Content-Length to cached responses.
- Handlers calling FetchOrCacheMetadata (pypi, composer, pub, nuget) now
check for ErrUpstreamNotFound and return 404 instead of 502, matching
the existing npm and cargo behavior.
- Mirror jobs report live progress via a periodic callback while running,
so API polls return real counts instead of zeroed progress.
- Registry mirroring removed from CLI flags, API acceptance, README, and
docs since every enumerator was a stub returning "not yet implemented".
- Added tests for the conditional metadata path (ETag/If-None-Match,
Last-Modified/If-Modified-Since, 304 responses, header omission).
- Fix race where runJob could overwrite canceled state set by Cancel()
- Fix Debian ecosystem name inconsistency ("deb" -> "debian")
- Stream metadata responses when caching is disabled to avoid buffering
- Add metadata_cache table to initial schema strings for consistency
- Gate mirror API behind mirror_api config flag (disabled by default)
- Fix goconst lint in metadata_cache_test.go
Add a `proxy mirror` CLI command and `/api/mirror` API endpoints that
pre-populate the cache from various input sources: individual PURLs,
SBOM files (CycloneDX and SPDX), or full registry enumeration.
The mirror reuses the existing handler.Proxy.GetOrFetchArtifact()
pipeline so cached artifacts are identical to those fetched on demand.
A bounded worker pool controls download parallelism.
Metadata caching is opt-in via `cache_metadata: true` in config (or
PROXY_CACHE_METADATA=true). The mirror command always enables it. When
enabled, upstream metadata responses are stored for offline fallback
with ETag-based conditional revalidation.
New internal/mirror package with Source interface, PURLSource,
SBOMSource, RegistrySource, and async JobStore. New metadata_cache
database table for offline metadata serving.
Decode the Hex registry protobuf format, filter releases by fetching
timestamps from the Hex HTTP API (hex.pm/api/packages/{name}), and
re-encode without the original signature.
The protobuf handling uses protowire for low-level encoding/decoding
of the Signed wrapper, Package, and Release messages. Timestamps come
from the inserted_at field in the JSON API response.
Since the proxy re-encodes the payload without the original signature,
users need to disable registry signature verification.
Filter versions from the compact index (/info/{name}) by fetching
timestamps from the versions API (/api/v1/versions/{name}.json).
Both requests run concurrently to minimize latency. If the versions
API is unavailable, the compact index is proxied unfiltered.
Handles platform-specific versions (e.g. 1.0.0-java) by matching
the compact index format.
* Add cooldown support for Conda
Filter entries from Conda repodata.json based on the timestamp field
(milliseconds since epoch). Filters both packages and packages.conda
sections. When cooldown is disabled, repodata requests are proxied
directly without parsing.
* Update README table to mark Conda cooldown support
* Add cooldown support for NuGet
Filter versions from NuGet registration pages based on the
catalogEntry.published timestamp. Handles both RFC3339 and NuGet's
fractional-second timestamp formats. When cooldown is disabled,
registration requests are proxied directly without parsing.
* Update README table to mark NuGet cooldown support
Hides package versions published too recently from metadata responses,
giving the community time to spot malicious releases. Configurable
per-ecosystem and per-package with duration overrides. Supported for
npm, PyPI, pub.dev, and Composer.
