* Use shared artifacts at cache boundaries
* Address artifact cache review
* Defer cached artifact validation to checkCache
Construct artifacts.Artifact from the row without validation so a
malformed content hash reaches newIntegrityChecks in checkCache, which
clears the row and treats the request as a miss. Erroring at the DB
boundary instead surfaced a 500 and left the bad row in place.
* Build stored Artifact after digest and scan checks
Construct the shared artifact struct from trusted storage output as a
literal, after the hash-mismatch and scanner paths that delete failed
downloads, so no path between Store and updateCacheDB can leave bytes
in storage without a database row.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes#183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Runs fetched artifacts through pluggable external scanners after they're
staged in storage but before they're committed to the cache DB, so a
block verdict deletes the object instead of ever exposing it to a
client. Scanners pull the staged bytes themselves via a short-lived
HMAC-signed internal route rather than the proxy pushing bytes to them,
keeping the mechanism storage-backend-agnostic and avoiding uploading
potentially huge artifacts through the proxy's own egress.
Hardening baked in from the start: the internal scan-fetch route is
gated both at router-mount time and in the handler so it's inert
whenever scanning is disabled or unsigned; the signing key is mandatory
whenever scanning is enabled, enforced directly in scanner.NewGroup
rather than relying on callers to invoke config validation; the scan
call and the delete-on-block cleanup both run on a context detached
from the client's, so a client disconnecting mid-scan can't be mistaken
for a scanner failure, doesn't cause a legitimate artifact to be
deleted, and doesn't leave a genuinely blocked artifact's bytes
orphaned in storage; and scanner infrastructure errors (connection
failures, internal hostnames) are never forwarded verbatim to anonymous
clients, only a generic message. The scan-error metric also correctly
distinguishes a scanner's own timeout from being cancelled because a
sibling scanner already decided the verdict.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(handler): preserve upstream content-encoding for cached metadata
Signed and hash-pinned index files (debian Release/Packages.gz, rpm
repomd.xml, helm index.yaml, conda repodata.json, apk APKINDEX.tar.gz)
were cached after Go's transport transparently decompressed any
Content-Encoding: gzip response, so the proxy served bytes that differ
from what the upstream signed and broke client verification.
Request the identity encoding on the shared metadata fetch so Go no
longer auto-decompresses, and persist the upstream Content-Encoding in
a new metadata_cache column so both the cached and offline responses
replay the exact bytes and header. The uncached streaming path now
forwards Content-Encoding too.
Fixes#300
* fix(handler): scope verbatim metadata fetch to the ProxyCached path
The first cut forced Accept-Encoding: identity in the shared
fetchUpstreamMetadata, which an adversarial review showed both missed
the bug and regressed unrelated ecosystems:
- proxyMetadataStream (the default path, since cache_metadata is off)
never forced identity, so a client sending no Accept-Encoding still
triggered Go's transparent gzip decompression and served altered bytes
for apk/debian/rpm/conda indexes.
- Direct FetchOrCacheMetadata callers that parse or rewrite the body
(npm, pypi, cargo, composer, pub, nuget, swift, maven, helm) were
forced to identity too, losing wire compression and 502-ing against
upstreams that ignore identity and gzip anyway (helm rewriteIndex).
- Rows cached before the fix kept serving decompressed bytes via ETag
304 revalidation.
Scope the verbatim behavior to the ProxyCached code path, which serves
upstream bytes through unchanged (apk, debian, rpm, go, hex, conda,
cran, gem, conan, julia). That path now requests identity on both the
cached fetch and the uncached stream branch and replays Content-Encoding;
direct callers keep transparent compression, matching main. Migration
008 clears etag/fetched_at so legacy rows refetch once with identity.
Tests now exercise the stream path with no client Accept-Encoding and
pin that direct callers are not forced to identity.
goconst tripped on main after #259 landed on top of #280: five
composite-literal occurrences each of "Content-Length" and
"Content-Type" across container.go, container_manifest.go,
container_tags.go, handler.go, and swift.go crossed the
min-occurrences: 5 threshold. Neither PR hit it alone.
Add headerContentType and headerContentLength beside
headerAcceptEncoding and use them throughout the package rather than
only at the flagged sites, so the next handler that adds one does not
re-trip the check.
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
* Add ECR auto-refreshing upstream authentication
- Add "ecr" auth type to upstream.auth config with optional region
- Cache ecr:GetAuthorizationToken results per region and refresh
shortly before expiry via the AWS SDK default credential chain
- Route type: ecr through the token cache in Server.authForURL
- Document the new type in config.example.yaml and docs/configuration.md
Fixes#276
* Collapse ecrTokens.header to a single return path
Drops the internal/server package below the goconst min-occurrences
threshold for the Authorization literal.
* Coalesce concurrent ECR token fetches with singleflight
Concurrent cache misses for the same region now share a single
GetAuthorizationToken call instead of each issuing their own, avoiding
a request burst against the ECR API at cold start and at each 12-hour
refresh. golang.org/x/sync is already a direct dependency.
* Improve ECR token refresh and region inference
* Back off failed ECR token refreshes
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Consult versions.published_at before fetching the packument, and persist
the parsed time after the packument fallback, so each version's metadata
is fetched and parsed at most once
- Add DB.SetVersionPublishedAt, an upsert that writes only the publish time
- Preserve a stored published_at in UpsertVersion when the incoming value
is NULL, so the artifact-cache upsert cannot erase it
- Add handler tests for stored-time downloads and single-fetch behavior,
and a database test for preserve-on-NULL in both dialects
- Request application/json from the npm upstream when the option is set,
independent of cooldown, so served packuments carry the "time" map
- Wire the option through the shared Proxy struct and the
PROXY_UPSTREAM_NPM_FULL_METADATA environment override
- Document it in config.example.yaml and docs/configuration.md
- Test that the option forces full metadata with cooldown disabled
* Add GCS storage backend with Workload Identity support
Register gocloud.dev/blob/gcsblob so gs:// URLs are accepted as a storage
backend. Authentication uses Application Default Credentials, which makes
GKE Workload Identity work out of the box; signed URLs (direct_serve)
fall back to the IAM Credentials signBlob API when no private key is
available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Replace gcsblob with lightweight GCS backend
* Extract GCS client into standalone module
---------
Co-authored-by: Anthony A. <github@anthony-arnaud.fr>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
* Bump go tool golangci-lint to v2.13.1
The .golangci.yml goconst.ignore-tests setting was added in v2.12.0
(golangci/golangci-lint#6480). On the previously pinned v2.10.1,
config verify fails with "additional properties 'ignore-tests' not
allowed" and the setting is silently ignored at run time, so goconst
counts test-file literals toward min-occurrences.
* Apply gofmt and CutSuffix simplification
- gofmt -w internal/server/health_test.go
- Replace HasSuffix+TrimSuffix with CutSuffix in ParseSize
* Remove dead code and migrate tests off legacy Filesystem storage
Migrate the three test call sites of storage.NewFilesystem to
storage.OpenBucket("file://...") and drop the deprecated
StorageConfig.Path field from test configs, then delete code that
deadcode reports as unreachable from cmd/proxy:
- internal/storage/filesystem.go and its tests
- storage.HashingReader
- enrichment.Service.BulkCheckVulnerabilities and NormalizeLicense
- server.ActiveRequestsMiddleware (no-op body; the real tracking
is the inline r.Use at server.go:226)
- mirror.RegistrySource (unimplemented stub)
metrics.UpdateCircuitBreakerState and RecordCircuitBreakerTrip are
kept because #275 wires them.
Update the CONTRIBUTING.md storage section to reflect blob.go.
* Show build information in web UI
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
* Fix footer build info shadowed by page Version fields
Shared footer templates were reading .Version and .Commit, which resolve
to package data on VersionShowData and BrowseSourceData. Point the footer
at Layout.BuildInfo and cover both pages so the proxy version stays visible.
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Layout build info field promotion
---------
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
MakePURL/MakePURLString/New now apply the same per-type normalization as
Parse (git-pkgs/purl#30), so canonicalPackagePURL no longer needs its own
Normalize call and DB writes/lookups produce canonical keys.
Existing rows written under a non-canonical purl (mixed-case pypi,
composer, etc) become cache misses on lookup and re-populate under the
canonical key on the next fetch; the old rows are left in place.
Closes#207
Cooldown filtering only ran when rewriting metadata, so a version could
be missing from the npm packument and the PyPI simple index while its
tarball stayed reachable. Lockfiles record artifact URLs verbatim, so
npm ci and pinned pip requirements reach handleDownload without ever
requesting metadata.
The shared artifact path has no publish time to check against, since
updateCacheDB upserts versions without PublishedAt and the column is
only set by enrichment. Each handler now resolves the publish time from
metadata it already fetches and returns 404 while a version is inside
the window. Versions with no usable publish time are still served, as
they are when filtering metadata.
When cooldown is disabled, send:
Accept: application/vnd.npm.install-v1+json;q=1.0, application/json;q=0.8
This allows upstreams like JFrog Artifactory that return 406 for the
abbreviated packument type to fall back to full JSON metadata, while
letting the public npm registry continue to serve the smaller
abbreviated format it prefers.
When cooldown is enabled, keep sending only application/json because
the abbreviated format omits the "time" map required for version age
filtering.
Fixes#228
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Refactor LoadFromEnv to use helpers
Avoid triggering the linter about the cyclomatic complexity of the LoadFromEnv
function in later changes by refactoring it to use setEnvString/setEnvBool
helpers.
No functional change, just collapse ~29 repetitive if-blocks into single-line
calls to two small helpers.
* Make Debian upstream repository configurable
Support overriding the Debian handler's upstream (e.g. Ubuntu archives)
via PROXY_UPSTREAM_DEBIAN or upstream.debian in the config file.
Tested with PROXY_UPSTREAM_DEBIAN=http://archive.ubuntu.com/ubuntu
to get Ubuntu Resolute packages.
* resolve name and version for PEP 658 metadata sidecars
* fix(pypi): parse Windows installer and egg filenames separately
The bdist_wininst and bdist_msi layout joins the platform to the version
with a '.' rather than a '-', so treating .exe/.msi like a wheel folded
the platform into the version: foo-1.0.win32-py2.0.exe resolved to
version "1.0.win32". Eggs shared the problem, as setuptools' hyphen
escaping is not universal: aws-sdk-1.0.0-py3.11.egg resolved to name
"aws", version "sdk".
Give each format its own parser. Wheels keep the PEP 427
spec-guaranteed field positions, eggs locate the version relative to the
py{X.Y} interpreter field, and Windows installers strip the platform and
interpreter fields before splitting name from version.
A PEP 658 sidecar resolves to the same name and version as the
distribution it describes, so it is cached under that version. Browse and
compare took the first cached artifact without checking its extension,
handing openArchive plain text: a version pip had only fetched metadata
for reported hasCached and then 500'd.
Add firstBrowsableArtifact, replacing five duplicated selection loops,
and export PyPIMetadataSuffix so the suffix has a single definition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
NormalizedPackages runs config keys through purl.Parse().String(), which
applies per-type rules like lowercasing pypi names. The handler side was
building lookup keys with MakePURLString, which does not, so a config
entry for pkg:pypi/Django would be normalized to pkg:pypi/django and
never match the runtime key pkg:pypi/Django.
Route both sides through the same canonical form: a new
canonicalPackagePURL helper calls Normalize() on the constructed PURL
before stringifying, and all cooldown IsAllowed call sites use it.
Return the repository-standard not-found response when an upstream module is absent.
Constraint: GOPROXY advances to direct only after 404 or 410 responses.
Rejected: Preserve a handler-specific response body | sibling handlers consistently use not found.
Confidence: high
Scope-risk: narrow
Directive: Keep missing-artifact responses consistent across registry handlers.
Tested: go test ./internal/handler -run TestGoModuleDownloadUpstreamErrors -count=1; prior go test -race ./...; go build ./...; golangci-lint; go vet.
Add http_timeout config option (and PROXY_HTTP_TIMEOUT env var) to set
the per-request timeout on the shared HTTP client used by protocol
handlers for upstream metadata fetches and pass-through file requests.
Defaults to the previous hardcoded 30s; "0" disables the timeout.
Fixes#187
- Expand minified Composer v2 metadata in findDownloadURLFromMetadata
so versions that inherit dist from a previous entry resolve correctly
- Add Debug()-level log statements in the download resolution path
(handleDownload, findDownloadURLFromMetadata) so -log-level debug
produces meaningful output
The startup log and /stats endpoint always reported Database.Path,
which is the sqlite default even when PROXY_DATABASE_DRIVER=postgres
is set and a postgres URL is in use. Add DatabaseConfig.String() that
returns the sqlite path or the postgres URL with the password redacted,
and use it in both places.
Fixes#173
- Move third-party JS into static/vendor/ and add lucide for icons; refine
.gitignore so embedded vendor dirs aren't caught by the Go vendor rule.
- Replace folder/file emojis in the source browser with lucide icons.
- Wrap the header logo and title in a single anchor so the icon is clickable.
- Drop the redundant "Powered by git-pkgs" footer block; add a GitHub repo
link to the About column and bump the ecosystem count from 16+ to 17+.
- Sticky footer pattern: body is min-h-full flex column with main growing to
fill, so the footer sits at the bottom of short pages.
- Hamburger menu under md: search and nav links collapse into a drawer
toggled by a menu button; theme toggle stays visible at both sizes.
