proxy list and proxy delete sat next to serve, stats and mirror and read as cache operations. The commands are now proxy hosted push, proxy hosted list and proxy hosted delete, before a release makes the names hard to change. proxy hosted without a known subcommand prints the three.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
proxy push, proxy list and proxy delete call the /api/hosted endpoints of a running proxy. The proxy URL comes from -url or PROXY_URL and the token from PROXY_HOSTED_TOKEN, so it never ends up in shell history or process listings.
Document the hosted config block, the endpoints, the supported ecosystems and how pushed archives interact with eviction.
Refs #133
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
go install does not apply goreleaser's -ldflags -X, so binaries
installed that way reported 'dev'. Read debug.BuildInfo.Main.Version
when the ldflag is unset.
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
* Show build information in web UI
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
* Fix footer build info shadowed by page Version fields
Shared footer templates were reading .Version and .Commit, which resolve
to package data on VersionShowData and BrowseSourceData. Point the footer
at Layout.BuildInfo and cover both pages so the proxy version stays visible.
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Layout build info field promotion
---------
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* config: add Health.StorageProbeInterval
* metrics: add proxy_health_probe_failures_total counter
* server: add storageProbe with happy-path test
* server: add storageProbe failure-mode tests
* server: add healthCache with TTL, single-flight, transition logging
* server: wire storage probe into /health
* server: update TestHealthEndpoint for JSON; wire healthCache into newTestServer
Also fix Windows file-locking issue in storageProbe: close the reader
explicitly before Delete so the file handle is released prior to os.Remove.
* server: clean up stale comment in storageProbe
* docs: document storage health probe and new metric
* docs: regenerate Swagger for /health JSON response
* server: simplify rc.Close error handling in storageProbe
* server: defer probe cleanup so size/open/read/verify failures don't leak objects
Previously, storageProbe only called Delete on the success path. Any
failure between Store and the final Delete (size mismatch, Open error,
mid-stream read failure, content mismatch) left the probe object orphaned
in the storage backend. With caching disabled and Kubernetes-rate probing,
the leak could accumulate noticeably on backends like S3.
Use a named return + defer to attempt Delete after every successful Store.
The earlier-step failure remains the primary error; Delete failure only
surfaces as step="delete" when nothing else went wrong. Add a table-driven
test that asserts cleanup runs for each non-delete failure path.
Reported by Copilot on #119.
* config: validate health.storage_probe_interval in Config.Validate
The new duration field was only validated at use time in newHealthCache.
The existing codebase already validates other duration fields
(MetadataTTL, DirectServeTTL, Gradle.MaxAge, Gradle.SweepInterval) in
Config.Validate() so misconfiguration fails fast at startup with a
config-key-specific error.
Match that pattern. The parse-at-use code in newHealthCache stays as
a safety net, mirroring the MetadataTTL precedent.
Reported by Copilot on #119.
* docs: lowercase "counter" in metrics table for consistency
Other rows in the table use lowercase type names (counter/gauge/histogram).
Match that style.
Reported by Copilot on #119.
* docs: include size-check step in /health probe description
The probe is write → size-check → read → verify → delete; the
architecture note was missing the size-check step.
Reported by Copilot on #119.
* server: address andrew's review on #119
- Drop unused callerCtx parameter from healthCache.Check (Check is now
parameter-less; the comment-only "accepted for symmetry" justification
wasn't carrying its weight).
- Emit "storage": {"status": "skipped"} on DB short-circuit instead of
omitting the key, so monitors expecting a fixed key set keep working.
- Reject negative storage_probe_interval at config validation time
(previously parsed and silently behaved like "0").
- Extract HealthConfig.Validate to keep Config.Validate under the
gocognit threshold and match the existing GradleBuildCacheConfig pattern.
- README Health Check section: note that /health is intended as a
readiness probe rather than a liveness probe (Check holds a mutex
for up to the 10s probe timeout).
- cmd/proxy/main.go godoc: column-align the new env var with the
surrounding Gradle entries.
Reported by andrew on #119.
* add Gradle Build Cache support with handler and tests
* linting issue
* MR Suggestions: Add Gradle HTTP Build Cache configuration to README
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
Co-authored-by: Copilot <copilot@github.com>
* Add Gradle build cache configuration and eviction support
- Introduced configuration options for Gradle build cache in config files and documentation.
- Implemented read-only mode and upload size limits for the Gradle build cache.
- Added cache eviction logic based on age and size, with corresponding tests.
- Enhanced storage interfaces to support listing objects by prefix.
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
* last finding fix
* fix tests and implement PR suggestions
Co-authored-by: Copilot <copilot@github.com>
* unify path
---------
Co-authored-by: Mateusz (Mati) Kepa <m.kepa@sportradar.com>
Co-authored-by: Copilot <copilot@github.com>
Cached metadata is now served directly within a configurable TTL window
(default 5m) without contacting upstream, reducing latency and upstream
load. When upstream is unreachable and the cache is past its TTL, stale
content is served with a Warning: 110 header per RFC 7234.
New config: `metadata_ttl` (YAML) / `PROXY_METADATA_TTL` (env).
Set to "0" to always revalidate with upstream.
- ProxyCached now stores upstream Last-Modified in the cache and uses it
(along with ETag) for conditional request handling, returning 304 when
client validators match. Adds Content-Length to cached responses.
- Handlers calling FetchOrCacheMetadata (pypi, composer, pub, nuget) now
check for ErrUpstreamNotFound and return 404 instead of 502, matching
the existing npm and cargo behavior.
- Mirror jobs report live progress via a periodic callback while running,
so API polls return real counts instead of zeroed progress.
- Registry mirroring removed from CLI flags, API acceptance, README, and
docs since every enumerator was a stub returning "not yet implemented".
- Added tests for the conditional metadata path (ETag/If-None-Match,
Last-Modified/If-Modified-Since, 304 responses, header omission).
- Wire job contexts to server shutdown context so jobs are canceled on
server stop instead of running indefinitely
- Defer context cancel in runJob so completed jobs don't leak contexts
- Cap error accumulation in progressTracker to 1000 entries to prevent
OOM on large mirror operations with many failures
- Add panic recovery in errgroup workers to prevent process crashes
- Use defer for db.Close() in runMirror CLI to ensure cleanup on all
error paths
Add a `proxy mirror` CLI command and `/api/mirror` API endpoints that
pre-populate the cache from various input sources: individual PURLs,
SBOM files (CycloneDX and SPDX), or full registry enumeration.
The mirror reuses the existing handler.Proxy.GetOrFetchArtifact()
pipeline so cached artifacts are identical to those fetched on demand.
A bounded worker pool controls download parallelism.
Metadata caching is opt-in via `cache_metadata: true` in config (or
PROXY_CACHE_METADATA=true). The mirror command always enables it. When
enabled, upstream metadata responses are stored for offline fallback
with ETag-based conditional revalidation.
New internal/mirror package with Source interface, PURLSource,
SBOMSource, RegistrySource, and async JobStore. New metadata_cache
database table for offline metadata serving.
* Fix all golangci-lint issues across the codebase
Resolve 77 lint issues reported by golangci-lint with gocritic, gocognit,
gocyclo, maintidx, dupl, mnd, unparam, ireturn, goconst, and errcheck
enabled. Net reduction of ~175 lines through shared helpers and
deduplication.
* Suppress staticcheck SA1019 for intentional deprecated field usage
The Storage.Path field is deprecated but still read for backwards
compatibility with existing configs that haven't migrated to the URL field.
Replace raw database/sql with jmoiron/sqlx for cleaner query handling.
Support both SQLite (default) and PostgreSQL as configurable backends.
Configuration via:
- CLI flags: -database-driver, -database-path, -database-url
- Environment: PROXY_DATABASE_DRIVER, PROXY_DATABASE_PATH, PROXY_DATABASE_URL
- Config file: database.driver, database.path, database.url
Tests run against both databases when PROXY_DATABASE_URL is set.
