The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
Whether a packument is the abbreviated format is a property of the
document, so it sits with the rest of the npm metadata code; the
handler keeps only the error it turns that into. Add a test that keeps
the package's imports to the portable set, give jsonscan a package
comment, and describe the new packages in the architecture docs.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
On a cache miss the proxy stored the whole artifact before sending any of
it, so clients with an idle timeout gave up on large artifacts: git-lfs
after 30 seconds, pacman after 10. The caller that starts a shared fetch
now gets the bytes as they arrive while they are stored; requests joining
the fetch, HEAD and range requests still wait for the stored copy.
The client's copy is checked against the upstream's digest as it is read,
and a mismatch aborts the response before it completes. A client going
away leaves the copy to storage running. A storage error leaves the
client's copy running: the client gets the artifact, and the failure is
logged at error level and counted in proxy_artifacts_served_uncached_total.
A block-mode scanner has to see the whole artifact first, so with one
configured for the ecosystem a miss is handled as before.
- Keep the download URLs of a batch response in memory until shortly
before they expire, so the GET that follows needs no second batch
request. A kept URL that fails is replaced by a fresh one.
- Cut off a download that isn't the object's size.
- Copy an object whose content is already stored, such as for a fork,
instead of downloading it, once the upstream has confirmed the
repository has it. Artifacts are indexed by content hash for this.
- Keep signed download URLs out of logs and the database: the fetch uses
the signed URL, everything else the URL without its query.
- Answer a batch from the cache when the upstream can't be reached or
fails, so git-lfs can use cached objects during an outage. Refusals
are still passed on.
- Answer a HEAD for an uncached object from the batch response instead
of downloading it.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
SQLite used one connection for everything, so request lookups queued
behind the per-minute stats queries, dashboard pages and eviction scans.
Writes keep the single connection. Get, Select, Query and QueryRow now go
to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode
lets run alongside the writer. A write sent to that pool fails.
Pragmas move into the DSN so every connection gets them. Postgres and
in-memory SQLite keep one pool for both.
The Helm index.yaml and PyPI simple pages and JSON API documents were
parsed and rewritten on every request, cache hits and conditional
requests included. They now go through the rewrite cache like npm and
Composer, so a repeat request is served from memory with an ETag of
its own, a dated Last-Modified, a gzip copy and 304 when it matches.
PyPI simple pages are keyed by their content type and the versions
filtered out as well as the upstream page. Download hrefs are now
rewritten with a literal prefix search that makes the same changes as
the old regular expression, and the link filter's expressions are
compiled once.
A chart cache miss no longer parses the whole Helm index again while
the stored index is unchanged: the chart downloads it lists are kept
per repository, keyed by the index digest.
Nothing set GOMEMLIMIT, so in a container the heap could grow until the
container was OOM-killed while much of it was garbage, for example while
large metadata documents were being rewritten. At startup on Linux, read
the cgroup v2 or v1 memory limit and set the runtime's soft limit to 80%
of it, leaving room for git subprocesses. An explicit GOMEMLIMIT wins.
A cache hit hashed the stored object with SHA-256 for content_hash and
again with every native algorithm in versions.integrity, SHA-512 for npm.
The native integrity was already checked when the bytes were stored and
content_hash was computed from the same bytes, so the second hash adds
CPU per byte served without catching anything more. Records without a
content hash still fall back to the native integrity.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Replicas sharing a database need one place to agree on the Last-Modified
of rewritten metadata, which process memory cannot give them.
- Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the
ETag being dated and its date in Unix seconds, so no zone is involved
- Insert a package's first date only if no replica recorded one first
- Swap a date only if it is still the one read, so of two replicas
switching a package at once only one succeeds
- Create it in both schemas and in migration 011
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
* fix(handler): preserve upstream content-encoding for cached metadata
Signed and hash-pinned index files (debian Release/Packages.gz, rpm
repomd.xml, helm index.yaml, conda repodata.json, apk APKINDEX.tar.gz)
were cached after Go's transport transparently decompressed any
Content-Encoding: gzip response, so the proxy served bytes that differ
from what the upstream signed and broke client verification.
Request the identity encoding on the shared metadata fetch so Go no
longer auto-decompresses, and persist the upstream Content-Encoding in
a new metadata_cache column so both the cached and offline responses
replay the exact bytes and header. The uncached streaming path now
forwards Content-Encoding too.
Fixes#300
* fix(handler): scope verbatim metadata fetch to the ProxyCached path
The first cut forced Accept-Encoding: identity in the shared
fetchUpstreamMetadata, which an adversarial review showed both missed
the bug and regressed unrelated ecosystems:
- proxyMetadataStream (the default path, since cache_metadata is off)
never forced identity, so a client sending no Accept-Encoding still
triggered Go's transparent gzip decompression and served altered bytes
for apk/debian/rpm/conda indexes.
- Direct FetchOrCacheMetadata callers that parse or rewrite the body
(npm, pypi, cargo, composer, pub, nuget, swift, maven, helm) were
forced to identity too, losing wire compression and 502-ing against
upstreams that ignore identity and gzip anyway (helm rewriteIndex).
- Rows cached before the fix kept serving decompressed bytes via ETag
304 revalidation.
Scope the verbatim behavior to the ProxyCached code path, which serves
upstream bytes through unchanged (apk, debian, rpm, go, hex, conda,
cran, gem, conan, julia). That path now requests identity on both the
cached fetch and the uncached stream branch and replays Content-Encoding;
direct callers keep transparent compression, matching main. Migration
008 clears etag/fetched_at so legacy rows refetch once with identity.
Tests now exercise the stream path with no client Accept-Encoding and
pin that direct callers are not forced to identity.
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
* Mount web UI under /ui (closes#123)
The UI now lives under /ui so reverse proxies can apply different
access rules to it (e.g. require auth) while leaving the package
endpoints (/npm, /pypi, /v2, ...) open to build machines.
- GET / redirects to /ui/
- /api/browse and /api/compare move to /ui/api/browse and
/ui/api/compare since only the browser JS calls them
- /health, /stats, /metrics, /openapi.json and /api/* stay at root
* README: nginx example for splitting UI auth from package endpoints
The /ui prefix lets a reverse proxy apply different access rules to the
web UI than to the package endpoints. Adds the snippet that motivated
the prefix change so deployers don't have to derive it.
* config: add Health.StorageProbeInterval
* metrics: add proxy_health_probe_failures_total counter
* server: add storageProbe with happy-path test
* server: add storageProbe failure-mode tests
* server: add healthCache with TTL, single-flight, transition logging
* server: wire storage probe into /health
* server: update TestHealthEndpoint for JSON; wire healthCache into newTestServer
Also fix Windows file-locking issue in storageProbe: close the reader
explicitly before Delete so the file handle is released prior to os.Remove.
* server: clean up stale comment in storageProbe
* docs: document storage health probe and new metric
* docs: regenerate Swagger for /health JSON response
* server: simplify rc.Close error handling in storageProbe
* server: defer probe cleanup so size/open/read/verify failures don't leak objects
Previously, storageProbe only called Delete on the success path. Any
failure between Store and the final Delete (size mismatch, Open error,
mid-stream read failure, content mismatch) left the probe object orphaned
in the storage backend. With caching disabled and Kubernetes-rate probing,
the leak could accumulate noticeably on backends like S3.
Use a named return + defer to attempt Delete after every successful Store.
The earlier-step failure remains the primary error; Delete failure only
surfaces as step="delete" when nothing else went wrong. Add a table-driven
test that asserts cleanup runs for each non-delete failure path.
Reported by Copilot on #119.
* config: validate health.storage_probe_interval in Config.Validate
The new duration field was only validated at use time in newHealthCache.
The existing codebase already validates other duration fields
(MetadataTTL, DirectServeTTL, Gradle.MaxAge, Gradle.SweepInterval) in
Config.Validate() so misconfiguration fails fast at startup with a
config-key-specific error.
Match that pattern. The parse-at-use code in newHealthCache stays as
a safety net, mirroring the MetadataTTL precedent.
Reported by Copilot on #119.
* docs: lowercase "counter" in metrics table for consistency
Other rows in the table use lowercase type names (counter/gauge/histogram).
Match that style.
Reported by Copilot on #119.
* docs: include size-check step in /health probe description
The probe is write → size-check → read → verify → delete; the
architecture note was missing the size-check step.
Reported by Copilot on #119.
* server: address andrew's review on #119
- Drop unused callerCtx parameter from healthCache.Check (Check is now
parameter-less; the comment-only "accepted for symmetry" justification
wasn't carrying its weight).
- Emit "storage": {"status": "skipped"} on DB short-circuit instead of
omitting the key, so monitors expecting a fixed key set keep working.
- Reject negative storage_probe_interval at config validation time
(previously parsed and silently behaved like "0").
- Extract HealthConfig.Validate to keep Config.Validate under the
gocognit threshold and match the existing GradleBuildCacheConfig pattern.
- README Health Check section: note that /health is intended as a
readiness probe rather than a liveness probe (Check holds a mutex
for up to the 10s probe timeout).
- cmd/proxy/main.go godoc: column-align the new env var with the
surrounding Gradle entries.
Reported by andrew on #119.
- Implement /julia/* handler for the Pkg server protocol
(registries, registry, package, artifact, meta)
- Resolve package UUIDs to names by parsing Registry.toml from
the General registry tarball, with a hash-guarded background
refresh on registry updates
- Wire into router, ecosystem list, install page, badge styles
- Update README and architecture docs
- Bump github.com/git-pkgs/registries to v0.6.0: the fetcher now
honours HTTP_PROXY, gates dialled IPs against the safehttp block
list, and Version.Integrity is populated for pub, julia and nuget
- Replace internal/cooldown with github.com/git-pkgs/cooldown v0.1.1
(identical surface, lifted from this repo)
- Update docs/architecture.md to point at the external package
Add a `proxy mirror` CLI command and `/api/mirror` API endpoints that
pre-populate the cache from various input sources: individual PURLs,
SBOM files (CycloneDX and SPDX), or full registry enumeration.
The mirror reuses the existing handler.Proxy.GetOrFetchArtifact()
pipeline so cached artifacts are identical to those fetched on demand.
A bounded worker pool controls download parallelism.
Metadata caching is opt-in via `cache_metadata: true` in config (or
PROXY_CACHE_METADATA=true). The mirror command always enables it. When
enabled, upstream metadata responses are stored for offline fallback
with ETag-based conditional revalidation.
New internal/mirror package with Source interface, PURLSource,
SBOMSource, RegistrySource, and async JobStore. New metadata_cache
database table for offline metadata serving.
* Track applied migrations to skip column checks on startup
Add a migrations table that records which migrations have been applied.
On boot, load the set of applied names in one query and only run new ones.
A fully migrated database now does 1 query instead of ~12 HasColumn/HasTable
checks.
Fresh databases created via CreateSchema record all migrations as already
applied. Old databases get the migrations table on first MigrateSchema call
and each migration is recorded after it runs.
Closes#54
* Add benchmark for MigrateSchema on fully migrated database
* Optimize MigrateSchema to single query for fully migrated databases
Skip HasTable/HasColumn checks when the migrations table already exists.
A fully migrated database now does one SELECT instead of ~12 individual
column and table checks.
* Add migration docs and link from architecture
* Add test for upgrade from fully migrated database without migrations table
Add web interface section to README describing all pages (dashboard,
package browser, source browser, version diff). Add monitoring section
with the full Prometheus metrics table and scrape config. Add cooldown
column to the registry support table. Update architecture doc with
accurate database schema including all columns and indexes, and add
entries for metrics, cooldown, and enrichment packages.
