FromPURL now names packages the way their handlers store them:
- NuGet IDs are lowercased, as the NuGet handler stores them.
- PyPI names are normalized as in PEP 503. The PyPI handler takes the
name from the file name, which spells "." as "_", so
pkg:pypi/zope.interface has to match the cached zope-interface.
- Conda package URLs without a channel qualifier are skipped: the conda
handler records packages as channel/name, so they can never match.
- The Go main-module check ignores case. syft keeps the module path's
case in its mainModule property, while golang package URLs are
lowercased.
A table test checks every mapped PURL type against the name its handler
passes when caching.
The vulnerabilities table is only written by tests: nothing in the proxy
calls UpsertVulnerability, so the dashboard card, the per-project count
and the per-component markers were always empty. Remove them along with
GetComponentVulnerabilities and inventory.Affected, and keep the
outdated marker, which uses the latest version the handlers record.
The page asks for the upload token and sends it to DELETE
/api/projects/{project}, so /ui gains no unauthenticated way to delete.
The token is kept in session storage for the next delete.
A syft SBOM of a repository lists the repository itself: the module in
go.mod and the root of package-lock.json, both without a version or with
the version from package.json, and the module its Go binaries were built
from. The project page showed them as dependencies.
Entries without a version are now skipped, as are lock-file roots (lock
entries that depend on others while nothing depends on them) and modules
syft marks as a binary's main module.
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
Within metadata_ttl cached npm and Composer metadata is served without
asking upstream, so a new release can take that long to show up. Both
registries publish a feed of the packages that changed.
upstream.npm_change_feed and upstream.composer_change_feed (off by
default, require cache_metadata) start a reader per feed that runs every
minute and clears fetched_at on the cached rows of listed packages that
were fetched before the read. The rows keep their document and ETag, so
the next request revalidates them upstream. Routed packages are cached
under other keys and never match. A Packagist resync expires all cached
Composer metadata. Failed reads back off up to 15 minutes. The feed
position is kept in memory, so after a restart the reader starts from the
feed's current position.
`npm dist-tag ls` reads /-/package/{name}/dist-tags. The proxy had no
route for it, so the request ended up in the metadata path and failed.
The tags are taken from the packument the proxy serves rather than
relayed from upstream, so latest never names a version cooldown holds
back, routed packages use their route, and pushed versions and
hosted-only scopes count as they do for the packument. Changing tags
stays unsupported and still gets 405.
Alpine mirrors keep only the current build of each package, so once a
package is upgraded `apk add curl=8.5.0-r0` fails: the current index no
longer lists the old build, even when the proxy still has it cached.
With upstream.apk_snapshots set (e.g. "90d"), each version of an index or
detached signature the metadata cache stores is copied to
_snapshots/alpine/ and recorded in a new metadata_snapshots table with the
time it was first served. /apk/{name}@{YYYY-MM-DD}/ serves the version that
was current at the end of that day; packages below it are the plain path's
requests. The kept bytes are the signed upstream bytes, so apk verifies
them unchanged.
An hourly job deletes versions replaced longer ago than the configured
duration and queues their blobs on the pending-delete queue; the newest
version of each index stays. Packages listed by kept snapshots are not
protected from eviction or retention; the docs say so.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
The proxy's packages.json listed only metadata, search and notify URLs,
so a client using the proxy as its only Composer repository lost
`composer audit`, `audit.block-insecure` and Packagist's malware filter
list without any warning.
The service index now carries the security-advisories and filter entries
of the upstream repository's own packages.json, cached for the metadata
TTL, with URLs pointing at the proxy. An upstream that does not offer
them gets nothing advertised, so a custom upstream.composer_repository
without the API keeps working as before. Advisories and filter entries
embedded in p2 metadata were already passed on unchanged.
Advisories queries and the filter summary are relayed uncached. Names of
routed packages and hosted-only vendors are removed from advisories
queries first, so private package names are not sent to the public API.
The dashboard shows the default and the ecosystem overrides, and counts
package overrides and patterns without naming them, since the UI has no
auth. A package page shows the cooldown that applies to the package and
the kind of rule it comes from.
Also add FillVersionPublishedAt, which stores a publish time only where
none is stored, for recording the versions cooldown holds back.
The upstream read fed the client through a pipe, so it ran at the client's
speed. The client is now fed from a bounded buffer and, once it falls
behind, continues from the stored copy where it left off.
A caller handed the stream takes no hold on an unrecorded object.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
Whether a packument is the abbreviated format is a property of the
document, so it sits with the rest of the npm metadata code; the
handler keeps only the error it turns that into. Add a test that keeps
the package's imports to the portable set, give jsonscan a package
comment, and describe the new packages in the architecture docs.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
On a cache miss the proxy stored the whole artifact before sending any of
it, so clients with an idle timeout gave up on large artifacts: git-lfs
after 30 seconds, pacman after 10. The caller that starts a shared fetch
now gets the bytes as they arrive while they are stored; requests joining
the fetch, HEAD and range requests still wait for the stored copy.
The client's copy is checked against the upstream's digest as it is read,
and a mismatch aborts the response before it completes. A client going
away leaves the copy to storage running. A storage error leaves the
client's copy running: the client gets the artifact, and the failure is
logged at error level and counted in proxy_artifacts_served_uncached_total.
A block-mode scanner has to see the whole artifact first, so with one
configured for the ecosystem a miss is handled as before.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
- Keep the download URLs of a batch response in memory until shortly
before they expire, so the GET that follows needs no second batch
request. A kept URL that fails is replaced by a fresh one.
- Cut off a download that isn't the object's size.
- Copy an object whose content is already stored, such as for a fork,
instead of downloading it, once the upstream has confirmed the
repository has it. Artifacts are indexed by content hash for this.
- Keep signed download URLs out of logs and the database: the fetch uses
the signed URL, everything else the URL without its query.
- Answer a batch from the cache when the upstream can't be reached or
fails, so git-lfs can use cached objects during an outage. Refusals
are still passed on.
- Answer a HEAD for an uncached object from the batch response instead
of downloading it.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
The proxy filters repodata.json and current_repodata.json, but
repodata.json.bz2 went through the metadata cache unfiltered, and
repodata.json.zst and repodata.jlap reached the download handler and were
passed through. conda tries repodata.json.zst first, so with cooldown on it
got versions still in cooldown.
With cooldown on these now answer 404, and conda falls back to
repodata.json. repodata_from_packages.json is filtered like repodata.json.
With cooldown on, the RubyGems compact index and versions API, the Hex
package and Hex API entry, and conda's repodata.json were fetched with
plain HTTP requests on every client request: no TTL, no revalidation,
no coalescing and no stale fallback, even with cache_metadata.
Gem and Hex now read both documents through the metadata cache and
filter the cached copies. The filtered result goes through the rewrite
cache and is served with an ETag. When the times document is older than
the index and misses a version, it is fetched again; if that fails the
version is left out. Conda filters a fresh stored repodata.json as it
reads it from storage, uses the metadata cache otherwise, and remembers
the ETag of the filtered copy until a withheld entry leaves cooldown.
SQLite used one connection for everything, so request lookups queued
behind the per-minute stats queries, dashboard pages and eviction scans.
Writes keep the single connection. Get, Select, Query and QueryRow now go
to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode
lets run alongside the writer. A write sent to that pool fails.
Pragmas move into the DSN so every connection gets them. Postgres and
in-memory SQLite keep one pool for both.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
The Helm index.yaml and PyPI simple pages and JSON API documents were
parsed and rewritten on every request, cache hits and conditional
requests included. They now go through the rewrite cache like npm and
Composer, so a repeat request is served from memory with an ETag of
its own, a dated Last-Modified, a gzip copy and 304 when it matches.
PyPI simple pages are keyed by their content type and the versions
filtered out as well as the upstream page. Download hrefs are now
rewritten with a literal prefix search that makes the same changes as
the old regular expression, and the link filter's expressions are
compiled once.
A chart cache miss no longer parses the whole Helm index again while
the stored index is unchanged: the chart downloads it lists are kept
per repository, keyed by the index digest.
A stale tag was revalidated by repeating the client's GET with
If-None-Match. Docker Hub counts manifest GETs against its pull limit,
so every TTL expiry of a tag could cost a pull. The tag is now resolved
with a HEAD and the Docker-Content-Digest compared with the cached one;
the manifest is fetched by digest only when that digest isn't cached.
A HEAD without a digest falls back to the conditional GET.
A client HEAD whose digest is cached now points the tag at it, so the
GET that follows is a cache hit. Concurrent requests for the same
manifest share one upstream exchange, and instances sharing Postgres
take turns through the fetch lease.
Every npm and Composer download ran GetLocalArtifact, a full artifact
row lookup, even with hosted packages disabled. localVersions already
returns early in that case, so metadata does not list pushes, and
/generic/hosted/ is only served while hosting is enabled.
GetLocalArtifact now returns nil without a lookup when hosting is off.
Downloads then take the normal proxy path. An archive pushed earlier is
still answered from storage by that path and is never replaced by the
upstream copy; only cooldown now applies to it, as metadata no longer
lists it either. The hosted docs say so.
Every PyPI download with cooldown on read the JSON API first, cached
files included. Without cache_metadata that was a full /pypi/<name>/json
per file.
A download that looks up the JSON API now stores the release's newest
upload time on its versions row. A cached file is then checked against
the stored time without asking upstream. A file that isn't cached is
still looked up, since files can be added to a release later, and the
stored time is refreshed. A stored time inside the window withholds the
release without a lookup.
