The feed readers move to registries' changes package. The proxy keeps the
cursor and moves it past the changes only once their metadata is expired,
so a failed expiry reads them again.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
Packuments list each version's provenance at dist.attestations.url on
registry.npmjs.org. npm takes only the path of that URL and resolves it
against the configured registry, so behind the proxy `npm audit
signatures` asked for /-/npm/v1/attestations/... at the proxy's root,
outside /npm, and could not verify attestations.
The rewrite now points dist.attestations.url at
/npm/-/npm/v1/attestations/{name}@{version}, and the handler serves that
path. It reads the version's attestations URL from the packument and
fetches it from the package's registry, the route's for routed packages;
a URL on another host is not followed and the registry's standard path
is asked instead. Attestations never change once published, so they are
cached as immutable metadata.
Versions blocked by the denylist get 403 and versions held back by
cooldown 404, as their tarballs do. Pushed versions have no upstream
attestations and get 404.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
pnpm 12 checks every lockfile entry against the registry on each install
unless trustLockfile is set, so each install reads the packument of every
package in the lockfile. The rewrite cache kept each document twice, plain
and gzipped, and with cooldown on for a minute only, so these packuments
were rewritten and compressed again on every install.
Keep only the gzip copy of a cached rewrite, about an eighth of the
document, and decompress it for clients and handlers that need the plain
bytes. Keep rewrites made with cooldown on for an hour. Answer HEAD for
packuments, which pnpm 12 sends before each GET.
The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
Serving packages.json fetched the upstream's on every request with
cache_metadata off, and after each TTL with it on, so every Composer
command waited on the upstream and on http_timeout when it was down.
The parsed index is now kept for the metadata TTL, or an hour when it
is 0, and a failed fetch is not retried within that time.
- GetProjectsUsingPackage no longer counts each project's components
per row; the package page only needs names and versions. The ?purl=
listing leaves component_count out accordingly.
- The delete button sends to DELETE /ui/api/projects/{project}, the same
handler as the API and still behind the token, so it sits under /ui
like the cache deletes.
- The project page answers 500 when its components can't be read,
instead of showing an empty project.
- DELETE of a path ending in /sbom that names no project answers 400
with a hint, since it's most likely the upload path.
- The upload endpoint lists JSON, XML and plain text (SPDX tag-value)
as accepted.
- The docs say that versions are compared as the SBOM spells them.
FromPURL now names packages the way their handlers store them:
- NuGet IDs are lowercased, as the NuGet handler stores them.
- PyPI names are normalized as in PEP 503. The PyPI handler takes the
name from the file name, which spells "." as "_", so
pkg:pypi/zope.interface has to match the cached zope-interface.
- Conda package URLs without a channel qualifier are skipped: the conda
handler records packages as channel/name, so they can never match.
- The Go main-module check ignores case. syft keeps the module path's
case in its mainModule property, while golang package URLs are
lowercased.
A table test checks every mapped PURL type against the name its handler
passes when caching.
The vulnerabilities table is only written by tests: nothing in the proxy
calls UpsertVulnerability, so the dashboard card, the per-project count
and the per-component markers were always empty. Remove them along with
GetComponentVulnerabilities and inventory.Affected, and keep the
outdated marker, which uses the latest version the handlers record.
The page asks for the upload token and sends it to DELETE
/api/projects/{project}, so /ui gains no unauthenticated way to delete.
The token is kept in session storage for the next delete.
A syft SBOM of a repository lists the repository itself: the module in
go.mod and the root of package-lock.json, both without a version or with
the version from package.json, and the module its Go binaries were built
from. The project page showed them as dependencies.
Entries without a version are now skipped, as are lock-file roots (lock
entries that depend on others while nothing depends on them) and modules
syft marks as a binary's main module.
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
Within metadata_ttl cached npm and Composer metadata is served without
asking upstream, so a new release can take that long to show up. Both
registries publish a feed of the packages that changed.
upstream.npm_change_feed and upstream.composer_change_feed (off by
default, require cache_metadata) start a reader per feed that runs every
minute and clears fetched_at on the cached rows of listed packages that
were fetched before the read. The rows keep their document and ETag, so
the next request revalidates them upstream. Routed packages are cached
under other keys and never match. A Packagist resync expires all cached
Composer metadata. Failed reads back off up to 15 minutes. The feed
position is kept in memory, so after a restart the reader starts from the
feed's current position.
`npm dist-tag ls` reads /-/package/{name}/dist-tags. The proxy had no
route for it, so the request ended up in the metadata path and failed.
The tags are taken from the packument the proxy serves rather than
relayed from upstream, so latest never names a version cooldown holds
back, routed packages use their route, and pushed versions and
hosted-only scopes count as they do for the packument. Changing tags
stays unsupported and still gets 405.
Alpine mirrors keep only the current build of each package, so once a
package is upgraded `apk add curl=8.5.0-r0` fails: the current index no
longer lists the old build, even when the proxy still has it cached.
With upstream.apk_snapshots set (e.g. "90d"), each version of an index or
detached signature the metadata cache stores is copied to
_snapshots/alpine/ and recorded in a new metadata_snapshots table with the
time it was first served. /apk/{name}@{YYYY-MM-DD}/ serves the version that
was current at the end of that day; packages below it are the plain path's
requests. The kept bytes are the signed upstream bytes, so apk verifies
them unchanged.
An hourly job deletes versions replaced longer ago than the configured
duration and queues their blobs on the pending-delete queue; the newest
version of each index stays. Packages listed by kept snapshots are not
protected from eviction or retention; the docs say so.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
The proxy's packages.json listed only metadata, search and notify URLs,
so a client using the proxy as its only Composer repository lost
`composer audit`, `audit.block-insecure` and Packagist's malware filter
list without any warning.
The service index now carries the security-advisories and filter entries
of the upstream repository's own packages.json, cached for the metadata
TTL, with URLs pointing at the proxy. An upstream that does not offer
them gets nothing advertised, so a custom upstream.composer_repository
without the API keeps working as before. Advisories and filter entries
embedded in p2 metadata were already passed on unchanged.
Advisories queries and the filter summary are relayed uncached. Names of
routed packages and hosted-only vendors are removed from advisories
queries first, so private package names are not sent to the public API.
The dashboard shows the default and the ecosystem overrides, and counts
package overrides and patterns without naming them, since the UI has no
auth. A package page shows the cooldown that applies to the package and
the kind of rule it comes from.
Also add FillVersionPublishedAt, which stores a publish time only where
none is stored, for recording the versions cooldown holds back.
A client that fell behind is switched to the stored copy, which for an
unrecorded refetch is deleted once the waiters close. It now takes its
own hold, released when it closes.
The upstream read fed the client through a pipe, so it ran at the client's
speed. The client is now fed from a bounded buffer and, once it falls
behind, continues from the stored copy where it left off.
A caller handed the stream takes no hold on an unrecorded object.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
