The lookup took any non-200 metadata response for a version that isn't
listed, so a Packagist outage or rate limit answered the archive request
with 404. Composer treats that as the version being gone.
Only 404 and 410 now mean the document is missing. A 429 is passed on as
503 with Retry-After while the upstream is held, and any other status
answers 502.
The API handler takes the enrichment calls it uses as an interface, so
the test can answer them without reaching a registry.
Remove FillVersionPublishedAt until the shared cooldown filter calls it.
metadata_max_size bounds what is read into memory, but a packument
streamed from upstream into storage had no limit, and with cooldown was
stored twice. It is now capped by metadata_stream_max_size, 1 GB by
default. A document over the cap fails as an upstream failure: nothing is
stored, and the stored copy, if any, is served instead.
The metadata settings move into configureMetadata to keep serve under
the maintidx limit.
The change to ReadMetadataSized and fetchUpstreamMetadata, with its tests,
is now #649. upstreamBodyReader stays for storeUpstreamMetadata, in a file
of its own that #649 adds with the same content.
TestAPKSnapshots_CacheHit covers the cache-hit branch with a metadata TTL
above 0. proxyCachedWithEncoding takes optional hooks for a cache hit and a
fetched body, so the APK handler no longer keeps a copy of it.
ReadMetadataSized took the transport's io.ErrUnexpectedEOF for the end of
a body shorter than its size hint. A document upstream closed before its
Content-Length was served with a 200 and cached, or failed the cooldown
filter with a 502, instead of the cached copy being served. Upstream
bodies are now read through a reader that marks their read errors, and
such a body fails the read as an upstream failure.
FuzzParse checks that an uploaded body never makes the inventory panic
and that every component it records has a package URL, ecosystem and
version. TestSaveProjectConcurrentUploads saves one project from eight
uploads at once and checks that each succeeds and the components of
exactly one remain.
errgroup.WithContext cancelled the blobs being fetched as soon as one
failed. A client pull sharing one of those fetches failed with it, and the
next run had to fetch them again. A failure now only stops further blobs
from starting.
Covers namespaced npm and Maven PURLs, a publish time under an alias of
a package row's ecosystem, and a PURL starting with the orphan prefix,
and checks the per-ecosystem versions against GetCacheStats' total.
A body that ended before its Content-Length was read as complete in
memory: ReadMetadataSized took the transport's io.ErrUnexpectedEOF for
the end of a body shorter than its size hint. The half document was
served and cached, or failed the filter with a 502, instead of the stored
copy being served. Upstream bodies are now read through a reader that
marks their read errors, and such a failure is an upstream failure on
both paths: streamed, it no longer counts as a failure to store, which
had sent the request to the bounded in-memory path.
A stored packument whose bytes no longer match its row's digest is now
fetched again like a missing one.
Tests also cover concurrent requests sharing one fetch, client
revalidation and upstream failures with nothing cached. The stream tests
build cooldown through cooldown.NewPolicy as the server does, since a
bare cooldown.Config parses itself on first use without locking.
A row within its TTL was served without checking its blob, so a missing
blob failed the streamed read and the request fell back to the bounded
in-memory path. A read that finds the stored copy missing now fetches the
packument again into storage, ignoring that row, and reads it once more,
for packument, version and tarball requests.
Cooldown stores the publish time of every version it looks up, also of
packages nobody downloaded, which have no package row. The ecosystem stats
counted those versions as unattributed. They now count under the type of
their package PURL; only a PURL that can't be parsed stays unattributed.
* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
purl v0.2.1 keeps the case of Go module paths, so the inventory no
longer lowercases syft's mainModule. Module paths are case-sensitive:
github.com/azure/foo is a dependency of github.com/Azure/foo, not part
of it.
With purl v0.2.1, FullName returns a Swift package as its registry
identity, scope.name. The mirror splits namespace/name again to find
the scope or the source repository, so HandlerPackage keeps that form,
as mirrorTarget did before the two mappings were shared.
TestMirrorImageKeepsTagMovedByPullDuringWalk moves a tag while a blob is
served. With blobs fetched concurrently, another blob request reads the
manifest map at the same time, which the race detector caught on
Windows.
The image mirror fetched the config and layer blobs of a manifest one
after another, so a run took as long as all layers together. It now
fetches up to three at once, as many as a Docker pull does by default.
For a Home Assistant image (550 MB in ten blobs) that would leave
mostly the time of its largest layer.
packageurl.HandlerPackage returns the ecosystem and name the handlers store
a package URL under. The mirror and the inventory both use it, which
settles where they disagreed: distribution namespaces of apk, alpm, bsd,
deb and rpm are dropped, PyPI and NuGet names use purl's comparison form,
and Maven names are group:artifact as the handler stores them, where the
mirror used group/artifact.
A snapshot was recorded after the response by reading the metadata cache
again, so a refresh that replaced the cached copy in between lost the
version the request served. Record it from the served bytes before
responding instead, and compare a new snapshot with the version current
at its own time, so one recorded late keeps its place in the history.
parseComposerUpstreamIndex now returns an error for a packages.json that
is not a JSON object or has the advisories and filter fields in the wrong
shape, and fetchUpstreamIndex passes it to the metadata cache as validate.
A malformed 200 is no longer stored: with metadata caching the stored
copy is served as when upstream fails, and without it the error keeps
the copy in memory.
AddMetadataSnapshot takes a transaction-scoped advisory lock on the
document before its check and insert, so two replicas recording the same
version can't both read the previous digest and both insert. SQLite runs
one write at a time, so it keeps the single statement.
A fetched packument is now stored under a staging path and checked before
it replaces the cached copy, so an abbreviated one no longer overwrites a
full copy; the full copy is served as when upstream fails. A failure to
store falls back to the in-memory path, as caching is best effort there.
Requests for a single version or dist-tag and the tarball download's
cooldown and integrity lookup read the stored packument as a stream too,
keeping only the one version entry they need.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
* Add --latest to mirror only the newest version
An unversioned PURL mirrors every version the upstream lists. With
--latest, or "latest": true in a mirror API job, only the newest is
mirrored: the highest by the ecosystem's version ordering, without
yanked or retracted versions and, unless there is nothing else,
pre-releases.
* Pick the latest version with registries.FetchLatestVersion
--latest now uses the shared policy from git-pkgs/registries: the version
the registry advertises as latest where it has one, otherwise the newest
active version by publish time. Deprecated versions are left out too, and
newestVersion and the direct vers dependency go away.
Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
Part of #617. The package page marks versions in cooldown with when they
become available, and lists a version only seen in metadata (no artifact
row) only while it is held back. /api/package returns the package's
cooldown and available_at on a held-back version.
FillVersionPublishedAt stores a publish time without overwriting one, for
the cooldown filters to call once they record the versions they hold back.
Two requests recording the same index version in the same millisecond
stored to one path, and the one whose row was not added deleted the
blob the other's row points at. The path now ends in a random suffix.
Also trims comments that restated the PR, notes in the docs that turning
snapshots off leaves them in place, and moves the git cache setup out of
serve, which lint found too long after the merge.
