With metadata_ttl 0 another instance can replace the blob between
reading the row and reading the blob, and the fallback then served
the new bytes under the old row's ETag, so a client holding that ETag
got a wrong 304. Serve the blob without the row's validators when it
doesn't match the row's digest.
The refresh of a row confirmed by a 304 had moved ahead of validate,
so a stored document validate then rejected was still recorded as
fetched now. Validate first, as before.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
A SNAPSHOT file named without a timestamp was dropped from the cache
before it was fetched again, so an unreachable upstream gave a 502
instead of the last build, concurrent requests could drop each
other's fresh copy, and HEAD downloaded the file.
Fetch it first under the coalescing key and let the new copy replace
the cached one once stored. When the fetch fails for any reason but
the file being missing or blocked, serve the cached copy. HEAD is
answered from cache.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
The cache key for an RPM package is its request path, so a path with
an empty or "." segment made a second entry for the same upstream
file. The path is now cleaned before it is used as the key and the
upstream URL.
The cache keys Maven artifacts as group:artifact, so org:foo/bar and
org/foo:bar both came out as org:foo:bar and shared one entry. Such
paths are now proxied without caching, like dotted group directories.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Non-timestamped SNAPSHOT files went straight to upstream, past the
denylist, cooldown, scanning and the Plugin Portal fallback. They now take
the normal download path, with the cached copy dropped first so each
request still gets the current build.
A database error while refreshing a verified 304 no longer forces a full
download: the bytes were checked against the row and upstream confirmed
them.
Without cooldown, maven-metadata.xml and its .md5/.sha1/.sha256/.sha512
files were cached under separate keys with separate TTLs, so a deploy
between the two fetches left the checksum describing another document
and strict builds failed. The checksum files are now computed from the
cached metadata, as the cooldown path already did. The metadata is
also written with the validators of the document served.
SNAPSHOT files named with SNAPSHOT rather than a build timestamp
(lib-1.0-SNAPSHOT.jar, as JitPack and non-unique snapshot repositories
publish them) matched the artifact rules and were cached for good, so
the first build was served forever. They are now passed through
uncached. Timestamped SNAPSHOT files never change and stay cached.
InRelease/Release and repomd.xml had their own metadata cache entries
and TTLs, independent of the Packages, Sources or plainly named
primary.xml.gz files they list. Within a TTL a client could get the
cached release file and a freshly fetched index, or a fresh release
file and a cached index, and apt then failed with "Hash Sum mismatch"
(repositories without by-hash, or RPM repositories built with
--simple-md-filenames).
An index file that the cached release file lists is now served by
that listing: the cached copy while its digest matches the listed
sha256, whatever its age, and otherwise a copy fetched from upstream.
If upstream's copy does not match either, the repository has moved on
since the release file was cached; that copy is served and the
release file is expired so the next update gets a matching pair.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
pypi.org's HTML root index links each project as /simple/<name>/. The
proxy relayed it unchanged, but it is mounted at /pypi, so following a
link went to /simple/<name>/ on the proxy and got 404. Rewrite those
hrefs to the proxy's /pypi/simple/ while streaming the response, since
the index is tens of megabytes. The JSON index has no links and is left
alone.
parsePath joins the group directories with dots, and the cache filename
is the file's base name, so org.foo/bar/1.0/bar-1.0.jar and
org/foo/bar/1.0/bar-1.0.jar shared one cache entry. Only upstreams that
serve content at such directories, such as raw or hosted repositories,
are affected.
The Maven layout splits a group at its dots, so a group directory never
contains a dot or is empty. Treat such paths as outside the layout and
proxy them uncached, which keeps every existing cache entry valid.
Cached metadata responses took their ETag, Last-Modified and stale
flag from the cache row as it was after the fetch, not from the
document actually sent. When a refetch got a new document but could
not store it, the client got the new body with the old ETag and a
Warning header, and a client still holding the old document got a
304 for it. With metadata_ttl 0 the stale fallback had no Warning at
all, since staleness was inferred from the row's age.
The lookup result now carries the validators of its body (upstream's
for a fetched document, the row's for a stored one) and whether it is
a stale fallback, and the response is written from those.
A 304 also trusted whatever was stored at the row's path. Another
instance sharing the store may have replaced it meanwhile, and the
new bytes were then served and stored under the old ETag. The stored
bytes are now checked against the row's digest, and a 304 whose row
can no longer be refreshed is fetched again without validators
instead of storing the old document back over the new one.
npm: a tarball request for a version the cached packument did not list
yet (cached within metadata_ttl before the release) found no publish
time and was let through cooldown, and fetched without integrity. When
cooldown applies and the cached packument lacks the version, fetch it
again and decide on what upstream lists now; if it can't be fetched,
fail as when a publish time can't be read.
npm: cooldown only moved dist-tags.latest. Other tags kept naming
versions in cooldown, while the denylist removes such tags. Cooldown
now removes them too, and drops latest when no version is left, in
both the in-memory and streamed rewrites.
PyPI: the simple JSON rewrite removed filtered files but left their
versions in the PEP 700 versions list. Remove them there too.
pub: latest was never rewritten, so its archive_url pointed at pub.dev,
it kept naming a version in cooldown when no version was left, and the
replacement was the last list entry, which may be a prerelease. Point
latest's archive_url at the proxy, replace a held back latest with the
newest stable version left (or the newest prerelease), and drop it
when nothing is left.
The blob route and blobPathPattern only matched sha256 digests, so a
sha512 blob got 404 although the fetch path can verify sha512. The image
mirror rejected sha512 digests in index entries, configs and layers for
the same reason. Accept sha512 in both, with exact digest lengths.
A HEAD for an uncached manifest by digest passed on whatever digest the
upstream named. GET checks the body against the requested digest; HEAD
now answers DIGEST_INVALID when the upstream names a different digest
of the same algorithm.
RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
R 4.3 and later on macOS fetch binaries from a two-segment platform path
such as bin/macosx/big-sur-arm64/contrib/4.4/. The router only matched a
single platform segment, so both the PACKAGES indexes and the .tgz
downloads returned 404. Add routes for that layout and include the arch
in the stored version so arm64 and x86_64 builds stay apart.
src/contrib/Meta/archive.rds and current.rds, which remotes and pak use
to install archived versions, had no route either. Serve .rds files under
Meta as cached metadata.
ScanPackument and LookupVersion read with a jsontext.Decoder, skipping
values they don't keep with SkipValue and reading the one LookupVersion
returns with ReadValue, instead of decoding each into a json.RawMessage.
Repeated names and invalid UTF-8 are still accepted as encoding/json
accepts them. Go 1.27 is now required, for encoding/json/jsontext.
Benchmarks for both use the 5,000-version packument the handler benchmarks
build, now shared in internal/metadata/npm/npmbench.
A pkg:git version can now be a tag as well as a full commit ID. With a
version, the submodules the commit records are mirrored at the commits it
pins, from the upstream.git hosts serving them; one no host serves is
skipped with a warning.
Mirroring a commit on no branch or tag again writes a fresh wanted ref, so
its 7 days count from the last mirror run. A cached repository whose refresh
fetched new commits is no longer reported as cached.
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
A mirror run that finds a blob cached records it in mirrored_at, which
retention counts as use. LRU eviction keeps using last_accessed_at, so
#508's decision not to count mirror runs as cache hits stands.
Fixes#648.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
The lookup took any non-200 metadata response for a version that isn't
listed, so a Packagist outage or rate limit answered the archive request
with 404. Composer treats that as the version being gone.
Only 404 and 410 now mean the document is missing. A 429 is passed on as
503 with Retry-After while the upstream is held, and any other status
answers 502.
