A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
A file whose group directories hold a dot or colon was proxied without
caching, which skipped the denylist, scan and cooldown. Such paths aren't
in the Maven layout, so they now get 404. Directory listings and catalogs
still go through proxyUpstream, which shares its error handling with
Proxy.ProxyUpstream in relayUpstream.
AppendUnquote reports invalid UTF-8 after replacing it, which failed the
whole copy although the decoder allows it. Key now holds U+FFFD in its
place, as with encoding/json and ScanPackument, and the key is still
written with its original bytes.
CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
The relay turned a redirect into a failed fetch, which counted as the
upstream failing rather than refusing, so the cached copy was still
served and no refusal was recorded. Report it as a refusal, as for a
redirected ref listing.
stale_max_age plus the upload-pack grace overflowed for a stale_max_age
near the largest duration, which made every upload-pack request check
the upstream. Cap the sum instead.
A refusal was cleared only by a refresh that succeeded as a whole. When
the upstream listed the refs again but the fetch that followed failed,
say because the cache was full or the fetch timed out, the old refusal
stayed and the repository was still refused, though the upstream had
just confirmed access. Clear it as soon as the ref check succeeds.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
applicable compared the caller's ecosystem raw against the normalised
configured keys, so a caller passing an alias such as go or rubygems
skipped a scanner restricted to golang or gem. Normalise it first.
NewGroup now also rejects unknown names the way Validate does, through
a shared ScannerEcosystemKey, instead of building a scanner that can
never match.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
ParseSize truncated fractional sizes such as 0.5B to 0, and a
storage.max_size of 0 means unlimited. Reject a non-zero size that
would round down to 0 bytes.
The existing test only runs where the file system ignores case, so it
was skipped on Linux, and it passed if the second spelling had no entry
at all. Add a test that sets foldCase directly and checks both
spellings share one entry, require the entry in the existing test, and
work out the repository key once in acquire.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
The cache key for an RPM package is its request path, so a path with
an empty or "." segment made a second entry for the same upstream
file. The path is now cleaned before it is used as the key and the
upstream URL.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Escaping _ in the search pattern stopped typing_extensions from finding
typing-extensions, which worked before because _ matched any single
character. Escape only % and the escape character.
The cache keys Maven artifacts as group:artifact, so org:foo/bar and
org/foo:bar both came out as org:foo:bar and shared one entry. Such
paths are now proxied without caching, like dotted group directories.
This reverts commit fc0e94b. The cleanup stopped at the first segment
that looked like a fetch id, but named Maven repositories put a hex
upstream hash in the filename, so it stopped early. For objects in the
old layout, with no fetch directory, the walk could remove version or
package directories under a concurrent fetch. Leftover empty
directories will be handled separately.
APK and Arch packages, and now RPM ones, have filenames with directories
in them, which put the object below its fetch directory. Delete only
removed the object's own directory when it was the fetch directory, so
those directories stayed behind. It now removes them up to and including
the fetch directory while they are empty.
A check that ran past refresh_wait or failed served the cached copy, also
right after the upstream had refused the repository. With protocol v0 the
refs are in the info/refs response itself, so they were advertised. The
stored refusal now refuses those too, until a check succeeds.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
exec.CommandContext kills only the git parent. Its git-remote-http helper
kept its relay connection and went on downloading for up to
http.lowSpeedTime after a client disconnect, Close or fetchTimeout. Each
read it relayed called cover, which reserved the relayed total again on a
reservation already settled or released, so c.reserved grew for good and
could evict repositories, and the helpers outlived the fetch cap.
Tie each relay route to the fetch's context and cancel it when the fetch
returns, so the upstream request ends and the helper's connection is cut.
Make cover refuse a reservation that has been given back.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
SearchPackages and CountSearchResults matched names with
p.name LIKE '%query%'. LIKE ignores ASCII case on SQLite but not on
Postgres, so the same search found different packages depending on the
database. The % and _ in a query were also taken as wildcards. Compare
LOWER(name) with LOWER(pattern) and escape the wildcards.
