With cache_delete set, DELETE /api/git/{upstream}/{repo} removes one
repository from the git cache and /ui/git gets a Delete button per
repository. The repository is moved aside under the cache's lock, as
eviction does, so the next clone fetches it again. One that a clone, a
fetch or a client is using answers 409. The response names the
repository and the bytes freed.
The state shown on the git page was read again after every check of the
upstream, and a check runs on every client info/refs. On a busy proxy
each view then read the same repositories again and never got past what
fits in two seconds.
The state is now kept until the repository's refs or HEAD are written: by
a clone, a fetch, a fetch of wanted commits, or a change of the default
branch. A check that finds nothing new leaves it in place.
A state that fails to read for another reason than the view's deadline is
now logged at warn with the repository key.
Retention evicts every file that has not been served for the configured
time, also the newest version of a package. keep_latest: N holds back
the files of the N highest cached versions of each package from
eviction by age. Versions are ordered by the ecosystem's version rules.
It applies to cargo, golang, npm and pypi, where the cache stores a
release under its version. Other ecosystems ignore it for now.
A view ran four git processes for each repository whose state wasn't
cached, with no limit, and each concurrent view did the same.
RepositoryState now runs one read at a time and gives up with its
context. The page reads for at most two seconds, lists the rest without
branches and commits, and says how many are left; the cache keeps what
was read, so each reload reads more.
Also test that an upstream, a repos pattern and a repository with
uppercase letters are listed as spelled, and correct two comments.
Where cache_dir ignores case, the cache tracks repositories under a
lowercased key, and Repositories returned that key. The git page then
found no configured upstream for an upstream with an uppercase letter,
hid repositories that a repos pattern with one allows, and showed
lowercased names and clone URLs. The key now comes from the
repository's directory, which keeps the case of the clone that created
it.
The docs also say the page shows default branches and commit subjects,
and that the request count includes ls-remote.
Cooldown profiles and the unprefixed URLs share a packument's cache row,
but each chose its format by its own policy: abbreviated without
cooldown, full with it. With cooldown on for only some of them, a request
without cooldown that revalidated the row stored the abbreviated
packument. The next request under cooldown rejected it and fetched the
full one again, even within metadata_ttl, and had no stale copy to serve
when the upstream failed.
Cooldown in the global policy or in any profile now makes all of them
request the full packument, as upstream.npm_full_metadata does. URLs
without cooldown are then served full packuments too.
Since main gained the repos allowlist and the refusal of repositories the
upstream refused, the page could still list a repository the allowlist no
longer allows and show the cached commit of one the upstream refused.
Repositories now reports Refused, and the page leaves out what the
allowlist doesn't allow and shows no cached state for a refused one.
A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
The repeated dist-tags name in the jsontext test now has a different
first value, so the test checks that the last one counts. Comments
that repeat the code go, and the upload-pack paragraph states its
check rule once.
The docs say refreshing Debian, Alpine and RPM packages needs
cache_metadata, but nothing said so when it was off. Rejecting the
setting would also stop the refresh of every other ecosystem, so the
server logs a warning from the new Config.Warnings at startup.
A version's first number was taken as its major line, so certifi
2024.8.30 and tzdata 2024.2 never got the next year's release. A first
number of 1000 or more is a year, and all such versions are one line.
The refresh read InRelease, APKINDEX and repomd.xml again through the
metadata cache, which marked them accessed, so every suite looked
fetched within since and never aged out. Read them straight from
upstream, as the Packages and primary indexes already are.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Document that a mutable SNAPSHOT file falls back to the cached copy
and that HEAD is answered from cache, and that the most recently
fetched of InRelease and Release decides which index is served.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
Git LFS objects are artifacts under ecosystem git, so adding the
repositories to the per-ecosystem gauges mixed the two and broke the
per-ecosystem figures adding up to the totals. The git cache now has
proxy_git_cache_size_bytes and proxy_git_cached_repositories, updated
on each tick apart from the database gauges so a failed stats query
leaves them current.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
Non-timestamped SNAPSHOT files went straight to upstream, past the
denylist, cooldown, scanning and the Plugin Portal fallback. They now take
the normal download path, with the cached copy dropped first so each
request still gets the current build.
A database error while refreshing a verified 304 no longer forces a full
download: the bytes were checked against the row and upstream confirmed
them.
Without cooldown, maven-metadata.xml and its .md5/.sha1/.sha256/.sha512
files were cached under separate keys with separate TTLs, so a deploy
between the two fetches left the checksum describing another document
and strict builds failed. The checksum files are now computed from the
cached metadata, as the cooldown path already did. The metadata is
also written with the validators of the document served.
SNAPSHOT files named with SNAPSHOT rather than a build timestamp
(lib-1.0-SNAPSHOT.jar, as JitPack and non-unique snapshot repositories
publish them) matched the artifact rules and were cached for good, so
the first build was served forever. They are now passed through
uncached. Timestamped SNAPSHOT files never change and stay cached.
InRelease/Release and repomd.xml had their own metadata cache entries
and TTLs, independent of the Packages, Sources or plainly named
primary.xml.gz files they list. Within a TTL a client could get the
cached release file and a freshly fetched index, or a fresh release
file and a cached index, and apt then failed with "Hash Sum mismatch"
(repositories without by-hash, or RPM repositories built with
--simple-md-filenames).
An index file that the cached release file lists is now served by
that listing: the cached copy while its digest matches the listed
sha256, whatever its age, and otherwise a copy fetched from upstream.
If upstream's copy does not match either, the repository has moved on
since the release file was cached; that copy is served and the
release file is expired so the next update gets a matching pair.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
On a case-insensitive file system, as macOS and Windows use by default,
acme/WIDGETS and acme/widgets name the same cache directory but were
tracked as two repositories. Users of one spelling didn't keep the other
from being evicted, so eviction could move the directory away while it
was being read, and each entry had its own write lock, so two refreshes
could fetch into the directory at once.
Check at startup whether the cache directory ignores case and, if so,
track repositories under their lowercased key. The directory and the
upstream URL keep the spelling requested. Case-sensitive file systems,
and so existing caches on Linux, are unchanged.
parsePath joins the group directories with dots, and the cache filename
is the file's base name, so org.foo/bar/1.0/bar-1.0.jar and
org/foo/bar/1.0/bar-1.0.jar shared one cache entry. Only upstreams that
serve content at such directories, such as raw or hosted repositories,
are affected.
The Maven layout splits a group at its dots, so a group directory never
contains a dot or is empty. Treat such paths as outside the layout and
proxy them uncached, which keeps every existing cache entry valid.
