The state shown on the git page was read again after every check of the
upstream, and a check runs on every client info/refs. On a busy proxy
each view then read the same repositories again and never got past what
fits in two seconds.
The state is now kept until the repository's refs or HEAD are written: by
a clone, a fetch, a fetch of wanted commits, or a change of the default
branch. A check that finds nothing new leaves it in place.
A state that fails to read for another reason than the view's deadline is
now logged at warn with the repository key.
Retention evicts every file that has not been served for the configured
time, also the newest version of a package. keep_latest: N holds back
the files of the N highest cached versions of each package from
eviction by age. Versions are ordered by the ecosystem's version rules.
It applies to cargo, golang, npm and pypi, where the cache stores a
release under its version. Other ecosystems ignore it for now.
A view ran four git processes for each repository whose state wasn't
cached, with no limit, and each concurrent view did the same.
RepositoryState now runs one read at a time and gives up with its
context. The page reads for at most two seconds, lists the rest without
branches and commits, and says how many are left; the cache keeps what
was read, so each reload reads more.
Also test that an upstream, a repos pattern and a repository with
uppercase letters are listed as spelled, and correct two comments.
A Debian package was cached under its repository and file name, but
fetched from its full path. Docker's apt repository has a pool under each
suite with the same file names, so a bookworm client could get the
bullseye build that was fetched first, and fail its hash check. Packages
are now cached under their path.
Conan joined revision, package reference, package revision and file name
with underscores, which revisions can contain, so a recipe file and a
package file could share a cache entry. The parts are now joined with
slashes.
Entries cached under the old names are still served, but only for the
URL they were fetched from, so warm caches keep working.
Since main gained the repos allowlist and the refusal of repositories the
upstream refused, the page could still list a repository the allowlist no
longer allows and show the cached commit of one the upstream refused.
Repositories now reports Refused, and the page leaves out what the
allowlist doesn't allow and shows no cached state for a refused one.
The docs say refreshing Debian, Alpine and RPM packages needs
cache_metadata, but nothing said so when it was off. Rejecting the
setting would also stop the refresh of every other ecosystem, so the
server logs a warning from the new Config.Warnings at startup.
Repository paths can hold spaces, '#', '?' or '%', as Azure DevOps
project and repository names do, so each segment is escaped as the LFS
URLs are. The requests label also uses plural() now.
Git LFS objects are artifacts under ecosystem git, so adding the
repositories to the per-ecosystem gauges mixed the two and broke the
per-ecosystem figures adding up to the totals. The git cache now has
proxy_git_cache_size_bytes and proxy_git_cached_repositories, updated
on each tick apart from the database gauges so a failed stats query
leaves them current.
RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
The summary said "1 repositories" and counted the size of repositories it
didn't list. A repository whose state can't be read is no longer read on
every page view, only after its next upstream check.
/ui/git lists the git cache's repositories with their upstream, clone URL,
size, last check and use, requests served and the last refresh error. The
per-ecosystem cache size and artifact gauges report the git cache as
ecosystem="git".
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
The API handler takes the enrichment calls it uses as an interface, so
the test can answer them without reaching a registry.
Remove FillVersionPublishedAt until the shared cooldown filter calls it.
metadata_max_size bounds what is read into memory, but a packument
streamed from upstream into storage had no limit, and with cooldown was
stored twice. It is now capped by metadata_stream_max_size, 1 GB by
default. A document over the cap fails as an upstream failure: nothing is
stored, and the stored copy, if any, is served instead.
The metadata settings move into configureMetadata to keep serve under
the maintidx limit.
