A version's first number was taken as its major line, so certifi
2024.8.30 and tzdata 2024.2 never got the next year's release. A first
number of 1000 or more is a year, and all such versions are one line.
Refresh called mirrorOne on the scheduler's goroutine without the
recover Run puts around each item, so a panic took the proxy down.
Both now go through mirrorRecovered, which counts it as a failure and
carries on.
The refresh read InRelease, APKINDEX and repomd.xml again through the
metadata cache, which marked them accessed, so every suite looked
fetched within since and never aged out. Read them straight from
upstream, as the Packages and primary indexes already are.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
security.debian.org keeps files under pool/updates/main while its
InRelease lists main/binary-*, so taking the first segment after pool/
as the component found no index and security suites were never
refreshed. Take the first or second segment, whichever the release
file lists indexes for.
A pkg:git version can now be a tag as well as a full commit ID. With a
version, the submodules the commit records are mirrored at the commits it
pins, from the upstream.git hosts serving them; one no host serves is
skipped with a warning.
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
An index file was mirrored only when it was newer than every cached
version of the package and architecture. Debian suites share one pool, so
once curl from trixie was cached, bookworm's security updates to curl
were older than it and never mirrored. A run now mirrors the newest file
an index lists when it isn't cached and an older version is, which still
never downgrades and leaves a suite alone that only has a newer version
cached.
Tests also cover repositories listing several versions, an unreadable
index next to working ones, Architecture: all packages, scoped npm
packages, denied releases, a listing failure next to other packages, a
cancelled run and Go version lines.
These clients find new versions in a repository index. The refresh fetches
the InRelease, APKINDEX and repomd.xml files clients fetched again, reads
the Packages and primary indexes they point at for the repositories the
most used packages came from, and mirrors the newer file per package and
architecture, checked against the index's SHA-256 where it has one.
mirror.refresh mirrors, for the top packages per ecosystem by recent
downloads, the newest release of each major line a version is cached on.
Releases within cooldown or on the denylist are skipped, and PyPI and conda
get only the kinds of files clients fetched before. max_versions and
max_bytes cap one run. Instances sharing a database take turns as for
schedules.
* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
packageurl.HandlerPackage returns the ecosystem and name the handlers store
a package URL under. The mirror and the inventory both use it, which
settles where they disagreed: distribution namespaces of apk, alpm, bsd,
deb and rpm are dropped, PyPI and NuGet names use purl's comparison form,
and Maven names are group:artifact as the handler stores them, where the
mirror used group/artifact.
* Add --latest to mirror only the newest version
An unversioned PURL mirrors every version the upstream lists. With
--latest, or "latest": true in a mirror API job, only the newest is
mirrored: the highest by the ecosystem's version ordering, without
yanked or retracted versions and, unless there is nothing else,
pre-releases.
* Pick the latest version with registries.FetchLatestVersion
--latest now uses the shared policy from git-pkgs/registries: the version
the registry advertises as latest where it has one, otherwise the newest
active version by publish time. Deprecated versions are left out too, and
newestVersion and the direct vers dependency go away.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
