proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
- Keep the download URLs of a batch response in memory until shortly
before they expire, so the GET that follows needs no second batch
request. A kept URL that fails is replaced by a fresh one.
- Cut off a download that isn't the object's size.
- Copy an object whose content is already stored, such as for a fork,
instead of downloading it, once the upstream has confirmed the
repository has it. Artifacts are indexed by content hash for this.
- Keep signed download URLs out of logs and the database: the fetch uses
the signed URL, everything else the URL without its query.
- Answer a batch from the cache when the upstream can't be reached or
fails, so git-lfs can use cached objects during an outage. Refusals
are still passed on.
- Answer a HEAD for an uncached object from the batch response instead
of downloading it.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
The proxy filters repodata.json and current_repodata.json, but
repodata.json.bz2 went through the metadata cache unfiltered, and
repodata.json.zst and repodata.jlap reached the download handler and were
passed through. conda tries repodata.json.zst first, so with cooldown on it
got versions still in cooldown.
With cooldown on these now answer 404, and conda falls back to
repodata.json. repodata_from_packages.json is filtered like repodata.json.
With cooldown on, the RubyGems compact index and versions API, the Hex
package and Hex API entry, and conda's repodata.json were fetched with
plain HTTP requests on every client request: no TTL, no revalidation,
no coalescing and no stale fallback, even with cache_metadata.
Gem and Hex now read both documents through the metadata cache and
filter the cached copies. The filtered result goes through the rewrite
cache and is served with an ETag. When the times document is older than
the index and misses a version, it is fetched again; if that fails the
version is left out. Conda filters a fresh stored repodata.json as it
reads it from storage, uses the metadata cache otherwise, and remembers
the ETag of the filtered copy until a withheld entry leaves cooldown.
SQLite used one connection for everything, so request lookups queued
behind the per-minute stats queries, dashboard pages and eviction scans.
Writes keep the single connection. Get, Select, Query and QueryRow now go
to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode
lets run alongside the writer. A write sent to that pool fails.
Pragmas move into the DSN so every connection gets them. Postgres and
in-memory SQLite keep one pool for both.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
The Helm index.yaml and PyPI simple pages and JSON API documents were
parsed and rewritten on every request, cache hits and conditional
requests included. They now go through the rewrite cache like npm and
Composer, so a repeat request is served from memory with an ETag of
its own, a dated Last-Modified, a gzip copy and 304 when it matches.
PyPI simple pages are keyed by their content type and the versions
filtered out as well as the upstream page. Download hrefs are now
rewritten with a literal prefix search that makes the same changes as
the old regular expression, and the link filter's expressions are
compiled once.
A chart cache miss no longer parses the whole Helm index again while
the stored index is unchanged: the chart downloads it lists are kept
per repository, keyed by the index digest.
A stale tag was revalidated by repeating the client's GET with
If-None-Match. Docker Hub counts manifest GETs against its pull limit,
so every TTL expiry of a tag could cost a pull. The tag is now resolved
with a HEAD and the Docker-Content-Digest compared with the cached one;
the manifest is fetched by digest only when that digest isn't cached.
A HEAD without a digest falls back to the conditional GET.
A client HEAD whose digest is cached now points the tag at it, so the
GET that follows is a cache hit. Concurrent requests for the same
manifest share one upstream exchange, and instances sharing Postgres
take turns through the fetch lease.
Every npm and Composer download ran GetLocalArtifact, a full artifact
row lookup, even with hosted packages disabled. localVersions already
returns early in that case, so metadata does not list pushes, and
/generic/hosted/ is only served while hosting is enabled.
GetLocalArtifact now returns nil without a lookup when hosting is off.
Downloads then take the normal proxy path. An archive pushed earlier is
still answered from storage by that path and is never replaced by the
upstream copy; only cooldown now applies to it, as metadata no longer
lists it either. The hosted docs say so.
Every PyPI download with cooldown on read the JSON API first, cached
files included. Without cache_metadata that was a full /pypi/<name>/json
per file.
A download that looks up the JSON API now stores the release's newest
upload time on its versions row. A cached file is then checked against
the stored time without asking upstream. A file that isn't cached is
still looked up, since files can be added to a release later, and the
stored time is refreshed. A stored time inside the window withholds the
release without a lookup.
A Go module's .mod and .info for a canonical version, and RPM repodata
files named after their checksum, never change once published. They
went through the metadata TTL, so each cost a revalidation every TTL with
cache_metadata on and a full fetch on every request with it off.
They are now kept in the metadata cache with or without cache_metadata
and served whatever their age. A stored copy is checked against its
digest and fetched again if it doesn't match. Go queries, @v/list,
@latest and repomd.xml stay on the TTL.
Nothing set GOMEMLIMIT, so in a container the heap could grow until the
container was OOM-killed while much of it was garbage, for example while
large metadata documents were being rewritten. At startup on Linux, read
the cgroup v2 or v1 memory limit and set the runtime's soft limit to 80%
of it, leaving room for git subprocesses. An explicit GOMEMLIMIT wins.
Store passed empty writer options, so the S3 driver allocated six 8 MiB
part buffers up front for every object over 16 MiB, on top of reading
the first 16 MiB whole, and Azure used five 8 MiB blocks. Use 5 MiB
parts, the S3 minimum, with two uploads in flight.
A 32 MiB store against a fake S3 endpoint allocates 54 MB instead of
88 MB.
A cache hit hashed the stored object with SHA-256 for content_hash and
again with every native algorithm in versions.integrity, SHA-512 for npm.
The native integrity was already checked when the bytes were stored and
content_hash was computed from the same bytes, so the second hash adds
CPU per byte served without catching anything more. Records without a
content hash still fall back to the native integrity.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
RubyGems, Hex and PyPI keep publish times outside the metadata clients
read, so cooldown fetches them from a second API. When that fetch
failed, was rate limited, or returned something that didn't parse, the
compact index, Hex package or simple page was served unfiltered, and a
Hex package that couldn't be re-encoded was passed on as upstream sent
it. A flaky or rate-limited timestamp API let every version in
cooldown through.
These now fail with 502, or with 503 and Retry-After when the rate
limit gate held the request, as the Go handler already does. PyPI file
downloads check the same way. A PyPI package the JSON API answers 404
for still has no times and is served unfiltered, as before.
npm, Composer, pub.dev and PyPI JSON metadata that couldn't be
rewritten was also passed on unfiltered unless the package had denied
versions. It now fails with 502 whenever cooldown applies too. npm
tarball downloads whose packument couldn't be fetched skipped the
cooldown check and downloaded straight from upstream; they now fail.
All of this only applies when cooldown covers the package: packages
exempted with a duration of 0 skip the timestamp lookups, and the gem
and Hex handlers serve them through the metadata cache as when cooldown
is off.
The upstream client refuses loopback, private and link-local addresses
when it dials, but with HTTP_PROXY or HTTPS_PROXY set it dials the
forward proxy, so only the forward proxy's address was checked. A URL
taken from upstream metadata, such as a Helm chart URL, a Composer dist
URL, a token realm or a redirect, could then reach 169.254.169.254 or
an internal host through the forward proxy.
The transport's proxy function now resolves the destination of every
request it sends to a forward proxy and applies the same rules,
including upstream.allow_loopback and upstream.allow_private_hosts,
before handing it over. A name that cannot be resolved locally is still
passed on, since some networks only resolve public names at the
forward proxy. Direct requests are unchanged.
Git fetches and token requests use the same transport, so they are
covered too.
