Every PyPI download with cooldown on read the JSON API first, cached
files included. Without cache_metadata that was a full /pypi/<name>/json
per file.
A download that looks up the JSON API now stores the release's newest
upload time on its versions row. A cached file is then checked against
the stored time without asking upstream. A file that isn't cached is
still looked up, since files can be added to a release later, and the
stored time is refreshed. A stored time inside the window withholds the
release without a lookup.
A Go module's .mod and .info for a canonical version, and RPM repodata
files named after their checksum, never change once published. They
went through the metadata TTL, so each cost a revalidation every TTL with
cache_metadata on and a full fetch on every request with it off.
They are now kept in the metadata cache with or without cache_metadata
and served whatever their age. A stored copy is checked against its
digest and fetched again if it doesn't match. Go queries, @v/list,
@latest and repomd.xml stay on the TTL.
Nothing set GOMEMLIMIT, so in a container the heap could grow until the
container was OOM-killed while much of it was garbage, for example while
large metadata documents were being rewritten. At startup on Linux, read
the cgroup v2 or v1 memory limit and set the runtime's soft limit to 80%
of it, leaving room for git subprocesses. An explicit GOMEMLIMIT wins.
A HEAD on a cache hit opened the stored object and closed it unread.
On S3 and Azure that is a GET whose body is thrown away, which can also
drop the pooled connection; on GCS it is a GET as well.
The server now marks HEAD requests in the request context, and readStored
answers a marked cache hit from the database record. Size, content type
and ETag already came from the record. Accept-Ranges depended on the
opened reader being seekable, so backends now say whether their readers
are (Blob does, GCS does not) and the HEAD response matches the GET.
A HEAD still counts as a cache hit, as before. DirectServe redirects are
unchanged since they are decided before the object would be opened.
ReadMetadata grew its buffer with io.ReadAll, which allocates about 2.5
times the document size in total. Cached documents have a recorded size
and most upstream responses a Content-Length, so read those into one
buffer of that size. The size is a hint: a missing, wrong or over-limit
one still reads the body correctly and enforces the limit.
Store passed empty writer options, so the S3 driver allocated six 8 MiB
part buffers up front for every object over 16 MiB, on top of reading
the first 16 MiB whole, and Azure used five 8 MiB blocks. Use 5 MiB
parts, the S3 minimum, with two uploads in flight.
A 32 MiB store against a fake S3 endpoint allocates 54 MB instead of
88 MB.
A cache hit hashed the stored object with SHA-256 for content_hash and
again with every native algorithm in versions.integrity, SHA-512 for npm.
The native integrity was already checked when the bytes were stored and
content_hash was computed from the same bytes, so the second hash adds
CPU per byte served without catching anything more. Records without a
content hash still fall back to the native integrity.
Each download and metadata handler logged its own request line at info,
next to the request line LoggerMiddleware already writes for every
request with path, status and duration. That doubled the info log volume
on the hot path. The handler lines only repeat values parsed from the
path, so they move to debug.
Kept at info: the npm audit line (it carries the request body size), the
julia package line (it carries the name resolved from the UUID) and the
gem and hex cooldown lines.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
Mirroring a PyPI version passed each file's sha256 as the integrity
GetOrFetchArtifactFromURLWithIntegrity records with the version. Wheels of
one version share the name and version they are cached under, so the last
file's checksum replaced the others', and serving the other wheels from
the cache failed the integrity check.
The mirror now passes the sha256 as a digest, which is checked against
that file's content hash alone.
A streamed cache hit whose bytes don't match the row's digest now
records an integrity failure and deletes the row, queueing its blob for
deletion, as a corrupt cached artifact is handled. The next request
fetches the document from upstream instead of aborting until the TTL
runs out.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
PyPI and npm downloads look up the publish time before the cache. When
that lookup fails, a file that is already cached is now served, and only
a cache miss fails with 502 or 503. The proxy keeps serving what it has
while the upstream is unreachable, and nothing new is fetched unchecked.
Also drops the unattached comment block in cooldown_fail_closed.go.
The case-variant lookup filtered on lower(name), which no index covers,
so a legacy row's first lookup scanned every metadata_cache row in its
ecosystem. Reclaim already skips a queued path a row still points at,
so each row now queues just its own legacy blob once it has moved off
it. On case-insensitive storage a case variant still at its legacy path
can lose its blob that way, and is then fetched again.
With cache_metadata on, every metadata cache hit read the whole stored
document into its own buffer before any coalescing, since
coalesceMetadata only covers upstream misses. metadata_max_size bounds
one request, not the sum: 30 concurrent hits on a 20 MB document held
about 820 MB.
The handlers that go through ProxyCached serve the stored document
unchanged, so a hit there no longer needs the bytes in memory.
serveCachedMetadata streams the blob from storage to the response
instead, with the row's Content-Type, Content-Encoding, size, ETag and
Last-Modified, and answers If-None-Match and If-Modified-Since with 304
as before. The bytes are checked against the row's content digest as
they are sent, and a mismatch aborts the response like a corrupt cached
artifact. A row without a digest or size, a document over the size
limit, or a blob that can't be opened takes the buffered path as before.
Handlers that rewrite or filter metadata keep reading it into memory,
and the rewrite cache stays in front of them. writeMetadataValidators
is split out of writeMetadataCachedResponseWithEncoding so both paths
share the conditional request handling.
registries main streams npm packuments (git-pkgs/registries#109), so
listing the versions of packages whose packument is over 10 MiB, such as
typescript, no longer fails with "unexpected end of JSON input". It also
reads Maven versions from maven-metadata.xml (#114).
go.mod points at a pseudo-version of registries main until those are
released.
registries' safehttp now checks the destination of requests sent through
a forward proxy (git-pkgs/registries#116), so the proxy's own guard is
removed and newUpstreamClient is back to its earlier form.
go.mod points at a pseudo-version of registries main until #116 is
released.
The per-ecosystem rules for when two spellings name the same package
(PEP 503, NuGet case, Composer's leading v, conda channel, OCI paths,
Swift registry case) now live in purl. The denylist keeps only the
proxy's own pkg:generic/swift-registry spelling and the gem platform
match. Keys, cache paths and stored PURLs are unchanged.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
Ignoring every row still at its unhashed path dropped the whole metadata
cache on upgrade, including on Linux and S3 where keys never collided,
and left nothing to serve while the upstream was down.
A legacy row's blob is now hashed once and compared with the row's
content_digest. When it matches, the blob is this key's document: it is
copied to the hashed path and the row moved there, so it is served
fresh or as a stale fallback as before. Only a mismatch, where another
key's document overwrote the shared file, or a row without a digest is
dropped and fetched again.
The legacy blobs are queued for deletion once no key differing only in
case still reads its legacy path, which on a case-insensitive
filesystem is the same file.
Debian, RPM, APK, PyPI, CRAN and Maven build upstream URLs from the
decoded request path, where a '?' or '#' would end the path and cache
one file under another's name. Only the server middleware refused them;
Arch and FreeBSD also check in the handler. Check in these handlers
too, so they stay safe when mounted without the middleware.
With cooldown on, the Hex handler read the /packages response, its decompressed payload and the Hex API's JSON without a limit, and the gem handler decoded the versions API's JSON the same way. A misbehaving upstream could make the proxy hold any amount in memory, and a small gzip payload could expand without bound.
These reads now go through ReadMetadata, so metadata_max_size applies to each, and to the Hex payload after decompression as the NuGet handler does. A response over the limit fails the request with 502.
RubyGems, Hex and PyPI keep publish times outside the metadata clients
read, so cooldown fetches them from a second API. When that fetch
failed, was rate limited, or returned something that didn't parse, the
compact index, Hex package or simple page was served unfiltered, and a
Hex package that couldn't be re-encoded was passed on as upstream sent
it. A flaky or rate-limited timestamp API let every version in
cooldown through.
These now fail with 502, or with 503 and Retry-After when the rate
limit gate held the request, as the Go handler already does. PyPI file
downloads check the same way. A PyPI package the JSON API answers 404
for still has no times and is served unfiltered, as before.
npm, Composer, pub.dev and PyPI JSON metadata that couldn't be
rewritten was also passed on unfiltered unless the package had denied
versions. It now fails with 502 whenever cooldown applies too. npm
tarball downloads whose packument couldn't be fetched skipped the
cooldown check and downloaded straight from upstream; they now fail.
All of this only applies when cooldown covers the package: packages
exempted with a duration of 0 skip the timestamp lookups, and the gem
and Hex handlers serve them through the metadata cache as when cooldown
is off.
upload-pack was started with its stdin attached to the request body, after
the read deadline had been extended to two hours, and ls-refs requests took
no slot. A client that sent part of a request and then nothing kept an
upload-pack process waiting, and any number of them could.
Stateless requests are complete once sent, so the handler now reads the
body in full first, within the server's read timeout, keeping bodies past
1 MiB in a temporary file. Only then are the deadlines extended for the
response. info/refs and ls-refs share their own slots, four per CPU, so they
still never wait behind clones. Each response write is bounded to two
minutes, so a client that stops reading a pack frees its slot instead of
holding it until the request timeout.
Conan, CRAN, Conda, RPM, the main Debian archive and the default Maven
repository keyed cached metadata by the request path with '/' replaced
by '_'. Names and versions may contain '_', so foo/bar_1.0 and
foo_bar/1.0 shared an entry, and with cache_metadata one reference's
recipe files were served for the other. Conan also appended the query
with '_', so a path without a query could collide with a shorter path
plus one.
Hash the upstream URL, the path and, for Conan, the query instead, as
the APK, Arch, generic and Helm handlers already do. Each part is
length-prefixed so no two inputs share a key.
