* Coalesce concurrent metadata fetches
Concurrent cache misses for one artifact already share a single upstream
fetch (#329), but metadata misses did not: every request for a package's
metadata went to the registry, even when the same document was already
being fetched. In CI that is the common case. Composer resolves versions
on every job, since MediaWiki core and its extensions commit no
composer.lock, and each package is one /p2/ document, so a burst of jobs
fetched each document once per job.
Metadata misses now follow the same pattern as artifacts: the first caller
fetches, the rest wait for its result, and the caller that takes the key
rechecks the cache first so a fetch that just finished is not repeated.
The key includes the Accept and Accept-Encoding headers, which change the
bytes upstream returns (npm's abbreviated and full documents), and whether
the caller validates the response.
One difference from artifacts: the shared fetch is detached from the first
caller's cancellation. Artifact fetches keep it for scanning and mirroring,
neither of which applies to metadata, and otherwise one job disconnecting
would fail every job waiting on the same document. runScan detaches the
same way, and the HTTP client's timeout still bounds the fetch.
This covers every ecosystem that goes through fetchOrCacheMetadata
(Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with
metadata caching on or off. Streamed metadata with caching off is not
coalesced.
On an 8 vCPU test instance with caching off, 10 concurrent clients
requesting the npm metadata of 1,263 packages went from 12,630 upstream
requests to 1,898. With cache_metadata on, 20 clients requesting 1,538
packages made exactly 1,538.
* Run each caller's validate on a shared metadata fetch
A caller that joined another's metadata fetch got the shared bytes
without running its own validate. NuGet's cooldown path uses validate to
decode the document for the request, so a joined download found no
publication date and was allowed, bypassing the cooldown.
coalesceMetadata now reports whether the caller joined, and
coalescedMetadataMiss runs that caller's validate on the shared bytes.
Validation before caching is unchanged: the first caller still runs it
inside the fetch.
TestNuGetCooldownConcurrentDownloads reproduces the bypass through
NuGetHandler.Routes() with two overlapping downloads of a version inside
its cooldown.
