npm metadata rewriting depended on handler.Proxy, which pulls SQL
drivers, cloud SDKs and metrics into its import graph. Move the
transformation into internal/metadata/npm with explicit policy,
evaluation time, public URL and hosted-version inputs, and share the
JSON scanner between npm and Composer through internal/jsonscan.
Rebased onto main: rewritten tarball URLs are still logged once per
document, now through an OnTarballsRewritten callback that reports the
count.
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
With cache_metadata on, a failed upstream metadata fetch fell back to the
cached copy for any error. A 404 (package removed) and a 401/403 (the
proxy's credentials rejected) are definite answers, though, and serving
the old version list in their place kept them from reaching the client.
They now pass through: 404 as 404, 401/403 as 502 like other upstream
failures. Timeouts, connection errors, 5xx and 429 still fall back.
The npm, Composer and Chef handlers serve metadata through the rewrite
cache, which dropped the stale flag, so a fallback looked like a fresh
200. They now set the same Warning: 110 header as other metadata.
Fixes#499
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
With hosted packages enabled, the packument of a package with pushed versions lists each of them, built from its package.json with dist pointing at the proxy and carrying the archive's sha512, and the time it was pushed. A pushed version takes the place of an upstream entry with the same version. Cooldown and the denylist still filter only upstream versions, and dist-tags stay as upstream has them.
When the upstream doesn't know the package, the packument holds only the pushed versions, with latest set to the highest of them. Requests for a single pushed version are answered from its entry.
Rewrites that include pushed versions bypass the rewrite cache, which is keyed by the upstream document alone, so a push or delete shows on the next request and changes the ETag. Packages without pushed versions take the cached path as before, and with hosted packages disabled there is no extra lookup.
Refs #134
