CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
ScanPackument and LookupVersion read with a jsontext.Decoder, skipping
values they don't keep with SkipValue and reading the one LookupVersion
returns with ReadValue, instead of decoding each into a json.RawMessage.
Repeated names and invalid UTF-8 are still accepted as encoding/json
accepts them. Go 1.27 is now required, for encoding/json/jsontext.
Benchmarks for both use the 5,000-version packument the handler benchmarks
build, now shared in internal/metadata/npm/npmbench.
A fetched packument is now stored under a staging path and checked before
it replaces the cached copy, so an abbreviated one no longer overwrites a
full copy; the full copy is served as when upstream fails. A failure to
store falls back to the in-memory path, as caching is best effort there.
Requests for a single version or dist-tag and the tarball download's
cooldown and integrity lookup read the stored packument as a stream too,
keeping only the one version entry they need.
Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
