A body that ended before its Content-Length was read as complete in
memory: ReadMetadataSized took the transport's io.ErrUnexpectedEOF for
the end of a body shorter than its size hint. The half document was
served and cached, or failed the filter with a 502, instead of the stored
copy being served. Upstream bodies are now read through a reader that
marks their read errors, and such a failure is an upstream failure on
both paths: streamed, it no longer counts as a failure to store, which
had sent the request to the bounded in-memory path.
A stored packument whose bytes no longer match its row's digest is now
fetched again like a missing one.
Tests also cover concurrent requests sharing one fetch, client
revalidation and upstream failures with nothing cached. The stream tests
build cooldown through cooldown.NewPolicy as the server does, since a
bare cooldown.Config parses itself on first use without locking.
A row within its TTL was served without checking its blob, so a missing
blob failed the streamed read and the request fell back to the bounded
in-memory path. A read that finds the stored copy missing now fetches the
packument again into storage, ignoring that row, and reads it once more,
for packument, version and tarball requests.
A fetched packument is now stored under a staging path and checked before
it replaces the cached copy, so an abbreviated one no longer overwrites a
full copy; the full copy is served as when upstream fails. A failure to
store falls back to the in-memory path, as caching is best effort there.
Requests for a single version or dist-tag and the tarball download's
cooldown and integrity lookup read the stored packument as a stream too,
keeping only the one version entry they need.
Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
