- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
Dates were kept in each process's rewrite cache, so replicas sharing a
database and storage dated independently: one dated a document, upstream
changed in that second, and another gave the new document the same date,
so a client holding it got 304 and kept the old document.
- Date from metadata_dates instead of process memory: a package's date is
the second it started serving its current representation, recorded once
for every replica
- Switch a package to another representation only in a second later than
its recorded date, and serve it undated until then
- Date nothing in the second the process started, so dates clients hold
from before this change, or from the process it replaced, stay older
- Serve no date ahead of the local clock, recorded by a replica whose clock
is ahead
- Date rewrites without the rewrite cache too; the cache keeps only ETags
- Add two-replica, pre-fix date and clock skew regression tests
Two false 304s remained. A rewrite kept the date it was made, so when
upstream went from A to B and back to A, A's cached rewrite came back with
a date earlier than B's and a client holding B got 304. And a second
rewrite in the same second was dated a second ahead of the clock, a date
only that process remembered: after a restart a changed document could be
dated earlier than one the client held.
- Record which rewrite each package is serving and since when, and date a
rewrite by the second its package switched to it, including back to one
still cached
- Never date ahead of the clock: a rewrite switched to in a second that
already dated another for the package goes undated until the next second
- Date nothing in the second the cache starts, the last second the previous
process may have dated a rewrite in
- Drop a package's record when its current rewrite is evicted, so records
stay bounded by the cache; forgetting one dated in the current second
holds back dates for new packages until the next second
Last-Modified was upstream's date clamped to the process start, so
different documents could share a date: after a restart, a cached document
and the newer one a refresh fetched were both dated at startup, and a client
holding the first got 304 for the second.
- Date each rewrite by when the rewrite cache first made it, and keep that
date while the entry stays cached
- Give every new rewrite of a package a date later than the last one it got,
even within the same second or after its entry was evicted, so no two
representations share a date; dates are remembered only until the clock
passes them
- Send no Last-Modified when the rewrite cache is off, since nothing records
when an uncached rewrite was made; cooldown still sends none
- Drop the upstream Last-Modified from metadata lookups, which nothing uses
now
- With cooldown on every rewrite runs, and hashing each rewritten body for
its ETag made every npm metadata request about 20% more expensive
- Hash what fixes the output instead: the upstream digest, proxy URL,
package name, the versions cooldown and the denylist kept, and a
per-process nonce standing for the startup configuration
- Carry the stored document's digest and upstream Last-Modified with the
metadata lookup result, so neither is read from a second cache row lookup
that a concurrent refresh could have changed
- Fall back to hashing the body when the digest is unknown, and keep the
body hash with cooldown off, where it is computed once per cached rewrite
- Rewritten metadata went out with only Content-Type, so pnpm, npm and
Composer could never revalidate and downloaded every document in full
- ETag is the SHA-256 of the rewritten body, computed once per rewrite and
kept with the rewrite cache entry; exact under cooldown as well
- Last-Modified is the upstream Last-Modified, never earlier than the process
start, since a restart can change the rewrite; omitted with cooldown on
- Answer If-None-Match, and If-Modified-Since when there is no If-None-Match,
with 304 on the rewrite fast path and on the fetch path
