A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Checksum and .asc files beside a versioned artifact are as immutable as the
artifact, but were routed through the metadata path: uncached until a
first successful fetch, with no retry or circuit breaker. A build pulling
hundreds of artifacts sent as many checksum requests straight to Maven
Central, which rate-limited the proxy, after which every further checksum
request still went upstream and failed.
Only maven-metadata.xml and its own checksums stay on the metadata path,
since the version listing changes with each release.
TestServerKeepsFetchAfterClientHangsUp gave the upstream fetch 5s to start
and the follow-up request 5s to be answered. On a Windows CI runner, where
the whole test took 13s, the follow-up timed out awaiting headers with
nothing wrong. The test checks the outcome, not the speed, so allow 30s
for both. A real hang still fails.
A version's integrity is a single value shared by every archive cached
for it, but a dev branch caches one archive per commit. Recording the
listed head's shasum made a cache hit for an older pinned commit fail its
check, so its archive was discarded and fetched again, which cleared the
head's recorded shasum in turn. The listed shasum only ever describes the
current head, and Composer checks dist.shasum itself.
- Skip the shasum for dev versions
- Test a pinned commit, then the head, then both again from the cache
npm versions published before dist.integrity existed carry only a SHA-1
dist.shasum, and Composer metadata lists only a SHA-1 dist.shasum, so
neither could be checked.
- Accept native integrity given as an algorithm-prefixed hex checksum,
verified with integrity's exact-digest check, so a recorded "sha1-<hex>"
is also checked on cache hits
- npm: fall back to dist.shasum when dist.integrity is missing or cannot
be parsed; a usable dist.integrity still takes precedence
- Composer: verify an archive against its dist.shasum, except another
commit of a branch fetched in place of the listed one, which the listed
shasum does not describe
- Fetch unverified as before when no shasum is listed, as Packagist does
for GitHub archives, or it is not a SHA-1 digest
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
Package routes share an origin with /ui, and serve bodies whose
Content-Type comes from upstreams or from whoever pushed a hosted package.
An artifact served as text/html or image/svg+xml ran its scripts with
access to the UI.
- Send X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox
on every package-route response: cached and streamed artifacts, cached
metadata, relayed upstream responses and errors
- Archive files browsed from /ui already get the same two headers
The proxy cached the first tarball upstream sent for a version without
checking it against the packument, so a tampered tarball was served to
every client after it.
- Pass the packument's dist.integrity to the existing integrity checks, so
a fresh tarball is verified as it is stored or streamed; a mismatch is
discarded and returns 502, or aborts the streamed response
- Record it with the cached version, so cache hits are verified against it
through the same check as before
- Verify with integrity.Result.Verify, which accepts any digest of the
strongest algorithm listed
- Fetch unverified as before when dist.integrity is missing or cannot be
parsed, or the packument cannot be fetched
Docker sends one Accept header per media type. Reading only the first
forwarded just that type upstream and ruled out cached manifests of the
others, so pulls failed when the upstream was down.
- GetLocalArtifact went through checkCache, which reports a miss when StreamArtifacts is set, so npm and Composer downloads fetched the upstream archive instead of the pushed one
- Split readStored out of lookupCache and read pushed artifacts through it regardless of StreamArtifacts
- Keep the denylist check and the 503/404 answers for unreadable or missing pushed artifacts
- Test npm and Composer downloads of pushed archives with StreamArtifacts set
Refs #133
Pushing and downloading refuse denied versions, but npm and Composer metadata listed every pushed version. A version denied after it was pushed stayed listed, so resolvers picked it and then got 403 on the download. localVersions now leaves denied versions out, which covers both ecosystems and the hosted-only namespaces; npm latest falls back to the highest allowed version. /api/hosted still lists them, so operators can see and delete them.
Refs #134
The manifest lookup took the first composer.json at the root or inside a top directory, in zip order. Dist archives keep composer.json files for tests and examples, so tests/composer.json stored before the root file refused the push with a name mismatch. The root composer.json now always wins. Without one, the manifest inside a top directory is used only when there is exactly one; more than one refuses the archive as ambiguous.
Refs #134
checkCache answered nil both when a stored object couldn't be opened and after an integrity failure had cleared the record. Download handlers then fetched the upstream archive for the same version and filename, and recording it turned the pushed row into an upstream one without its manifest and queued the pushed object for deletion. One storage error while downloading a patched version served the unpatched upstream archive and dropped the push.
checkCache now looks at the record on a miss. For a pushed artifact it returns ErrHostedArtifactUnavailable when the object is recorded but couldn't be served, and ErrHostedArtifactMissing when no object is recorded any more. Every path that would fetch upstream goes through it, so none of them can replace the pushed artifact. Downloads answer 503 and 404 for the two.
Refs #133
The artifact source change added source = excluded.source to the packages upsert as well. source isn't in that INSERT's column list, so every package upsert set the column to NULL. The packages upsert is back to what it was, with a test that an existing source survives it.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
The packument lists pushed versions without cooldown, but the tarball download checked cooldown against the upstream's publish times before looking at the cache. A pushed override of a version the upstream published recently was listed and then withheld with 404. Pushed tarballs are now served right after the denylist check.
Refs #134
With hosted packages enabled, a package's p2 metadata lists its pushed versions, built from composer.json without the root-only keys, with a zip dist at the proxy and the time they were pushed. dist.reference is the archive's sha256 so a re-push isn't served from Composer's cache, and shasum is left empty. Dev versions go to the ~dev file and tagged versions to the other.
A pushed version takes the place of an upstream entry with the same version, with or without a leading v. Pushed entries follow the upstream ones and go through the same minified writer, so fields an earlier upstream entry set and the pushed one lacks are written as __unset rather than inherited.
When the upstream doesn't know the package, the document holds only the pushed versions. Like npm, documents with pushed versions bypass the rewrite cache.
Refs #134
With hosted packages enabled, the packument of a package with pushed versions lists each of them, built from its package.json with dist pointing at the proxy and carrying the archive's sha512, and the time it was pushed. A pushed version takes the place of an upstream entry with the same version. Cooldown and the denylist still filter only upstream versions, and dist-tags stay as upstream has them.
When the upstream doesn't know the package, the packument holds only the pushed versions, with latest set to the highest of them. Requests for a single pushed version are answered from its entry.
Rewrites that include pushed versions bypass the rewrite cache, which is keyed by the upstream document alone, so a push or delete shows on the next request and changes the ETag. Packages without pushed versions take the cached path as before, and with hosted packages disabled there is no extra lookup.
Refs #134
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest).
The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage.
The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity.
Refs #134
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
hosted.enabled turns on pushing packages to the proxy and is off by default, like mirror_api. A pushed version is served in place of the upstream version with the same number, so anyone who can write could swap a package for every client. Enabling it therefore requires hosted.token, which supports ${VAR_NAME} expansion and can be set with PROXY_HOSTED_TOKEN. hosted.max_upload_size caps one archive and defaults to 100MB.
requireBearerToken compares SHA-256 digests of the presented and configured tokens in constant time and answers 401 with a WWW-Authenticate challenge otherwise.
Refs #133
Artifacts get a source column: upstream for cached copies of a registry, local for archives pushed to the proxy. Existing rows become upstream through the new 012_add_artifact_source migration.
Local artifacts aren't a cache, so the LRU size limit doesn't count or evict them and storage retention never selects them. They are only removed by deleting them. A local artifact has no upstream URL and stores it as empty.
Refs #133
The compare endpoint buffers both archives whole, up to 512 MB each, and
anyone who can reach the UI can call it. A handful of parallel requests for
large cached versions was enough to exhaust the proxy's memory.
- Run at most two diffs at once; refuse more with 503, code BUSY and
Retry-After instead of queueing them
- Keep the per-archive cap, which legitimately large packages need
- Move the compare test's setup into a helper shared by the new tests
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
Handlers set the configured upstream auth header on the request itself
before handing it to the client. When upstream redirected to another host,
Go's client copied that header to the next hop: it strips only
Authorization and Cookie, so a "header" type credential such as
X-JFrog-Art-Api reached the redirect target. The auth transport already
applies credentials per hop, scoped to each hop's URL.
- Remove Proxy.applyUpstreamAuth and Proxy.AuthForURL; auth comes only
from the transport
- Add a server test that redirects a header-authenticated upstream to
another host and asserts the header stays with the upstream
- Wire handler tests' auth through the transport, as the server does
The compare page loaded diff2html from cdn.jsdelivr.net without a version
or integrity hash, so whatever jsDelivr served as the latest release ran
with the proxy origin's privileges, and the page broke where the proxy
cannot reach the internet.
- Vendor diff2html 3.4.56 (MIT) under static/vendor like tailwind and lucide
- Load its script and stylesheet from /ui/static/vendor
- Fail a test when any template loads a script or stylesheet from another origin
The source browser built its file tree with onclick handlers such as
loadFile('${escapeHTML(file.path)}'). The browser decodes HTML entities in
an attribute before running the handler, so the ' that escapeHTML
produces turns back into a quote and ends the string. A cached package
with a crafted file name ran script on the proxy origin for anyone who
browsed it.
- Carry the path in data-path and dispatch clicks from one delegated
listener on the file tree
- Render the binary download button as a link instead of an onclick
that assigns window.location
- Fail the template tests when an inline handler interpolates data
