2026-01-29 16:33:09 +00:00
# Configuration
The proxy can be configured via command line flags, environment variables, or a configuration file. Command line flags take precedence over environment variables, which take precedence over the configuration file.
## Configuration File
Create a YAML or JSON file and pass it with `-config` :
```bash
proxy serve -config config.yaml
```
See `config.example.yaml` in the repository root for a complete example.
## Server Settings
| Config | Environment | Flag | Default | Description |
|--------|-------------|------|---------|-------------|
| `listen` | `PROXY_LISTEN` | `-listen` | `:8080` | Address to listen on |
2026-06-07 16:12:43 +01:00
| `base_url` | `PROXY_BASE_URL` | `-base-url` | `http://localhost:8080` | Public URL package managers use to reach this proxy |
| `ui_base_url` | `PROXY_UI_URL` | - | (defaults to `base_url` ) | Public URL where the web UI is reached. Set separately when the UI lives behind a different hostname than package endpoints (e.g. public domain vs Docker network alias). Used for canonical/og:url tags and the install guide banner. The proxy still serves package endpoints on the same listener, so any reverse proxy fronting the UI publicly should restrict the public route to `PathPrefix(/ui)` to avoid exposing package endpoints. |
2026-10-01 14:32:21 +02:00
| `trust_forwarded_for` | `PROXY_TRUST_FORWARDED_FOR` | - | `false` | Attribute requests to the leftmost `X-Forwarded-For` entry instead of the TCP peer address, in the structured log, the access log and the request-source table on `/ui/analytics` . |
| `ui_request_sources` | `PROXY_UI_REQUEST_SOURCES` | - | `false` | Show the request-source table on `/ui/analytics` , which reports caller addresses, the tool each ran and how much each pulled. |
2026-10-12 09:41:42 +02:00
| `ui_failure_details` | `PROXY_UI_FAILURE_DETAILS` | - | `false` | List the latest failures behind each failure count on `/ui/analytics` : the package, its upstream URL or storage path, and the error. URLs are shown without credentials or query strings. |
2026-01-29 16:33:09 +00:00
## Storage
The proxy stores cached artifacts using gocloud.dev/blob, supporting local filesystem and S3-compatible storage.
### Local Filesystem
```yaml
storage:
url: "file:///var/cache/proxy"
```
Or using the legacy path option:
```yaml
storage:
path: "./cache/artifacts"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `storage.url` | `PROXY_STORAGE_URL` | `-storage-url` | Storage URL (file:// or s3://) |
| `storage.path` | `PROXY_STORAGE_PATH` | `-storage-path` | Local path (deprecated, use url) |
| `storage.max_size` | `PROXY_STORAGE_MAX_SIZE` | - | Max cache size (e.g., "10GB") |
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:28:20 +07:00
| `storage.cache_artifacts` | `PROXY_STORAGE_CACHE_ARTIFACTS` | - | Store fetched artifacts (default: true); `false` streams them from upstream |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| `storage.retention.default` | `PROXY_STORAGE_RETENTION_DEFAULT` | - | Evict artifacts not served for this long (e.g. "30d"); see [Retention ](#retention ) |
| `storage.retention.sweep_interval` | `PROXY_STORAGE_RETENTION_SWEEP_INTERVAL` | - | How often expired artifacts are looked for (default: "10m", at least "1m") |
2026-01-29 16:33:09 +00:00
2026-09-29 10:24:22 -07:00
`storage.max_size` counts cached artifacts only. An artifact replaced by a refetch stays in storage for at least an hour, or `storage.direct_serve_ttl` if longer, so requests already reading it can finish, and storage use can exceed the limit by what was replaced in that time.
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:28:20 +07:00
### Serving artifacts without storing them
With `storage.cache_artifacts: false` the proxy streams every artifact download from upstream to the client and stores no artifacts. Metadata is still filtered and cached, so cooldown and the denylist apply as usual, and the database still holds the publish times cooldown needs. Use it when another caching layer, such as an Artifactory remote repository, sits in front of the proxy and caching artifacts twice only costs storage.
```yaml
storage:
cache_artifacts: false
```
Every download is a fresh upstream fetch, and concurrent requests for the same artifact are not combined. Artifacts with a digest known up front (OCI blobs, Swift archives, Helm charts) are verified while streaming: the response is sent chunked, and on a mismatch the connection is aborted before the response completes so the client never receives a tampered artifact as a good one. The same happens when the upstream connection fails mid-download.
`cache_artifacts: false` cannot be combined with `scanning.enabled` , `storage.direct_serve` or `mirror_api` , which all need stored artifacts, and the `mirror` command refuses to run with it.
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
### Retention
Retention evicts cached artifacts that nobody has downloaded for a while. It works alongside `storage.max_size` : retention removes what has gone unused, and the size limit still applies to whatever is left.
The `ecosystems` and `packages` entries below only pass validation once the ecosystems they name support retention; see the table further down.
```yaml
storage:
retention:
default: "30d"
ecosystems:
npm: "14d"
maven: "0"
packages:
"pkg:npm/lodash": "0"
sweep_interval: "10m"
```
Each cached file ages on its own, counted from the last time the proxy served it, or from when it was fetched if it has not been served since. A newer release of the same package does not shorten the life of older versions. Durations take a `d` suffix for days, and `"0"` or an empty value means never evict by age. With no retention configured, artifacts stay until `max_size` evicts them. `sweep_interval` must be at least `1m` .
2026-10-12 14:21:38 +02:00
A package rule wins over an ecosystem rule, which wins over `default` . Package rules are keyed by package PURL without a version, written in one of the forms the table below lists for the ecosystem. The ecosystems and package rules can only name ecosystems that support retention in the running version; anything else, including a typo, stops the proxy at startup with an error naming the setting. `default` applies only to supported ecosystems. Support is added one ecosystem at a time, so a later release can make an existing `default` cover more ecosystems. On startup the proxy logs which ecosystems retention covers (`covered` ) and which ones `default` does not reach yet (`default_not_applied` ).
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| Ecosystem key | Retention supported | Package rule form |
|---------------|---------------------|-------------------|
2026-10-11 01:07:07 +02:00
| `alpine` | yes | `pkg:apk/curl` or `pkg:apk/alpine/curl` ; case doesn't matter. A rule covers the package in every configured APK repository. Alpine mirrors keep only the current build of each package per release, so an evicted older one can't be fetched again. |
2026-10-10 22:10:31 +02:00
| `cargo` | yes | `pkg:cargo/serde` , the crate name as published; case, `-` and `_` have to match, so `pkg:cargo/inflector` doesn't cover `Inflector` . |
2026-10-11 00:25:33 +02:00
| `composer` | yes | `pkg:composer/symfony/console` ; case doesn't matter. A key without a vendor is refused. |
2026-10-11 18:01:15 +02:00
| `conan` | yes | `pkg:conan/zlib` , the recipe name; a rule covers every version, revision and user/channel of it. |
2026-10-10 23:37:32 +02:00
| `conda` | yes | `pkg:conda/conda-forge/numpy` , with the channel as the namespace; the package name's case doesn't matter. A key without a channel is refused. |
2026-10-11 01:50:16 +02:00
| `cran` | yes | `pkg:cran/ggplot2` , the package name as published, case included (`pkg:cran/Rcpp` ). A rule covers source and binary builds alike. |
2026-10-11 01:12:41 +02:00
| `deb` | yes | `pkg:deb/curl` ; case doesn't matter. A vendor namespace such as `pkg:deb/debian/curl` is accepted and dropped, so a rule covers the package from every configured archive. Debian and Ubuntu archives only keep the current version per suite, so an evicted older one can't be fetched again. |
2026-10-12 19:55:10 +02:00
| `gem` | yes | `pkg:gem/rails` , the gem name as published, case included. |
2026-10-12 14:04:16 +02:00
| `golang` | yes | `pkg:golang/github.com/stretchr/testify` , the module path as in `go.mod` ; case doesn't matter. A major version like `/v2` (or `.v3` on gopkg.in) is a module of its own and needs its own rule. The `!` -escaped form from proxy URLs and the module cache (`github.com/!azure/...` ) is refused. |
2026-10-11 01:55:16 +02:00
| `helm` | yes | No package rules: charts are cached per repository and digest, not under the chart name. The `helm` rule and `default` cover charts from HTTP repositories; charts pulled over OCI fall under `oci` . |
2026-10-12 19:57:57 +02:00
| `hex` | yes | `pkg:hex/phoenix` , the package name as published. |
2026-10-11 01:38:14 +02:00
| `julia` | yes | No package rules: a julia PURL needs a `uuid` qualifier, which the cache doesn't store. The `julia` rule and `default` cover package tarballs, artifacts and registry snapshots alike. |
2026-10-12 13:58:09 +02:00
| `maven` | yes | `pkg:maven/com.google.guava/guava` , group and artifact as published, case included. Gradle build cache entries are not covered; they use `gradle.build_cache.max_age` . |
2026-10-12 14:21:38 +02:00
| `npm` | yes | `pkg:npm/lodash` , `pkg:npm/@babel/core` ; the name has to match exactly, case included, so `pkg:npm/jsonstream` doesn't cover `JSONStream` . |
2026-10-12 19:56:38 +02:00
| `nuget` | yes | `pkg:nuget/Newtonsoft.Json` ; case doesn't matter. |
2026-10-12 14:00:57 +02:00
| `oci` | yes | `pkg:oci/library/nginx` , the repository name of the pull. A named upstream keeps its prefix, also when containerd reaches it through `ns` : `ghcr.io/owner/app` pulled that way is `pkg:oci/upstream/ghcr/owner/app` . Homebrew bottles use Homebrew's image name, with `@` written as `/` and `+` as `x` (`openssl@3` is `pkg:oci/homebrew/core/openssl/3` ). Helm charts pulled over OCI are stored like images. Only blobs are evicted; manifests stay in the metadata cache. |
2026-10-10 23:15:44 +02:00
| `pub` | yes | `pkg:pub/http` , the package name as published. |
2026-10-12 14:06:37 +02:00
| `pypi` | yes | `pkg:pypi/requests` ; case and single `_` /`-` don't matter. Dots and repeated separators are kept as written, so a project like `zope.interface` can be stored as `zope.interface` and `zope-interface` ; add a rule for each. |
2026-10-11 01:21:03 +02:00
| `rpm` | yes | `pkg:rpm/curl` , the name as packaged, case included (`pkg:rpm/NetworkManager` ). A vendor namespace such as `pkg:rpm/fedora/curl` is accepted and dropped. Fedora and EPEL update repositories keep only the latest build of a package, so an evicted older one may not be fetchable again. |
2026-10-11 01:34:46 +02:00
| `swift` | yes | `pkg:swift/apple/swift-nio` , with the registry scope as the namespace, or the stored form `pkg:generic/swift-registry/apple.swift-nio` ; case doesn't matter. A source repository key like `pkg:swift/github.com/apple/swift-nio` is refused, since it doesn't name a registry identity. |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
An evicted artifact is fetched again from upstream on the next request. Packages preloaded with `proxy mirror` age from the time they were mirrored, so an unused mirrored package is evicted like any other. Running `proxy mirror` again over a package that is still cached counts as a download and restarts its clock, so packages a scheduled mirror job keeps listing never age out. A registry that removes old releases (Linux distribution archives in particular) cannot serve an evicted version again, so set those ecosystems' rules with that in mind.
Expired artifacts are not deleted right away. Their records are cleared and the files wait in the same queue a replaced artifact goes through, for at least an hour (or `storage.direct_serve_ttl` if longer), so downloads already in progress can finish. `storage.max_size` counts the space as free from the moment the record is cleared. Once that wait is over, each minute the queue is worked through for up to 30 seconds, pausing until the next minute after a delete fails. One sweep clears at most 10,000 artifacts, so the first sweep over a large, old cache spreads its evictions, and the downloads that bring popular artifacts back, over several intervals.
With several proxies sharing one database and `database.hit_flush_interval` set, a proxy's sweep cannot see downloads another proxy has not written yet. Keep retention durations well above the flush interval.
2026-01-29 16:33:09 +00:00
### Amazon S3
```yaml
storage:
url: "s3://my-bucket"
```
Configure credentials via environment variables:
```bash
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_REGION=us-east-1
```
### S3-Compatible (MinIO, etc.)
```yaml
storage:
url: "s3://my-bucket?endpoint=http://localhost:9000&disableSSL=true&s3ForcePathStyle=true"
```
## Database
The proxy supports SQLite (default) and PostgreSQL for storing package metadata.
### SQLite
```yaml
database:
driver: "sqlite"
path: "./cache/proxy.db"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.driver` | `PROXY_DATABASE_DRIVER` | `-database-driver` | `sqlite` or `postgres` |
| `database.path` | `PROXY_DATABASE_PATH` | `-database-path` | SQLite file path |
### PostgreSQL
```yaml
database:
driver: "postgres"
url: "postgres://user:password@localhost:5432/proxy ?sslmode=disable"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.url` | `PROXY_DATABASE_URL` | `-database-url` | PostgreSQL connection URL |
2026-09-30 23:09:04 -07:00
### Hit counts
Every cache hit updates the artifact's hit count and last-access time, which the stats pages and LRU eviction use. Rather than writing each hit as its own transaction, the proxy counts hits in memory and writes them together every `hit_flush_interval` . SQLite allows one writer at a time, and the proxy uses a single SQLite connection, so with a write per hit, concurrent downloads queue behind each other.
```yaml
database:
hit_flush_interval: "1s"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.hit_flush_interval` | `PROXY_DATABASE_HIT_FLUSH_INTERVAL` | - | How often batched hits are written (default `1s` ). `0` writes each hit as it happens. |
Hits not yet written are lost if the process is killed; a normal shutdown writes them.
2026-01-29 16:33:09 +00:00
## Logging
```yaml
log:
level: "info"
format: "text"
```
| Config | Environment | Flag | Values |
|--------|-------------|------|--------|
| `log.level` | `PROXY_LOG_LEVEL` | `-log-level` | `debug` , `info` , `warn` , `error` |
| `log.format` | `PROXY_LOG_FORMAT` | `-log-format` | `text` , `json` |
2026-08-16 18:07:39 +01:00
## Access Log
The optional access log records client requests and each HTTP exchange with an upstream registry. It is always written as JSONL, with one JSON object per line. Records for the same client request share a `request_id` .
```yaml
access_log:
path: "/var/log/proxy/access.jsonl"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `access_log.path` | `PROXY_ACCESS_LOG_PATH` | `-access-log` | File to append JSONL records to; empty disables the log |
2026-10-01 14:32:21 +02:00
Each client request record carries `remote_addr` (the TCP peer, host and port), `remote_ip` (the address the request is attributed to), `user_agent` , `client` (the tool name derived from the User-Agent), `ecosystem` , and `bytes` (the response body size written to the client). The structured log carries the same `client` , `remote` and `remote_ip` fields.
`remote_ip` follows the top-level `trust_forwarded_for` setting. Enable that only when the proxy sits behind a load balancer or ingress that sets the header: any client can send `X-Forwarded-For` , so behind such a hop it is the only way to see the real caller, but in front of one it lets a caller choose what address it is logged as. It can also choose a fresh one per request, which fills the bounded source table on `/ui/analytics` and evicts the genuine callers from it. `remote_addr` is unaffected and always records the TCP peer.
2026-08-16 18:07:39 +01:00
The parent directory must exist and be writable when the proxy starts. A newly created log file is readable and writable only by the proxy process owner.
A request that receives a rate limit response from an upstream can produce records like these:
```json
{"time":"2026-08-16T12:00:00Z","event":"upstream","request_id":"host/example-000001","method":"GET","url":"https://registry.example/packages/example","status_code":429,"duration_ms":42}
{"time":"2026-08-16T12:00:00Z","event":"request","request_id":"host/example-000001","method":"GET","path":"/npm/example","status_code":502,"duration_ms":43,"remote_addr":"192.0.2.10:41234"}
```
Upstream retries and OCI authentication calls are separate `upstream` records, so the log preserves every status returned over the wire. Network failures have an `error` field and no `status_code` . URL credentials, query strings, and fragments are omitted from both upstream URLs and client paths.
2026-01-29 16:33:09 +00:00
## Upstream Registries
2026-08-29 11:19:54 +01:00
Each upstream used by a built-in package route can be set in YAML or JSON under `upstream` , or with its matching environment variable. Existing installations keep the same public upstreams by default. Trailing slashes are ignored.
| Config | Environment | Default |
|--------|-------------|---------|
| `upstream.allow_private_hosts` | `PROXY_UPSTREAM_ALLOW_PRIVATE_HOSTS` | `[]` |
| `upstream.allow_loopback` | `PROXY_UPSTREAM_ALLOW_LOOPBACK` | `false` |
| `upstream.npm` | `PROXY_UPSTREAM_NPM` | `https://registry.npmjs.org` |
2026-09-02 07:13:12 -04:00
| `upstream.npm_full_metadata` | `PROXY_UPSTREAM_NPM_FULL_METADATA` | `false` |
2026-08-29 11:19:54 +01:00
| `upstream.cargo` | `PROXY_UPSTREAM_CARGO` | `https://index.crates.io` |
| `upstream.cargo_download` | `PROXY_UPSTREAM_CARGO_DOWNLOAD` | `https://static.crates.io/crates` |
| `upstream.gem` | `PROXY_UPSTREAM_GEM` | `https://rubygems.org` |
| `upstream.go` | `PROXY_UPSTREAM_GO` | `https://proxy.golang.org` |
2026-10-12 22:35:23 +02:00
| `upstream.go_sumdbs_offline` | `PROXY_UPSTREAM_GO_SUMDBS_OFFLINE` | `false` |
2026-08-29 11:19:54 +01:00
| `upstream.hex` | `PROXY_UPSTREAM_HEX` | `https://repo.hex.pm` |
| `upstream.hex_api` | `PROXY_UPSTREAM_HEX_API` | `https://hex.pm` |
| `upstream.pub` | `PROXY_UPSTREAM_PUB` | `https://pub.dev` |
| `upstream.pypi` | `PROXY_UPSTREAM_PYPI` | `https://pypi.org` |
| `upstream.pypi_download` | `PROXY_UPSTREAM_PYPI_DOWNLOAD` | `https://files.pythonhosted.org` |
| `upstream.maven` | `PROXY_UPSTREAM_MAVEN` | `https://repo1.maven.org/maven2` |
| `upstream.gradle_plugin_portal` | `PROXY_UPSTREAM_GRADLE_PLUGIN_PORTAL` | `https://plugins.gradle.org/m2` |
| `upstream.nuget` | `PROXY_UPSTREAM_NUGET` | `https://api.nuget.org` |
| `upstream.nuget_search` | `PROXY_UPSTREAM_NUGET_SEARCH` | `https://azuresearch-usnc.nuget.org` |
| `upstream.composer` | `PROXY_UPSTREAM_COMPOSER` | `https://packagist.org` |
| `upstream.composer_repository` | `PROXY_UPSTREAM_COMPOSER_REPOSITORY` | `https://repo.packagist.org` |
| `upstream.conan` | `PROXY_UPSTREAM_CONAN` | `https://center.conan.io` |
| `upstream.conda` | `PROXY_UPSTREAM_CONDA` | `https://conda.anaconda.org` |
| `upstream.cran` | `PROXY_UPSTREAM_CRAN` | `https://cloud.r-project.org` |
| `upstream.julia` | `PROXY_UPSTREAM_JULIA` | `https://pkg.julialang.org` |
2026-09-02 13:06:34 +01:00
| `upstream.swift` | `PROXY_UPSTREAM_SWIFT` | `https://tuist.dev/api/registry/swift` |
2026-08-29 11:19:54 +01:00
| `upstream.oci_default` | `PROXY_UPSTREAM_OCI_DEFAULT` | `https://registry-1.docker.io` |
| `upstream.debian` | `PROXY_UPSTREAM_DEBIAN` | `http://deb.debian.org/debian` |
| `upstream.rpm` | `PROXY_UPSTREAM_RPM` | `https://dl.fedoraproject.org/pub/fedora/linux` |
2026-09-03 16:59:12 +01:00
| `upstream.homebrew_api` | `PROXY_UPSTREAM_HOMEBREW_API` | `https://formulae.brew.sh/api` |
| `upstream.homebrew_artifact` | `PROXY_UPSTREAM_HOMEBREW_ARTIFACT` | `https://ghcr.io` |
2026-08-29 11:19:54 +01:00
Private, ULA, CGNAT, and loopback addresses are rejected by default. Add each private upstream hostname or IP address to `upstream.allow_private_hosts` . The matching environment variable accepts a comma-separated list. Loopback upstreams also require `upstream.allow_loopback: true` . That setting permits upstream requests and redirects to reach any loopback address.
2026-01-29 16:33:09 +00:00
```yaml
upstream:
2026-08-29 11:19:54 +01:00
allow_private_hosts:
- "upstream-proxy.internal"
pypi: "http://upstream-proxy.internal/pypi"
pypi_download: "http://upstream-proxy.internal/pypi"
```
For protocols that use separate metadata and download services, configure both values. They may point to the same endpoint when chaining proxies:
```yaml
upstream:
pypi: "https://upstream-proxy.example.com/pypi"
pypi_download: "https://upstream-proxy.example.com/pypi"
nuget: "https://upstream-proxy.example.com/nuget"
nuget_search: "https://upstream-proxy.example.com/nuget"
composer: "https://upstream-proxy.example.com/composer"
composer_repository: "https://upstream-proxy.example.com/composer"
```
`upstream.hex_api` is used for cooldown timestamps and must expose Hex's `/api/packages/{name}` JSON endpoint.
Add package routes for private Composer and npm registries (#426)
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
2026-10-11 19:06:18 +02:00
### Private package routes
Clients that use the proxy as their only registry can still install private packages. A package route sends every package whose name matches a pattern to a private registry, and never to the default upstream:
```yaml
upstream:
composer_routes:
"example/*": "https://packages.example.com/composer"
npm_routes:
"@example/ *": "https://packages.example.com/npm"
allow_private_hosts:
- "packages.example.com"
auth:
"https://packages.example.com":
type: bearer
token: "${PRIVATE_REGISTRY_TOKEN}"
```
- Patterns use Go's [`path.Match` ](https://pkg.go.dev/path#Match ) syntax and are matched case-insensitively. `*` does not cross `/` , so `example/*` matches `example/library` but not `example-fork/library` . The longest matching pattern wins.
- A Composer route URL is the repository base that serves `/p2/{vendor}/{package}.json` . A route also covers the package's dev versions in `/p2/{vendor}/{package}~dev.json` ; patterns are matched against the package name without `~dev` .
- An npm route URL is the registry base that serves `/{package}` . Tarball URLs in its packuments must be on the same host and below the same path.
- A routed package is not looked up anywhere else. If the private registry does not have it, the client gets a 404. If the registry fails or rejects the credentials, the client gets a 502. A package with the same name on the public registry is never served in its place (dependency confusion).
- Metadata and artifacts of routed packages are cached under keys that include the route's URL. Entries cached from the public registry before a route was added are never served for a routed package.
- With cooldown enabled, the publish time of a routed npm version always comes from the route's metadata. Times recorded for the public package of the same name, or for a route's previous URL, are not used, and routed downloads do not record times of their own.
- Composer metadata of routed packages carries an empty `notification-url` , so Composer does not report their installs to the default upstream's `notify-batch` endpoint.
- Use `upstream.auth` for the registry's credentials. The proxy has no authentication of its own, so every client that can reach it can download routed packages.
- Search (`/composer/search.json` ) and the package list (`/composer/packages/list.json` ) still come from the default upstream only.
- npm audit and signing-key requests still go to `upstream.npm` . An audit request lists every package in the dependency tree, routed ones included.
- Routes can only be set in the configuration file, not with environment variables.
2026-08-29 11:19:54 +01:00
Helm HTTP repositories and additional OCI registries are configured as named maps:
2026-08-16 22:42:55 +05:30
2026-08-29 11:19:54 +01:00
```yaml
upstream:
2026-08-16 22:42:55 +05:30
# Named HTTP Helm chart repositories, served at /helm/{name}/.
helm:
bitnami: "https://charts.bitnami.com/bitnami"
# Named OCI registries. Select one with the repository prefix
# upstream/{name}/, e.g. oci://proxy.example.com/upstream/ghcr/owner/chart.
oci:
ghcr: "https://ghcr.io"
2026-01-29 16:33:09 +00:00
```
2026-08-16 22:42:55 +05:30
Helm HTTP repositories are read-only. The proxy fetches and rewrites each
repository's `index.yaml` so chart archives are downloaded through the proxy.
Chart archives are retained only when their SHA-256 digest matches the digest
listed in the index. Relative and absolute chart URLs are both supported.
2026-09-19 14:39:29 +05:30
HTTP indexes can also contain `oci://` chart references. References matching the
host and port of a configured `upstream.oci` registry or `upstream.oci_default`
are rewritten through the OCI proxy, with named registries preferred. Unmatched
references are left unchanged, so Helm contacts those registries directly.
Automatic rewriting requires both the OCI upstream URL and `base_url` to be
root URLs without a path prefix. Repository paths, tags, and digests are preserved.
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
Generic HTTP upstreams proxy plain downloads from fixed base URLs:
```yaml
upstream:
# Named HTTP upstreams, served at /generic/{name}/. The rest of the
# request path and the query string are appended to the upstream URL.
generic:
github: "https://github.com"
github-api: "https://api.github.com"
auth:
# Optional: raise the GitHub API rate limit. Scoped to this host only,
# so the token is never sent to the object store GitHub redirects to.
"https://api.github.com":
type: bearer
token: "${GITHUB_TOKEN}"
```
Only configured upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like `{owner}/{repo}/releases/download/{tag}/{asset}` are
version-pinned GitHub release assets: they are stored in the artifact cache
and served from it without revalidation, including while the upstream is
2026-10-12 09:12:16 +01:00
down.
2026-10-12 15:39:23 +01:00
Upstreams pointed at these hosts get version-pinned paths built in, whatever
the upstream is named:
| Upstream URL | Cached as artifacts | Package name |
|---|---|---|
| `https://archive.apache.org` | `dist/{project dirs}/{version}/[binaries/]{file}` , where `{version}` starts with a digit | the project dirs, e.g. `maven/maven-3` , `maven/mvnd` |
| `https://nodejs.org` | `dist/v{version}/{file}` | the upstream name |
| `https://dl.google.com/go` | `go{version}.{os}-{arch}.tar.gz` , `.zip` and their `.sha256` | the upstream name |
2026-10-12 09:12:16 +01:00
Other upstreams can declare their own version-pinned paths in
2026-10-12 19:22:42 +01:00
`generic_artifacts` . These are tried before any built-in ones, so a configured
pattern that overlaps a built-in decides the package name. Each pattern is a regular expression that must match the
2026-10-12 09:12:16 +01:00
whole path after `/generic/{name}/` , with `version` and `file` named groups
(single path segments) and an optional `name` group for the package name,
which otherwise is the upstream name. Matching downloads are cached like
release assets and listed under the `generic` ecosystem:
```yaml
upstream:
generic:
2026-10-12 15:39:23 +01:00
eclipse: "https://download.eclipse.org"
2026-10-12 09:12:16 +01:00
generic_artifacts:
2026-10-12 15:39:23 +01:00
eclipse:
- 'jdtls/milestones/(?P<version>[^/]+)/(?P<file>[^/]+)'
2026-10-12 09:12:16 +01:00
```
Only list paths whose content never changes for a given version: a match is
never revalidated.
Every other path is served through the metadata cache (`cache_metadata`
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
must be enabled for offline fallback): fresh within `metadata_ttl` , then
revalidated with the upstream's `ETag` /`Last-Modified` , and served stale with
a `Warning: 110` header when the upstream fails, refuses or rate-limits the
request. Metadata responses are buffered up to `metadata_max_size` , so keep
large mutable downloads (`releases/latest/download/...` ) off this route.
This is the cache behind [mise ](https://mise.jdx.dev )'s aqua backend; see the
mise section in the README for the client-side `url_replacements` .
2026-08-29 11:19:54 +01:00
`upstream.oci_default` sets the registry used by unprefixed `/v2` requests,
while `upstream.oci` selects named registries through the `upstream/{name}/`
repository prefix. For example, `oci://proxy.example.com/upstream/ghcr/owner/chart`
uses the `ghcr` registry with `owner/chart` as its repository.
Route containerd ns mirror requests to configured registries (#405)
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
2026-10-10 08:35:50 +02:00
containerd mirror requests carry the original registry host in an `ns` query
parameter (see the containerd section in the README). The proxy only looks the
host up and never connects to it: `docker.io` , `index.docker.io` ,
`registry-1.docker.io` and the host of `upstream.oci_default` select the default
registry, and the host of each `upstream.oci` URL selects that named registry.
Hosts are compared case-insensitively. The scheme's default port (443 for
`https` , 80 for `http` ) may be spelled out or left out in the image reference;
any other port must match exactly, so `https://registry.example:80` and
`https://registry.example` are two different registries. An
unknown host returns `404 NAME_UNKNOWN` , so containerd falls back to its next
host. Registry URLs with a path (for example an Artifactory repository path)
are not reachable through their own host in `ns` , only through
`upstream/{name}/` or through `upstream.oci_mirrors` . When two
entries share a host, the proxy logs a warning at startup; the default registry
wins, otherwise the alphabetically first name. Pulls through `ns` ,
`upstream/{name}/` and unprefixed requests share the same cache entries.
Requests that combine the `upstream/{name}/` prefix with `ns` , as per-registry
containerd mirrors with `override_path = true` send them, are routed by the
prefix. They are accepted only when `ns` is the upstream's own host, a host
listed for it in `upstream.oci_mirrors` , or Docker Hub (`docker.io` and its
aliases). Docker Hub repository names have two path components and can never
start with `upstream/` , so a Docker Hub mirror behind any prefix stays
reachable. Any other host gets `404 NAME_UNKNOWN` : a containerd `_default`
mirror sends the same request for an image such as
`unconfigured.example/upstream/ghcr/owner/app` , and that pull must not be
answered from the `ghcr` upstream.
When a named upstream mirrors another registry, for example an Artifactory
remote of `ghcr.io` , say so in `upstream.oci_mirrors` :
```yaml
upstream:
oci:
ghcr: "https://artifactory.example.com/artifactory/api/docker/ghcr-remote"
oci_mirrors:
ghcr: ["ghcr.io"]
```
Each entry lists bare registry hosts as containerd sends them, optionally with
a port. A host can be listed for one upstream only. Those hosts select the
upstream for unprefixed `ns` requests and are accepted as `ns` on its
`upstream/{name}/` prefix. A host that is also the host of a configured
registry URL stays with that registry for unprefixed requests; the proxy logs a
warning at startup.
A host that is accepted on a prefix route gives up image paths of the form
`<host>/upstream/{name}/...` under a `_default` mirror: with `ghcr: ["ghcr.io"]` ,
a pull of `ghcr.io/upstream/ghcr/owner/app` is answered with `owner/app` from
the `ghcr` upstream, because containerd sends the same request for it as a
per-registry mirror does. The same applies to the upstream's own host.
2026-08-16 22:42:55 +05:30
When the proxy uses plain HTTP (for example `localhost:8080` ), pass
`--plain-http` to Helm OCI commands.
Add Alpine APK repository proxy support (#293)
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
2026-09-02 13:36:00 +02:00
```yaml
upstream:
# Named Alpine APK repositories, served at /apk/{name}/.
apk:
alpine: "https://dl-cdn.alpinelinux.org/alpine"
```
Alpine APK repositories are read-only. Requests to `/apk/{name}/…` mirror the
upstream layout, e.g. `/apk/alpine/v3.22/main/x86_64/APKINDEX.tar.gz` . Indexes
(v2 `APKINDEX.tar.gz` , v3 `Packages.adb` ) and detached signatures are cached
with the metadata TTL and served byte-for-byte unchanged so apk signature
verification keeps working; `.apk` packages use the shared artifact cache.
When `upstream.apk` is empty, a single repository named `alpine` pointing at
the official mirror is available; configuring any entry replaces that default.
2026-10-11 23:41:16 +02:00
```yaml
upstream:
# Named Arch Linux pacman mirrors, served at /arch/{name}/.
arch:
archlinux: "https://geo.mirror.pkgbuild.com"
```
Arch Linux repositories are read-only. Requests to `/arch/{name}/…` are passed
to the mirror unchanged, so both the Arch Linux layout
(`/arch/archlinux/core/os/x86_64/core.db` ) and the Arch Linux ARM layout
(`/arch/archlinuxarm/aarch64/core/core.db` ) work. Databases (`{repo}.db` ,
`{repo}.files` and their `.tar.*` forms) are cached with the metadata TTL and
served byte-for-byte unchanged; a database's `.sig` is refreshed together with
the database. Packages and package signatures use the shared artifact cache.
When `upstream.arch` is empty, a single repository named `archlinux` pointing at
`https://geo.mirror.pkgbuild.com` is available; configuring any entry replaces
that default. Arch Linux ARM has no default: pick a regional HTTPS mirror, not
the `mirror.archlinuxarm.org` redirector.
2026-10-10 09:57:43 +02:00
### Go checksum database
```yaml
upstream:
# Named Go checksum databases, served at /go/sumdb/{name}/.
go_sumdbs:
sum.golang.org: "https://sum.golang.org"
Add upstream.go_sumdbs_offline for networks where only the proxy has egress
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
2026-10-10 17:55:04 +02:00
# Keep answering for a database from its cached tree, however old, when
# the database cannot be reached. For networks where clients have no
# route to the database of their own. Requires cache_metadata.
go_sumdbs_offline: false
2026-10-10 09:57:43 +02:00
```
The `go` command asks `GOPROXY` for the checksum database before contacting it
directly. The proxy answers for every name listed here and fetches lookups and
tiles from the database's URL, not from `upstream.go` . Unknown names get 404,
so the `go` command falls back to contacting that database itself. When
`upstream.go_sumdbs` is empty, `sum.golang.org` is served; configuring any
entry replaces that default.
2026-10-10 10:18:40 +02:00
Once the proxy says it serves a database, the `go` command no longer contacts
that database directly. The proxy therefore only says so when it can fetch the
Don't let the supported check hang, and don't take a block page for an answer
Two problems with the check behind supported. The fetch it waits on is
shared with other callers and only bounded by the upstream timeout, which
defaults to 30 seconds. A firewall that silently drops packets to
sum.golang.org would stall the first go command every minute for that long,
and the go command itself has no timeout on this request. Now we wait five
seconds at most; the fetch carries on in the background, records how it
went and fills the cache for the next check. A timeout only counts as a
failure if the fetch hasn't succeeded in the meantime.
The fetch reports no error when it falls back to the cached tree after the
database failed, so the metadata result now says when it did, and the
check treats that as the database not answering. A fresh cached tree
(within metadata_ttl) still lets a timed-out check say yes, since the
fetch only falls back once upstream has failed, which a silent database
delays past our timeout. An older tree does not count: once the proxy has
lost the database, clients go there directly, as they would without the
proxy.
And the check believed any 200. Egress proxies tend to answer blocked hosts
with a 200 HTML page, which we would have cached as the signed tree and taken
as proof that the database is reachable. latest now has to start with
"go.sum database tree", or it is refused like a bad tile.
Lookups had the same gap. A block page cached as a lookup would also
replace the good copy that is served during an outage, so a lookup now has
to be a record followed by a signed tree, the way the go command reads it.
2026-10-10 11:31:01 +02:00
database's latest signed tree, or has a fresh one cached (fetched within
`metadata_ttl` ). If it cannot, it answers 404 as for an unknown name, and
clients go to the database themselves, as they would without the proxy. The
check waits five seconds at most, and after a failed check the proxy waits a
minute before it tries the database again.
2026-10-10 10:18:40 +02:00
Add upstream.go_sumdbs_offline for networks where only the proxy has egress
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
2026-10-10 17:55:04 +02:00
Set `upstream.go_sumdbs_offline: true` when clients have no route to the
database of their own. The proxy then keeps answering for it from the cached
tree for as long as the outage lasts and serves cached lookups and tiles.
That does not mean every cached module still verifies: the `go` command
checks each lookup against tiles of the signed tree that came with it, the
database attaches a different tree to almost every lookup, and a tile no
client fetched through the proxy before cannot be served during the outage.
Lookups for modules that were never cached fail as well. Builds whose
`go.sum` already lists every module do not ask the database at all and are
not affected. This requires `cache_metadata` .
2026-10-13 16:23:12 +02:00
With `cache_metadata` enabled, lookups, partial hash tiles and the latest
signed tree follow `metadata_ttl` and are served from cache when the database
is unreachable. Complete hash tiles never change and are not revalidated once
cached; a tile of the wrong size is refused and not stored. Data tiles are
always forwarded and never stored: the `go` command does not fetch them, only
mirrors and auditors do.
2026-10-10 10:19:46 +02:00
Without `cache_metadata` the requests are forwarded and nothing is stored.
A 404 from the database keeps its text, which the `go` command prints to
explain why a version is missing. With `cache_metadata` enabled the text is
cut at 4 KiB; without it the response is passed on unchanged.
2026-10-10 09:57:43 +02:00
If `upstream.go` points at a module proxy that serves the checksum database
itself, such as Athens, point the entry there instead, for example
`sum.golang.org: "https://athens.example/sumdb/sum.golang.org"` .
feat(debian): serve additional APT archives at /debian/{name}/
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
2026-09-14 15:41:13 -07:00
### Debian archives
```yaml
upstream:
# The main APT archive, served at /debian/.
debian: "http://deb.debian.org/debian"
# Additional APT archives, served at /debian/{name}/.
debian_repositories:
security: "https://security.debian.org/debian-security"
```
A Debian release is served by more than one archive: security updates live on
a separate host from the main archive, so `upstream.debian` alone cannot serve
a complete suite set. Requests to `/debian/{name}/…` mirror the upstream
layout, e.g. `/debian/security/dists/trixie-security/InRelease` .
`upstream.debian_repositories` is additive. `/debian/pool/…` and
`/debian/dists/…` continue to address `upstream.debian` with unchanged cache
identities, so existing deployments and their warm caches are unaffected. A
repository name shadows the main archive's root path of the same name; `pool`
and `dists` are refused at config load for that reason, and other root paths
an archive may serve (`indices` , `project` , `doc` , `tools` ) are not, so avoid
those names unless the shadowing is intended.
This field has no environment variable, as with the other named upstream maps.
`PROXY_UPSTREAM_DEBIAN` still sets `upstream.debian` .
2026-01-29 16:33:09 +00:00
## Authentication
2026-08-13 07:08:23 +01:00
Configure authentication for private upstream registries. The same authentication-aware client is used for metadata and artifact downloads, and credentials can reference environment variables using `${VAR_NAME}` syntax.
OCI registries that return a Bearer challenge from a `/v2/{repository}/…` endpoint are handled automatically. The proxy discovers the token realm from `WWW-Authenticate` , applies any configured credentials for the token URL, and reuses the scoped token until shortly before it expires.
2026-01-29 16:33:09 +00:00
2026-10-14 17:56:18 +02:00
A token fetched with credentials configured for the token URL is only fetched for, and sent to, a registry that is trusted with it: one with configured credentials of its own, or one on the same site (registrable domain) as the token service, as `registry-1.docker.io` is with `auth.docker.io` . Any other host that answers with a challenge naming that token service gets its 401 back, so it cannot obtain a token minted from your credentials. Tokens from a token service with no configured credentials are anonymous and are fetched for any registry, as before. If your registry and its token service are on different sites, configure credentials for the registry URL as well.
2026-01-29 16:33:09 +00:00
### Bearer Token
Used by npm, GitHub Package Registry, and many other registries:
```yaml
upstream:
auth:
"https://registry.npmjs.org":
type: bearer
token: "${NPM_TOKEN}"
"https://npm.pkg.github.com":
type: bearer
token: "${GITHUB_TOKEN}"
```
### Basic Authentication
Used by PyPI, Artifactory, and others:
```yaml
upstream:
auth:
"https://pypi.org":
type: basic
username: "__token__"
password: "${PYPI_TOKEN}"
"https://artifactory.mycompany.com":
type: basic
username: "deploy"
password: "${ARTIFACTORY_PASSWORD}"
```
### Custom Header
For registries that use non-standard authentication headers:
```yaml
upstream:
auth:
"https://maven.mycompany.com":
type: header
header_name: "X-Auth-Token"
header_value: "${MAVEN_TOKEN}"
```
2026-09-02 12:43:43 +01:00
### AWS ECR
Private ECR registries issue authorization tokens that expire after 12 hours. The `ecr` auth type calls `ecr:GetAuthorizationToken` on demand, caches the result, and refreshes it shortly before expiry, so no static credential appears in the config file:
```yaml
upstream:
oci:
ecr: "https://123456789012.dkr.ecr.eu-west-1.amazonaws.com"
auth:
"https://123456789012.dkr.ecr.eu-west-1.amazonaws.com":
type: ecr
```
AWS credentials are resolved by the SDK's default chain, which covers EKS IAM Roles for Service Accounts (IRSA), EC2/ECS instance profiles, `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables, and `~/.aws/credentials` . The IAM identity needs the `ecr:GetAuthorizationToken` action plus the usual `ecr:BatchGetImage` / `ecr:GetDownloadUrlForLayer` permissions on the target repositories. The region is inferred from private ECR IPv4, dual-stack, and FIPS hostnames. For other endpoint formats, set `region` explicitly or configure a default region for the SDK.
2026-01-29 16:33:09 +00:00
### URL Matching
2026-08-13 07:08:23 +01:00
Auth keys must be absolute URLs. Matching compares the scheme, host, effective port, and path-segment prefix, preventing credentials for `registry.example.com` from being sent to a lookalike host such as `registry.example.com.evil.test` . The longest matching scope wins, so you can configure different credentials for different paths:
2026-01-29 16:33:09 +00:00
```yaml
upstream:
auth:
# All requests to this registry
"https://registry.mycompany.com":
type: bearer
token: "${REGISTRY_TOKEN}"
# Override for a specific scope
"https://registry.mycompany.com/@private ":
type: bearer
token: "${PRIVATE_TOKEN}"
```
2026-05-04 12:15:16 +02:00
## Gradle Build Cache
The `/gradle` endpoint supports optional safeguards for upload control and cache retention.
```yaml
gradle:
build_cache:
read_only: false
max_upload_size: "100MB"
max_age: "168h"
max_size: "20GB"
sweep_interval: "10m"
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `gradle.build_cache.read_only` | `PROXY_GRADLE_BUILD_CACHE_READ_ONLY` | Disable PUT uploads and keep GET/HEAD read-only |
| `gradle.build_cache.max_upload_size` | `PROXY_GRADLE_BUILD_CACHE_MAX_UPLOAD_SIZE` | Maximum accepted PUT body size (must be > 0) |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| `gradle.build_cache.max_age` | `PROXY_GRADLE_BUILD_CACHE_MAX_AGE` | Delete entries older than this duration (default `168h` , also accepts days such as `7d` ; set `0` to disable) |
2026-05-04 12:15:16 +02:00
| `gradle.build_cache.max_size` | `PROXY_GRADLE_BUILD_CACHE_MAX_SIZE` | Total size cap for `_gradle/http-build-cache` , deleting oldest first (`0` disables) |
| `gradle.build_cache.sweep_interval` | `PROXY_GRADLE_BUILD_CACHE_SWEEP_INTERVAL` | Frequency for background eviction sweeps |
`max_age` and `max_size` are independent and can be combined. When both are set, age-based eviction runs first, then size-based eviction trims remaining entries oldest-first.
2026-03-04 19:00:31 +00:00
## Cooldown
The cooldown feature hides package versions published too recently, giving the community time to spot malicious releases before they reach your projects. When a version is within its cooldown period, it's stripped from metadata responses so package managers won't install it.
```yaml
cooldown:
default: "3d"
ecosystems:
npm: "7d"
cargo: "0"
packages:
"pkg:npm/lodash": "0"
"pkg:npm/@babel/core ": "14d"
2026-10-04 20:48:41 +01:00
package_patterns:
"pkg:npm/@example/ *": "0"
2026-03-04 19:00:31 +00:00
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `cooldown.default` | `PROXY_COOLDOWN_DEFAULT` | Global default cooldown |
| `cooldown.ecosystems` | - | Per-ecosystem overrides |
| `cooldown.packages` | - | Per-package overrides (keyed by PURL) |
2026-10-04 20:48:41 +01:00
| `cooldown.package_patterns` | - | Per-package glob overrides (keyed by PURL glob) |
2026-03-04 19:00:31 +00:00
Durations support days (`7d` ), hours (`48h` ), and minutes (`30m` ). Set to `0` to disable.
2026-07-17 18:11:47 -07:00
Package PURL keys are normalized to canonical form before matching, so `pkg:npm/@babel/core` and `pkg:npm/%40babel/core` are equivalent, as are `pkg:pypi/Django` and `pkg:pypi/django` . If both forms configure the same package, the canonical entry wins.
2026-07-13 17:11:04 -07:00
2026-10-04 20:48:41 +01:00
`package_patterns` matches canonical, versionless PURLs using `*` for zero or more characters and `?` for one character. Neither wildcard crosses `/` separators. Character classes (`[...]` ) and backslash escapes are rejected at startup. For example, `"pkg:npm/@example/*"` matches packages under the `@example` npm scope. Patterns accept `@` as an alias for `%40` ; other characters must use their canonical PURL form. Equivalent patterns with different durations are rejected at startup. Equal durations, such as `1d` and `24h` , are accepted.
Exact `packages` entries take precedence over patterns. When several patterns match, longer patterns win after excluding `*` and `?` from the length. Ties use lexical order of the normalized patterns.
Resolution order: exact package override, then package pattern, then ecosystem override, then global default. This lets you set a conservative default while exempting trusted package families.
2026-03-04 19:00:31 +00:00
2026-04-06 12:55:15 +01:00
Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, and Hex. These ecosystems include publish timestamps in their metadata.
Note: Hex cooldown requires disabling registry signature verification since the proxy re-encodes the protobuf payload without the original signature. Set `HEX_NO_VERIFY_REPO_ORIGIN=1` or configure your repo with `no_verify: true` .
2026-03-04 19:00:31 +00:00
2026-09-20 14:42:30 +05:30
## Version denylist
Block exact package versions independently of cooldown and artifact scanning:
```yaml
denylist:
packages:
- "pkg:pypi/requests@2 .31.0"
- "pkg:cargo/some-crate@1 .2.3"
- "pkg:npm/%40scope/example@4 .5.6"
```
Entries must be versioned PURLs without qualifiers or subpaths. Package names
are canonicalized (including PyPI case/separator normalization and npm scopes);
versions match exactly, not by range or glob. Invalid entries prevent startup.
An empty list disables the policy. Restart the proxy after changing the list.
Denied versions are removed from npm packuments, Cargo sparse indexes, and PyPI
simple HTML/JSON and release metadata, including responses served from cached
metadata during an upstream outage. No publication timestamp or enabled cooldown
is required. npm tags pointing at denied versions are removed; `latest` is moved
to the highest remaining stable version when available. Other tags are not
retargeted. PyPI version-specific JSON endpoints return 404 for denied versions.
The shared artifact download pipeline returns 403 for denied package/version
identities before reading storage, issuing a signed redirect, or fetching bytes.
This also applies to mirror downloads and recognized Debian package downloads.
Existing cached artifacts are retained, so removing a denial restores access.
Already-issued storage URLs and clients' local caches cannot be revoked by this
configuration.
Metadata filtering currently covers npm, PyPI, and Cargo. Other ecosystems'
metadata is unchanged; in particular, APT indexes cannot be rewritten without
regenerating their checksums and signatures. Pass-through resources without a
resolved package/version identity are not covered. This is an explicit operator
policy, not automatic CVE detection or cache re-scanning. Exact pins or dependency
constraints may still make an install fail when no allowed version can satisfy it.
2026-09-22 12:47:37 +05:30
### APT version pinning
For Debian and Ubuntu clients, combine the proxy's download denylist with
client-side APT preferences. The proxy blocks recognized denied `.deb` downloads
with 403, but does not remove versions from APT indexes. Without a client-side
pin, APT can select a denied version and then fail when downloading it.
On each client, add a record to `/etc/apt/preferences.d/proxy-denylist.pref` :
```text
Package: somepkg
Pin: version 1.2.3-1
Pin-Priority: -1
```
Replace `somepkg` with the binary package name and `1.2.3-1` with its full APT
version, including the epoch if present. A negative priority excludes that
version from normal candidate selection. For multi-architecture clients, use
`Package: somepkg:any` to cover all architectures. Separate additional records
with blank lines. See [apt_preferences(5) ](https://manpages.debian.org/stable/apt/apt_preferences.5.en.html ).
Check the effective priorities and simulate installation before deploying the
preferences broadly:
```bash
apt-cache policy somepkg
apt-get --simulate install somepkg
```
Confirm that the denied version has priority `-1` and is not selected. Existing
specific pins can take precedence, so check for conflicts. APT can select
another candidate only if one is available and satisfies dependency constraints;
a Debian suite commonly offers only one version. Pinning does not remove an
already-installed package or automatically downgrade it.
These preferences are managed on clients, not distributed by the proxy. Keep
the corresponding proxy denylist entries as a separate download guard.
APT indexes are left unchanged to preserve upstream authentication: `Release`
metadata contains index checksums and is signed by the repository. Filtering
would require regenerating metadata and signing it with a key clients trust,
making the proxy a repository authority rather than a transparent cache. Do not
disable signature verification to work around this limitation. See
[apt-secure(8) ](https://manpages.debian.org/stable/apt/apt-secure.8.en.html ).
If centrally curated APT indexes are required, use a repository manager such as
[aptly ](https://www.aptly.info/ ) or [reprepro ](https://salsa.debian.org/debian/reprepro )
as the upstream behind the proxy. Filtered APT repository generation and signing
are outside the proxy's current scope.
2026-09-03 05:06:44 -04:00
## Artifact Scanning
Cooldown only ever looks at a version's * publish timestamp * — it never inspects the actual bytes of an artifact. Artifact scanning runs after a fetched artifact is staged into storage but before it becomes visible from cache, so an external scanner (trivy, ClamAV, Wiz, or any custom service) can block a bad verdict from ever reaching a client.
```yaml
scanning:
enabled: true
fail_open: false
timeout: 30s
signing_key: ${PROXY_SCANNING_SIGNING_KEY}
fetch_base_url: http://proxy.internal:8080
scanners:
- name: clamav
url: http://clamav-adapter:8080/scan
mode: block
- name: trivy
url: http://trivy-adapter:8081/scan
mode: monitor
ecosystems: [npm, pypi]
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `scanning.enabled` | `PROXY_SCANNING_ENABLED` | Turn on the scan gate. When false (default), artifacts are cached exactly as if scanning didn't exist |
| `scanning.fail_open` | `PROXY_SCANNING_FAIL_OPEN` | Treat scanner errors/timeouts as allow instead of block. Default is fail-closed |
| `scanning.timeout` | `PROXY_SCANNING_TIMEOUT` | Per-scan-call timeout, Go duration syntax (default `30s` ) |
| `scanning.signing_key` | `PROXY_SCANNING_SIGNING_KEY` | Signs pull requests to the internal scan-fetch route. Required whenever `enabled` is true |
| `scanning.fetch_base_url` | `PROXY_SCANNING_FETCH_BASE_URL` | Address scanners use to reach this proxy to pull staged artifacts. Defaults to `base_url` |
| `scanning.scanners` | - | List of external scanning services (YAML only) |
| `scanning.scanners[].name` | - | Identifies this scanner in logs and metrics |
| `scanning.scanners[].url` | - | Endpoint the proxy POSTs scan notifications to |
| `scanning.scanners[].mode` | - | `block` (default) or `monitor` |
| `scanning.scanners[].ecosystems` | - | Restricts this scanner to specific ecosystems (e.g. `npm` , `pypi` ). Empty means all ecosystems |
| `scanning.scanners[].headers` | - | Extra HTTP headers sent with every scan request (e.g. for authenticating to the scanner service). Values support `${VAR_NAME}` expansion |
### How caching defers to a scan verdict
The proxy never uploads artifact bytes to a scanner. When an artifact is fetched from upstream, it's stored to the configured storage backend first, exactly as without scanning. If scanning is enabled for the artifact's ecosystem, the proxy then notifies each applicable scanner with package metadata and a short-lived, HMAC-signed URL pointing at the internal `/_internal/scan-fetch` route; each scanner GETs that URL itself to pull the exact bytes staged in storage and runs its own scan against them.
Scanners configured for the same ecosystem all run concurrently, never sequentially. The moment any `block` -mode scanner reports a not-allowed verdict (or errors, unless `fail_open` is set), the proxy cancels the in-flight calls to the other scanners and deletes the staged artifact — it's never committed to the cache database, so it was never visible to a client. If nothing blocks, the proxy waits for every `block` -mode scanner to finish before caching the artifact and serving it. A `monitor` -mode scanner's findings are logged and never gate the wait or the caching decision, even when it reports not-allowed.
A blocked download surfaces to the client as `403 Forbidden` with the scanner's reason, across every ecosystem handler.
### Scanner HTTP contract
Any external service that implements this contract can act as a scanner — a trivy wrapper, a clamav-rest bridge, a Wiz connector, or an in-house service. The proxy POSTs a notify request to `scanning.scanners[].url` and waits for a JSON verdict.
**Request**
| Field | Type | Description |
|-------|------|-------------|
| `ecosystem` | string | e.g. `npm` , `pypi` , `cargo` |
| `name` | string | Package name |
| `version` | string | Package version |
| `filename` | string | Artifact filename |
| `purl` | string | Package URL (PURL) identifying this exact version |
| `content_type` | string | Artifact content type |
| `size` | integer | Artifact size in bytes |
| `fetch_url` | string | Short-lived signed URL; GET this to retrieve the exact staged bytes |
```json
{
"ecosystem": "npm", "name": "left-pad", "version": "1.0.0",
"filename": "left-pad-1.0.0.tgz", "purl": "pkg:npm/left-pad@1 .0.0",
"content_type": "application/octet-stream", "size": 1234,
"fetch_url": "https://proxy.internal/_internal/scan-fetch?path=...&exp=...&sig=..."
}
```
**Response**
| Field | Type | Description |
|-------|------|-------------|
| `allowed` | boolean | Whether the artifact may be cached and served |
| `reason` | string | Human-readable reason, surfaced to the client when `allowed` is false |
| `findings` | array | Optional list of `{"severity", "title", "description"}` objects |
```json
{
"allowed": false,
"reason": "malware detected",
"findings": [
{"severity": "critical", "title": "Trojan.GenericKD", "description": "..."}
]
}
```
The scanner must respond within `scanning.timeout` (default `30s` ); a timeout is treated the same as a `block` verdict unless `fail_open` is set.
### The `/_internal/scan-fetch` route
`fetch_url` points at an internal route, `/_internal/scan-fetch` , that streams a staged object straight from the proxy's storage backend via a short-lived HMAC-signed token (`path` , `exp` , `sig` query parameters). This works identically across every storage backend — local filesystem, S3, GCS, Azure — since it never depends on a backend-specific presigned URL, only on the one storage operation every backend already implements.
This route is not part of the public API. It's meant only for scanners to pull artifacts they've been notified about, and should be restricted to internal-network access at the ingress/network-policy layer — the HMAC scoping (one object, a short TTL) limits what a leaked token can do, but isn't a substitute for network restriction. Its query parameters are also documented in the generated [OpenAPI spec ](../README.md#openapi-swagger ).
The route only exists when scanning is actually configured: it's not mounted at all unless at least one scanner is enabled and `scanning.signing_key` is set, and it also refuses every request with `404` if either condition somehow isn't met at request time. There is no way to reach it, even with a forged token, when scanning is disabled.
2026-03-19 21:06:02 +00:00
## Metadata Caching
By default the proxy fetches metadata fresh from upstream on every request. Enable `cache_metadata` to store metadata responses in the database and storage backend for offline fallback. When upstream is unreachable, the proxy serves the last cached copy. ETag-based revalidation avoids re-downloading unchanged metadata.
2026-09-02 17:10:35 +05:30
OCI manifests and tag lists are always cached because cached image blobs cannot be pulled without their manifests and offline clients may need tag resolution. Digest-addressed manifests are immutable and served directly from cache. Tag-addressed manifests and tag lists follow `metadata_ttl` , revalidate when stale, and fall back to the last cached response when the registry is unavailable.
2026-08-13 07:35:07 +01:00
2026-03-19 21:06:02 +00:00
```yaml
cache_metadata: true
```
Or via environment variable: `PROXY_CACHE_METADATA=true` .
The `proxy mirror` command always enables metadata caching regardless of this setting.
2026-04-06 19:30:59 +01:00
### Metadata TTL
When metadata caching is enabled, `metadata_ttl` controls how long a cached response is considered fresh before revalidating with upstream. During the TTL window, cached metadata is served directly without contacting upstream, reducing latency and upstream load.
```yaml
metadata_ttl: "5m" # default
```
Or via environment variable: `PROXY_METADATA_TTL=10m` .
Set to `"0"` to always revalidate with upstream (ETag-based conditional requests still avoid re-downloading unchanged content).
When upstream is unreachable and the cached entry is past its TTL, the proxy serves the stale cached copy with a `Warning: 110 - "Response is Stale"` header so clients can tell the data may be outdated.
2026-06-02 07:59:00 +01:00
### Metadata size limit
Upstream metadata responses are buffered in memory before being rewritten and served. `metadata_max_size` caps that buffer to protect against OOM from a misbehaving upstream. Some npm packages with thousands of versions (for example `renovate` ) exceed the 100 MB default, so raise this if you see `metadata response exceeds size limit` in the logs.
```yaml
metadata_max_size: "100MB" # default
```
Or via environment variable: `PROXY_METADATA_MAX_SIZE=250MB` .
2026-10-14 08:39:12 +02:00
### Artifact size limit
2026-10-14 12:02:54 +01:00
Artifacts have no size limit by default. `artifact_max_size` refuses any artifact larger than the limit, so a client cannot fill storage or tie up the proxy by requesting huge downloads. An artifact whose upstream `Content-Length` is over the limit is refused before its body is read; one without a declared size fails once the download passes the limit. Either way nothing is cached and the client gets `502 Bad Gateway` , except when artifacts are streamed: one without a declared size has already started sending when it passes the limit, so the client sees the download cut off. The limit applies to `serve` , `mirror` and streamed artifacts.
2026-10-14 08:39:12 +02:00
```yaml
artifact_max_size: "2GB" # default: unlimited ("" or "0")
```
Or via environment variable: `PROXY_ARTIFACT_MAX_SIZE=2GB` .
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
2026-10-03 04:09:15 -07:00
### Rewritten metadata cache
The npm and Composer handlers rewrite every metadata document they serve, so that download URLs point at the proxy. That means decoding the whole document and encoding it again, and Composer's minified documents are also expanded, which together cost milliseconds per typical package and far more for very large ones. The proxy keeps rewritten documents in memory, so each distinct upstream document is rewritten once and requests arriving while it is being rewritten wait for that rewrite. New bytes from upstream are rewritten again. With version cooldown enabled the cache is bypassed, because cooldown filtering depends on the current time.
2026-10-15 12:46:55 +02:00
Each cached document is also kept gzip-compressed, when that makes it smaller, and served with `Content-Encoding: gzip` to clients that accept it. A large packument is compressed once rather than on every request, by the proxy or by a reverse proxy in front of it. The compressed copy counts against the cache size and typically adds about an eighth to each entry. Documents rewritten outside the cache are served uncompressed.
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
2026-10-03 04:09:15 -07:00
```yaml
metadata_rewrite_cache_size: "256MB" # default; "0" rewrites on every request
```
Or via environment variable: `PROXY_METADATA_REWRITE_CACHE_SIZE=1GB` .
2026-07-07 17:48:36 +01:00
## Upstream HTTP timeout
Protocol handlers use a shared HTTP client for upstream requests such as metadata fetches and pass-through file downloads. `http_timeout` sets that client's per-request timeout. Raise it if slow upstreams or large metadata responses cause `context deadline exceeded` errors.
```yaml
http_timeout: "30s" # default
```
Or via environment variable: `PROXY_HTTP_TIMEOUT=2m` .
fix(server): tune the shared upstream transport defaults (#351)
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes #327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
2026-09-16 09:11:06 +02:00
Set to `"0"` to disable the timeout entirely (requests then rely only on the server's write timeout). Independently of this setting, the shared transport gives up on an upstream that has not sent response headers within 60 seconds.
2026-07-07 17:48:36 +01:00
2026-10-13 13:11:49 +02:00
## Hosted Packages
Operators can push a package archive into the proxy under a name and version. The proxy serves it in place of the upstream archive with the same version, or in addition to the upstream's versions. Use it to ship a patched build of a dependency without running a separate registry.
The `/api/hosted` endpoints are disabled by default. Because a pushed version replaces the upstream one for every client, enabling them requires a token:
```yaml
hosted:
enabled: true
token: "${PROXY_HOSTED_TOKEN}"
max_upload_size: "100MB"
2026-10-13 13:42:34 +02:00
npm_scopes: ["@acme "]
composer_vendors: ["acme"]
2026-10-13 13:11:49 +02:00
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `hosted.enabled` | `PROXY_HOSTED_ENABLED` | Register the `/api/hosted` endpoints (default `false` ) |
| `hosted.token` | `PROXY_HOSTED_TOKEN` | Bearer token required to push and delete; required when enabled, supports `${VAR}` |
| `hosted.max_upload_size` | `PROXY_HOSTED_MAX_UPLOAD_SIZE` | Maximum size of one pushed archive (default `100MB` ) |
2026-10-13 13:42:34 +02:00
| `hosted.npm_scopes` | `PROXY_HOSTED_NPM_SCOPES` | npm scopes served from pushed packages only (comma-separated in the environment) |
| `hosted.composer_vendors` | `PROXY_HOSTED_COMPOSER_VENDORS` | Composer vendors served from pushed packages only |
2026-10-13 13:11:49 +02:00
`hosted.enabled` cannot be combined with `storage.cache_artifacts: false` .
### Pushing a patched package
2026-10-13 13:27:27 +02:00
1. Build the archive with the original package name and the version you push in its manifest (`package.json` or `composer.json` ), e.g. `request-2.88.3.tgz` .
2026-10-13 13:11:49 +02:00
2. Push it:
```bash
export PROXY_HOSTED_TOKEN=...
2026-10-13 15:20:57 +02:00
proxy hosted push -url https://proxy.example.com npm request 2.88.3 ./request-2.88.3.tgz
2026-10-13 13:11:49 +02:00
```
or with curl:
```bash
curl -X PUT --data-binary @request -2.88.3.tgz \
-H "Authorization: Bearer $PROXY_HOSTED_TOKEN" \
https://proxy.example.com/api/hosted/npm/request/2.88.3
```
2026-10-13 13:27:27 +02:00
3. Install it as usual, e.g. `npm install request@2.88.3` . The version is listed in the package metadata, so ranges like `^2.88.0` resolve to it too.
2026-10-13 15:20:57 +02:00
4. Check what is pushed: `proxy hosted list npm request` or `GET /api/hosted/npm/request` .
5. Remove it again: `proxy hosted delete npm request 2.88.3` or `DELETE /api/hosted/npm/request/2.88.3` . If the upstream has the version, clients get the upstream archive again.
2026-10-13 13:11:49 +02:00
### Supported ecosystems
Download handlers look archives up by the filename clients request, so a push is stored under that name:
| Ecosystem | Name | Filename |
|-----------|------|----------|
| `npm` | `{name}` or `@{scope}/{name}` | `{name}-{version}.tgz` , without the scope |
| `composer` | `{vendor}/{package}` | `{package}-{version}.zip` |
Other ecosystems are not supported yet.
2026-10-13 13:27:27 +02:00
The archive must contain the package manifest: `package.json` in the top directory of an npm tarball, or `composer.json` at the root or in the single top directory of a Composer zip. Its name, and its version when it has one, must match the push. Otherwise the push is refused with 400.
### Package metadata
Pushed versions are listed in the package metadata clients resolve against, with the dependencies from their manifest:
- A pushed version takes the place of an upstream version with the same number.
- Pushed versions are added next to the upstream ones and are never held back by cooldown. npm `dist-tags` stay as the upstream has them.
- A package the upstream doesn't know is served from its pushed versions alone. Its npm `latest` tag is the highest pushed version.
- Pushed Composer dev versions (`dev-*` , `*-dev` ) are listed in the `~dev` metadata file.
- npm entries carry the archive's sha512 as `dist.integrity` . Composer entries carry its sha256 as `dist.reference` and leave `shasum` empty.
Metadata of packages with pushed versions is rewritten on every request instead of being cached, so a push or delete shows up on the next request.
2026-10-13 13:42:34 +02:00
### Hosted-only namespaces
A package the upstream doesn't know is served from its pushed versions, but the proxy still asks the upstream about it on every request. If someone later publishes the same name there, their versions are listed next to yours, and a range such as `^1.0.0` can resolve to one of them (dependency confusion).
For private packages:
1. Prefer a namespace you own on the public registry: an npm scope registered on npmjs.com, or a Composer vendor with packages on Packagist. Nobody else can publish under it.
2. List the namespace in `hosted.npm_scopes` or `hosted.composer_vendors` . Its packages are served from pushed versions alone: the upstream is never asked about them, and anything not pushed answers 404.
Other packages, including ones in other scopes and vendors, are proxied as usual.
2026-10-13 13:11:49 +02:00
### Storage and eviction
Pushed archives are stored like cached ones but marked as local. The cache size limit and storage retention never remove them; only a delete does. Scanning, when enabled, checks pushed archives like fetched ones.
2026-04-01 16:14:07 +01:00
## Mirror API
The `/api/mirror` endpoints are disabled by default. Enable them to allow starting mirror jobs via HTTP:
```yaml
mirror_api: true
```
Or via environment variable: `PROXY_MIRROR_API=true` .
When disabled, the endpoints are not registered and return 404.
2026-09-02 16:56:29 +05:30
Start a mirror job with either PURLs or an inline CycloneDX or SPDX JSON document:
```bash
curl -X POST http://localhost:8080/api/mirror \
-H "Content-Type: application/json" \
-d '{"sbom":{"bomFormat":"CycloneDX","components":[{"purl":"pkg:npm/lodash@4 .17.21"}]}}'
```
2026-10-12 00:20:10 +02:00
`platforms` limits which platforms of container images are mirrored, like the command's `--platform` flag:
```bash
curl -X POST http://localhost:8080/api/mirror \
-H "Content-Type: application/json" \
-d '{"purls":["pkg:docker/library/alpine@3 .20"],"platforms":["linux/amd64"]}'
```
2026-03-19 21:06:02 +00:00
## Mirror Command
The `proxy mirror` command pre-populates the cache from various sources. It accepts the same storage and database flags as `serve` .
| Flag | Default | Description |
|------|---------|-------------|
| `--sbom` | | Path to CycloneDX or SPDX SBOM file |
| `--concurrency` | `4` | Number of parallel downloads |
| `--dry-run` | `false` | Show what would be mirrored without downloading |
2026-10-12 00:20:10 +02:00
| `--platform` | | Comma-separated `os/arch[/variant]` list limiting which platforms of container images are mirrored |
2026-03-19 21:06:02 +00:00
| `--config` | | Path to configuration file |
| `--storage-url` | | Storage URL |
| `--database-driver` | | Database driver |
| `--database-path` | | SQLite database file |
| `--database-url` | | PostgreSQL connection URL |
Positional arguments are treated as PURLs:
```bash
proxy mirror pkg:npm/lodash@4 .17.21 pkg:cargo/serde@1 .0.0
```
2026-10-12 00:20:10 +02:00
### Container images
`pkg:docker` and `pkg:oci` PURLs mirror a container image the way a pull does: the manifest or index, the platform manifests, then config and layer blobs. Everything is stored under the keys a pull through `/v2` reads, so a later `docker pull` or containerd pull is served from cache, also when the upstream is unreachable.
```bash
proxy mirror --platform linux/amd64,linux/arm64 pkg:docker/library/alpine@3 .20
proxy mirror 'pkg:docker/containerd/alpine@3 .14.0?repository_url=ghcr.io'
proxy mirror 'pkg:oci/debian@sha256: ...?repository_url=docker.io/library/debian&tag=bookworm'
```
- The version is a tag or a digest. Unversioned image PURLs fail; mirroring all tags is not supported.
- Without `repository_url` the name is routed like a pull without containerd's `ns` parameter: the default OCI registry, repository prefix routes and `upstream/{name}/` . Single-segment Docker Hub names get the `library/` prefix.
- `repository_url` is resolved like containerd's `ns` parameter, so it has to name the default registry, Docker Hub, a configured `upstream.oci` host or a host listed in `upstream.oci_mirrors` . Other hosts are refused.
- `pkg:oci` needs `repository_url` . With `tag=` the tag is mirrored as well; if it now points to another digest, both are mirrored and a warning is logged. `arch=` limits the platforms when `--platform` is not given.
- Without `--platform` every platform of an index is mirrored, including Windows images, which can be large. Attestation manifests of the selected platforms are mirrored too, because Docker fetches them on every pull.
- Non-distributable (foreign) layers are skipped.
- While `scanning.enabled` is set, the command refuses images, because scanners fetch staged layers from the running server. Use the mirror API instead.
2026-01-29 16:33:09 +00:00
## Docker
### SQLite with Local Storage
```bash
docker compose up
```
### PostgreSQL with Local Storage
```bash
docker compose --profile postgres up
```
### PostgreSQL with S3 (MinIO)
```bash
docker compose --profile s3 up
```
## Example Configurations
### Minimal (defaults)
```yaml
listen: ":8080"
```
### Production with PostgreSQL and S3
```yaml
listen: ":8080"
base_url: "https://proxy.example.com"
storage:
url: "s3://my-cache-bucket"
max_size: "100GB"
database:
driver: "postgres"
url: "postgres://proxy:secret@db .example.com:5432/proxy?sslmode=require"
log:
level: "info"
format: "json"
```
### Private npm Registry
```yaml
listen: ":8080"
base_url: "http://localhost:8080"
upstream:
npm: "https://npm.pkg.github.com"
auth:
npm:
type: bearer
token: "${GITHUB_TOKEN}"
```
