A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
Escaping _ in the search pattern stopped typing_extensions from finding
typing-extensions, which worked before because _ matched any single
character. Escape only % and the escape character.
SearchPackages and CountSearchResults matched names with
p.name LIKE '%query%'. LIKE ignores ASCII case on SQLite but not on
Postgres, so the same search found different packages depending on the
database. The % and _ in a query were also taken as wildcards. Compare
LOWER(name) with LOWER(pattern) and escape the wildcards.
A mirror run that finds a blob cached records it in mirrored_at, which
retention counts as use. LRU eviction keeps using last_accessed_at, so
#508's decision not to count mirror runs as cache hits stands.
Fixes#648.
The API handler takes the enrichment calls it uses as an interface, so
the test can answer them without reaching a registry.
Remove FillVersionPublishedAt until the shared cooldown filter calls it.
TestAPKSnapshots_CacheHit covers the cache-hit branch with a metadata TTL
above 0. proxyCachedWithEncoding takes optional hooks for a cache hit and a
fetched body, so the APK handler no longer keeps a copy of it.
FuzzParse checks that an uploaded body never makes the inventory panic
and that every component it records has a package URL, ecosystem and
version. TestSaveProjectConcurrentUploads saves one project from eight
uploads at once and checks that each succeeds and the components of
exactly one remain.
Covers namespaced npm and Maven PURLs, a publish time under an alias of
a package row's ecosystem, and a PURL starting with the orphan prefix,
and checks the per-ecosystem versions against GetCacheStats' total.
Cooldown stores the publish time of every version it looks up, also of
packages nobody downloaded, which have no package row. The ecosystem stats
counted those versions as unattributed. They now count under the type of
their package PURL; only a PURL that can't be parsed stays unattributed.
These clients find new versions in a repository index. The refresh fetches
the InRelease, APKINDEX and repomd.xml files clients fetched again, reads
the Packages and primary indexes they point at for the repositories the
most used packages came from, and mirrors the newer file per package and
architecture, checked against the index's SHA-256 where it has one.
The refresh fetches Debian, Alpine and RPM indexes again from the URLs
clients fetched them from, which the hashed cache keys don't keep. Rows
cached before get their URL on the next revalidation, even on a 304.
mirror.refresh mirrors, for the top packages per ecosystem by recent
downloads, the newest release of each major line a version is cached on.
Releases within cooldown or on the denylist are skipped, and PyPI and conda
get only the kinds of files clients fetched before. max_versions and
max_bytes cap one run. Instances sharing a database take turns as for
schedules.
A snapshot was recorded after the response by reading the metadata cache
again, so a refresh that replaced the cached copy in between lost the
version the request served. Record it from the served bytes before
responding instead, and compare a new snapshot with the version current
at its own time, so one recorded late keeps its place in the history.
AddMetadataSnapshot takes a transaction-scoped advisory lock on the
document before its check and insert, so two replicas recording the same
version can't both read the previous digest and both insert. SQLite runs
one write at a time, so it keeps the single statement.
Part of #617. The package page marks versions in cooldown with when they
become available, and lists a version only seen in metadata (no artifact
row) only while it is held back. /api/package returns the package's
cooldown and available_at on a held-back version.
FillVersionPublishedAt stores a publish time without overwriting one, for
the cooldown filters to call once they record the versions they hold back.
Two requests recording the same index version in the same millisecond
stored to one path, and the one whose row was not added deleted the
blob the other's row points at. The path now ends in a random suffix.
Also trims comments that restated the PR, notes in the docs that turning
snapshots off leaves them in place, and moves the git cache setup out of
serve, which lint found too long after the merge.
- GetProjectsUsingPackage no longer counts each project's components
per row; the package page only needs names and versions. The ?purl=
listing leaves component_count out accordingly.
- The delete button sends to DELETE /ui/api/projects/{project}, the same
handler as the API and still behind the token, so it sits under /ui
like the cache deletes.
- The project page answers 500 when its components can't be read,
instead of showing an empty project.
- DELETE of a path ending in /sbom that names no project answers 400
with a hint, since it's most likely the upload path.
- The upload endpoint lists JSON, XML and plain text (SPDX tag-value)
as accepted.
- The docs say that versions are compared as the SBOM spells them.
The vulnerabilities table is only written by tests: nothing in the proxy
calls UpsertVulnerability, so the dashboard card, the per-project count
and the per-component markers were always empty. Remove them along with
GetComponentVulnerabilities and inventory.Affected, and keep the
outdated marker, which uses the latest version the handlers record.
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
Within metadata_ttl cached npm and Composer metadata is served without
asking upstream, so a new release can take that long to show up. Both
registries publish a feed of the packages that changed.
upstream.npm_change_feed and upstream.composer_change_feed (off by
default, require cache_metadata) start a reader per feed that runs every
minute and clears fetched_at on the cached rows of listed packages that
were fetched before the read. The rows keep their document and ETag, so
the next request revalidates them upstream. Routed packages are cached
under other keys and never match. A Packagist resync expires all cached
Composer metadata. Failed reads back off up to 15 minutes. The feed
position is kept in memory, so after a restart the reader starts from the
feed's current position.
Alpine mirrors keep only the current build of each package, so once a
package is upgraded `apk add curl=8.5.0-r0` fails: the current index no
longer lists the old build, even when the proxy still has it cached.
With upstream.apk_snapshots set (e.g. "90d"), each version of an index or
detached signature the metadata cache stores is copied to
_snapshots/alpine/ and recorded in a new metadata_snapshots table with the
time it was first served. /apk/{name}@{YYYY-MM-DD}/ serves the version that
was current at the end of that day; packages below it are the plain path's
requests. The kept bytes are the signed upstream bytes, so apk verifies
them unchanged.
An hourly job deletes versions replaced longer ago than the configured
duration and queues their blobs on the pending-delete queue; the newest
version of each index stays. Packages listed by kept snapshots are not
protected from eviction or retention; the docs say so.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
- Keep the download URLs of a batch response in memory until shortly
before they expire, so the GET that follows needs no second batch
request. A kept URL that fails is replaced by a fresh one.
- Cut off a download that isn't the object's size.
- Copy an object whose content is already stored, such as for a fork,
instead of downloading it, once the upstream has confirmed the
repository has it. Artifacts are indexed by content hash for this.
isMemoryPath missed URIs such as file::memory:?cache=shared, whose
readers would each have got an empty database. New tests open a database
left in rollback-journal mode and run writes, hit flushes, lookups and
stats scans concurrently.
SQLite used one connection for everything, so request lookups queued
behind the per-minute stats queries, dashboard pages and eviction scans.
Writes keep the single connection. Get, Select, Query and QueryRow now go
to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode
lets run alongside the writer. A write sent to that pool fails.
Pragmas move into the DSN so every connection gets them. Postgres and
in-memory SQLite keep one pool for both.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
A streamed cache hit whose bytes don't match the row's digest now
records an integrity failure and deletes the row, queueing its blob for
deletion, as a corrupt cached artifact is handled. The next request
fetches the document from upstream instead of aborting until the TTL
runs out.
