mirror of
https://github.com/git-pkgs/proxy.git
synced 2026-11-09 23:59:16 -05:00
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrew Nesbitt
|
3eadb3eb46 |
Preserve Swift and Go compatibility when upgrading PURL (#642)
* Handle Swift registry identities in cache and vulnerability APIs * Preserve Go cache and policy behavior when upgrading PURL |
||
Simon Chrzanowski
|
391c5b709f |
Index artifacts by filename and shorten Julia comments
CachedArtifactNames filters on filename alone, which the unique (version_purl, filename) index can't serve. |
||
|
Andrew Nesbitt
|
2334a51a52 |
Merge main into feat/mirror-julia
# Conflicts: # cmd/proxy/main.go # docs/configuration.md # internal/mirror/mirror.go # internal/mirror/versions.go # internal/server/mirror_versions.go # internal/server/mirror_versions_test.go |
||
Christian Heim
|
5953af9b05 | Merge main. Main now records UpsertArtifact and its queue entry in one transaction itself, so that part of this branch is gone; ClearArtifactCache, eviction and the unrecorded refetch remain, and an unrecorded refetch counts as failed for instances waiting on its lease. | ||
|
Simon Chrzanowski
|
a7fa07ffdb | Merge main into feat/mirror-julia | ||
|
Simon Chrzanowski
|
3f83d9c0ce |
Mirror Julia packages
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
|
||
|
Simon Chrzanowski
|
fe4be3ccd6 |
Reuse batch download URLs and stored copies for LFS objects
- Keep the download URLs of a batch response in memory until shortly before they expire, so the GET that follows needs no second batch request. A kept URL that fails is replaced by a fresh one. - Cut off a download that isn't the object's size. - Copy an object whose content is already stored, such as for a fork, instead of downloading it, once the upstream has confirmed the repository has it. Artifacts are indexed by content hash for this. |
||
|
Andrew Nesbitt
|
91d0df7141 |
Merge pull request #603 from simonchrz/perf/sqlite-read-pool
Read SQLite through a separate read-only pool |
||
|
Simon Chrzanowski
|
6347735679 |
Treat file::memory: URIs as in-memory and test more of the read pool
isMemoryPath missed URIs such as file::memory:?cache=shared, whose readers would each have got an empty database. New tests open a database left in rollback-journal mode and run writes, hit flushes, lookups and stats scans concurrently. |
||
|
Simon Chrzanowski
|
3bf0cda238 |
Read SQLite through a separate read-only pool
SQLite used one connection for everything, so request lookups queued behind the per-minute stats queries, dashboard pages and eviction scans. Writes keep the single connection. Get, Select, Query and QueryRow now go to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode lets run alongside the writer. A write sent to that pool fails. Pragmas move into the DSN so every connection gets them. Postgres and in-memory SQLite keep one pool for both. |
||
|
Simon Chrzanowski
|
3935ccd6f7 |
Share upstream rate-limit holds across instances
Holds placed by the rate limit gate are written to a new upstream_holds table, and each instance reads that table at most every two seconds and refuses requests to a host any instance holds. Before, every replica had to get its own 429 before it stopped sending. Instances waiting on a fetch lease reread the holds when the fetch fails, so when the holder was rate limited they answer 503 with its Retry-After instead of each sending one more request to the limited host. |
||
|
Andrew Nesbitt
|
7bc4a27ef1 |
Merge pull request #575 from simonchrz/fix/metadata-hit-memory
Stream metadata cache hits for ProxyCached |
||
|
Simon Chrzanowski
|
6c9505f322 |
Drop cached metadata that fails its integrity check
A streamed cache hit whose bytes don't match the row's digest now records an integrity failure and deletes the row, queueing its blob for deletion, as a corrupt cached artifact is handled. The next request fetches the document from upstream instead of aborting until the TTL runs out. |
||
|
Andrew Nesbitt
|
ff539550a3 |
Merge pull request #579 from simonchrz/fix/metadata-path-case
Store metadata under a hash of its cache key |
||
|
Simon Chrzanowski
|
25f62da322 |
Queue only a row's own legacy metadata blob
The case-variant lookup filtered on lower(name), which no index covers, so a legacy row's first lookup scanned every metadata_cache row in its ecosystem. Reclaim already skips a queued path a row still points at, so each row now queues just its own legacy blob once it has moved off it. On case-insensitive storage a case variant still at its legacy path can lose its blob that way, and is then fetched again. |
||
|
Simon Chrzanowski
|
fd4f067a1b |
Keep legacy metadata whose blob matches its digest
Ignoring every row still at its unhashed path dropped the whole metadata cache on upgrade, including on Linux and S3 where keys never collided, and left nothing to serve while the upstream was down. A legacy row's blob is now hashed once and compared with the row's content_digest. When it matches, the blob is this key's document: it is copied to the hashed path and the row moved there, so it is served fresh or as a stale fallback as before. Only a mismatch, where another key's document overwrote the shared file, or a row without a digest is dropped and fetched again. The legacy blobs are queued for deletion once no key differing only in case still reads its legacy path, which on a case-insensitive filesystem is the same file. |
||
|
Andrew Nesbitt
|
44b81a2f1e |
Merge pull request #570 from simonchrz/feat/fetch-leases
Fetch each cache miss once across instances |
||
|
Simon Chrzanowski
|
64162c77d8 |
Keep detached lease waits alive and stop waiting on a failed fetch
A detached fetch waiting on another instance's lease ran the wait under the request context, so a client leaving cancelled every caller joined to it. The wait now runs past the client, bounded by the holder's fetch and the lease expiry. When the holder's fetch failed, each waiting instance took the lease in turn and repeated it, so against a hanging upstream the k-th instance waited about k timeouts. The holder now marks the lease failed. The instances waiting on it then serve their stale metadata copy if they have one, and otherwise fetch at the same time. |
||
|
Simon Chrzanowski
|
743289c8ca |
Let an interrupted mirror schedule run again
A schedule's claim was kept when its run was cut off, so after a restart every instance skipped it for most of an interval. The scheduler now drops the claim when the run ends with its context cancelled. |
||
|
Simon Chrzanowski
|
84111c387a |
Fetch each cache miss once across instances
Instances sharing a Postgres database each fetched a missed artifact from upstream; only callers within one instance shared a fetch. The instance that fetches now holds a lease on the miss in a new fetch_leases table and renews it while it downloads. Another instance missing the same key polls the cache and serves the stored copy, or takes the lease over when the holder releases it without storing anything or stops renewing it. Metadata misses use the same lease when metadata caching is on with a TTL above zero; without a stored copy there is nothing to wait for. proxy_fetch_lease_waits_total counts the misses that waited. Reclaim deletes leases left by stopped instances. Refs #543 |
||
|
Simon Chrzanowski
|
3fcdb0b652 |
Close the SQLite test databases in the lock tests
Windows cannot remove the temporary directory while the file is open. |
||
|
Simon Chrzanowski
|
d02ad9b2ba |
Coordinate instances sharing a Postgres database
Instances started together on an empty database created the schema at the same time and all but one failed on duplicate types. Schema creation and migrations now run under a Postgres advisory lock. Eviction, reclaim, retention and the Gradle build cache sweep take a per-job advisory lock for each pass, and an instance that finds the job running elsewhere skips the pass. Before, every instance evicted the whole overshoot over max_size at once and emptied the cache. Mirror schedules record their last start in a new job_runs table, so each schedule runs once per interval instead of once per instance. Refs #543 |
||
|
Simon Chrzanowski
|
baeccd4408 |
Mirror gems and composer packages under the handler names
proxy mirror filed pkg:gem and pkg:composer packages under rubygems and packagist, the names purl gives their registries, while the handlers store them under gem and composer. Since the resolver accepts both spellings, mirrored gems were downloaded and filed under rubygems, so the packages list filtered by gem didn't show them. The mirror now uses the handler names. The package list, count and search match every spelling of the requested ecosystem, so rows already written as rubygems or packagist, by the mirror or by git-pkgs, show up too without a migration. Fixes #415 |
||
|
Simon Chrzanowski
|
0832c6142b |
Refuse pushes of a version that holds a pushed file under another name
Composer names are case-insensitive, so acme/private and acme/Private share the version record and its integrity but get different filenames. A second push under the other case was accepted and broke the integrity check of the first archive. For ecosystems with one file per version, any pushed file of the version now refuses the push, both before the upload and atomically when recording it. |
||
|
Simon Chrzanowski
|
48916bcb09 |
Never let a cache fill replace a pushed artifact
- Record a fetched artifact with its package and version in one transaction that refuses to replace a pushed artifact, so a fetch that finishes after a push keeps both the pushed file and its version's integrity. Pushes and cache fills share this write, including the compare-and-swap on the storage path that queues every replaced object. - Don't open the stored file for ecosystems without a manifest. - Derive multi-file versions from the file being named in the path. - Validate DELETE paths like PUT paths, answering 400 for malformed ones. - Parse ?sha256= with integrity.ParseHex. |
||
|
Simon Chrzanowski
|
5f431a3d89 |
Test concurrent pushes of one file at the database level
The handler-level test runs on SQLite only. This one also runs against Postgres when PROXY_DATABASE_URL is set. |
||
|
Simon Chrzanowski
|
5c0120f305 |
Record pushes atomically and address review findings
- Record the package, version and artifact of a push in one transaction that only inserts the artifact if no pushed copy exists. Of two concurrent pushes of the same file exactly one wins, and the loser's version integrity is rolled back with it. - Check for an existing file before reading the request body, and send Expect: 100-continue from proxy hosted push, so a refused re-push isn't uploaded first. - Leave an existing version record alone for multi-file ecosystems, so a generic push doesn't reset a version cached from upstream.generic. - Reserve the upstream.generic name "hosted" only while hosting is enabled, and only intercept /generic/hosted/ then. - Accept a file argument in proxy hosted delete for generic only. - Mark multi-file ecosystems explicitly, and give parse a min/max range. |
||
|
Simon Chrzanowski
|
d13b8be0ae |
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
|
||
Abhinav Gautam
|
a97b690858 |
Separate artifact and metadata cache metrics on /metrics and the analytics page (#463)
* Separate artifact and metadata cache metrics on /metrics and the analytics page * Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind |
||
Christian Heim
|
d56d4367c9 |
Count cached metadata against max_size and evict it LRU (#507)
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row. * Shorter comments on the metadata last-access migration. * Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test. * Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it. * The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time. --------- Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> |
||
|
simon chrzanowski
|
85b674f278 |
Keep the stored metadata on an upstream 304 (#495)
When upstream answered a revalidation with 304, the proxy read the cached document back from storage and then stored it again: a full upload and a SHA-256 over identical bytes, for every popular package once per TTL. A 304 now only moves the cache row's fetched_at on, keeping its digest. The update is conditional on the row still recording the digest of the document that was read, so a concurrent fetch that replaced it is not overwritten; in that case, and for rows written before digests were recorded, the document is stored as before. |
||
|
Christian Heim
|
c0320c7c87 | Clearing a record and queueing its file for deletion now happen in one transaction, so a failed queue write keeps the record instead of leaving a file nothing points at. Eviction drops the queue entry again once it has deleted the file itself. | ||
|
simon chrzanowski
|
3dae0fbae0 |
Stop UpsertPackage from clearing packages.source
The artifact source change added source = excluded.source to the packages upsert as well. source isn't in that INSERT's column list, so every package upsert set the column to NULL. The packages upsert is back to what it was, with a test that an existing source survives it. Refs #133 |
||
|
simon chrzanowski
|
e7449d1540 |
Read the package manifest of pushed archives
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest). The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage. The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity. Refs #134 |
||
|
simon chrzanowski
|
5e66688873 |
Push, list and delete npm and Composer archives through /api/hosted
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
|
||
|
simon chrzanowski
|
87505c01c1 |
Mark pushed artifacts as local and keep them out of eviction
Artifacts get a source column: upstream for cached copies of a registry, local for archives pushed to the proxy. Existing rows become upstream through the new 012_add_artifact_source migration. Local artifacts aren't a cache, so the LRU size limit doesn't count or evict them and storage retention never selects them. They are only removed by deleting them. A local artifact has no upstream URL and stores it as empty. Refs #133 |
||
|
simon chrzanowski
|
9f42425646 |
Test upgrading cached metadata through the metadata_dates migration
- Start from the schema before migration 011 on SQLite and PostgreSQL, with npm and Composer metadata cached, and check the rows and stored bytes survive the migration unchanged - After the upgrade, check two replicas serve the cached document with matching ETag and Last-Modified and answer conditional requests with 304 - Change the upstream document and check neither replica answers the old validators with 304, and both agree on the new ones - Give the PostgreSQL handler test a database of its own, so it shares no tables with the database package tests on the same server |
||
Simon Chrzanowski
|
599b2e1473 |
Add a metadata_dates table for rewritten metadata dates
Replicas sharing a database need one place to agree on the Last-Modified of rewritten metadata, which process memory cannot give them. - Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the ETag being dated and its date in Unix seconds, so no zone is involved - Insert a package's first date only if no replica recorded one first - Swap a date only if it is still the one read, so of two replicas switching a package at once only one succeeds - Create it in both schemas and in migration 011 |
||
|
Christian Heim
|
b664530f88 |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
|
||
|
Christian Heim
|
dcf90cfec3 |
Close the Composer download gaps left after #430 (#433)
* GitLab Composer archives are now cached per commit and fetched for the commit the lock file pins, and branch names with a slash are escaped in dist URLs so the download route matches them. * Composer branch archives that an older proxy cached from another commit's URL are now discarded and fetched again, and a refetched artifact records the URL it actually came from. * When a GitLab commit is fetched under its own name, the archive older proxies cached under the shared archive.zip name is dropped. A test now moves the branch from A to B and checks that A keeps coming from the cache. |
||
|
Christian Heim
|
16ebeaa975 |
On Postgres, metadata fetched_at is now stored in UTC, so metadata_ttl no longer expires hours late when the proxy runs east of UTC. Entries written before this expire late one last time and are stored in UTC on their next refresh. (#435) | ||
Ondrej Kokes
|
129e6c6c1b |
Filter cached packages by ecosystem using pills (#319)
* experimental filtering pills * align data in the dashboard table more nicely (#318) * Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#333) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits]( |
||
|
Andrew Nesbitt
|
9dff45d36c |
Merge remote-tracking branch 'origin/main' into analytics-2-ecosystem-stats
# Conflicts: # README.md |
||
montehurd
|
55ed6b9a16 |
Keep hit timestamps from moving backwards
With proxies sharing a database, a batch can flush after a newer hit is already written, and overwriting last_accessed_at and updated_at with its older time could reorder LRU eviction. Both now keep the later of the stored and incoming times. The count still adds up. |
||
Wicliff Wolda
|
4ebd805ffc |
Per-ecosystem cache and download statistics
- Add GetEcosystemStats, aggregating packages, versions, artifacts, cache size, downloads and downloaded bytes per ecosystem - Publish six proxy_ecosystem_* gauges from it on the existing one-minute cache-stats tick - Set and selectively delete rather than Reset, so no scrape lands on a half-populated vector - Report artifacts with no package row under "unattributed" rather than dropping them, so the figures still add up Part 2 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
montehurd
|
4827cb9d76 |
Batch cache hit writes
Every cached download wrote its hit in its own UPDATE. SQLite runs on one connection, so each download was a write transaction the rest queued behind, holding a warm npm cache near 1,000 requests a second with the CPU two thirds idle. Hits are now counted in memory and written in one transaction every database.hit_flush_interval, "1s" by default, "0" for a write per hit. A failed write keeps the hits for the next flush, and Close writes what is pending. The same test then served about 2,200 requests a second. Closes #322. |
||
|
montehurd
|
436155e50c |
Clear before deleting on eviction, and requeue failed reclaims
Eviction deleted an object before checking its record still pointed at it. If a refetch had moved the record, that object was queued and a request that read the record earlier could still be opening it. Eviction now clears first, skips the delete if the record moved, and queues the object when the delete fails after a clear. Reclaim queues a failed delete again, which moves it behind the rest, so objects the backend keeps refusing cannot fill every batch. Also drop the clearArtifactCache passthrough and ClearCachedArtifact's unused context, note that only tests call ArtifactPath, and document that max_size does not count objects waiting to be reclaimed. |
||
|
montehurd
|
4ce6c37bd9 |
Give each fetch its own storage path
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
|
||
|
pinguinfuss
|
ba45227e03 |
test(database): drop every schema table in the Postgres fixture (#349)
createTestPostgresDB dropped artifacts, versions, packages and schema_info before calling CreateSchema, but not the migrations table. On a database that has seen one test the migration records survive, so the next CreateSchema fails while recording 001_add_packages_enrichment_ columns with a duplicate key on migrations_pkey. Running the package against one Postgres therefore failed from the second Postgres-backed test on. Drop vulnerabilities, metadata_cache and migrations as well, so the fixture clears every table CreateSchema creates. The package now passes repeatedly against the same database. |
||
|
pinguinfuss
|
3db2cd5c71 |
fix(database): set connection pool limits for Postgres (#350)
* fix(database): set connection pool limits for Postgres OpenPostgres returned sqlx.Open's handle with database/sql's defaults: no cap on open connections and two idle ones. Under load nearly every request opened a new Postgres session, ran its few statements and closed it again, paying a backend fork and SCRAM authentication each time. Set the pool limits the issue suggests: 32 open and 32 idle connections, idle connections closed after 5 minutes and every connection recycled after 30 minutes. The values live in named constants because the mnd linter rejects the literals inline. The test (skipped without PROXY_DATABASE_URL, like the other Postgres tests) takes 16 connections from the pool, releases them and checks that all 16 stay idle; with the default pool only two survive. Fixes #323 * test(database): pin the pool properties instead of the wiring Review pointed out that the pool test compared MaxOpenConnections to the constant it was set from, so the assertion followed the constant and would have accepted postgresMaxOpenConns = 0 (unlimited), and that its burst of 16 only proved MaxIdleConns >= 16. The burst now takes postgresMaxIdleConns connections, so every configured idle slot has to survive the release, and the open cap is checked to be finite and large enough for that burst before any connection is taken, so a cap below the idle count fails fast instead of blocking in db.Conn. The doc comment now says which settings the test covers; the idle-time and lifetime settings only show up in DBStats.MaxIdleTimeClosed and MaxLifetimeClosed after minutes of wall-clock time and stay unexercised. * test(database): guard the pool test against a vacuous burst Review showed that with the burst tied to postgresMaxIdleConns the test also passed for a constant of 2, database/sql's default, or of 0, where it took no connections at all. It now fails outright unless the configured idle count exceeds the default, and every connection it takes is released in a cleanup, so an assertion failure mid-burst no longer leaves sessions open for the rest of the test binary. |