gocloud commits whatever a writer received when it is closed, unless its
context was cancelled first. Store closed the writer after a failed copy,
so every download that broke off midway left a partial object under its
fetch path. No database row points at it, so it was never served, but
nothing deleted it either and the orphans kept taking disk space.
- Cancel the writer's context before closing it when the copy fails
- Add a test storing from a reader that errors midway
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
* Stop writing fileblob's .attrs sidecar
fileblob stores blob metadata in an ".attrs" file per object and rewrites
it with os.Create, truncating in place outside the atomic rename that
protects the blob. A read overlapping a write decodes a partial file and
fails with "opening reader: EOF", served as a 502. One writer against
four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob
is most exposed to it, rewriting a key on every refresh while readers
are served from it.
Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is
imported only by internal/storage, Store sets no ContentType, and
Attributes is used only for Size, which comes from os.Stat. A missing
sidecar already defaults cleanly, so "metadata=skip" removes the hazard
rather than locking around it, and saves a write per store.
* Clear .attrs sidecars left by earlier versions
metadata=skip stops fileblob rewriting sidecars but does not delete ones
already on disk, so a sidecar left partial by an interrupted write now
fails every read of its key for good. Before, a later store repaired it
by rewriting.
Store therefore removes the sidecar for the key it writes. Removal is
atomic where the rewrite was not, so a concurrent reader gets the whole
old file or nothing. Delete already removes sidecars, so the two paths
drain a cache between them.
Deriving that path is necessary because fileblob's key escaping is
unexported. It is the identity for a plain key and parts from one only
for keys that are not valid local paths, which is what filepath.Localize
rejects. That also keeps the removal inside the cache directory: without
it a key holding ".." resolves outside.
The clearing test runs one key per storage path the proxy builds, seeded
through a bucket that still writes sidecars so the path under test is
fileblob's own.
* Explain why failed sidecar cleanup does not fail the store
Move the removal into clearLegacySidecar and say why its error is
dropped rather than returned. A failed removal leaves exactly the state
this change inherited, while failing the write would turn a cleanup miss
into a failed request.
Windows makes that concrete: Go opens files with FILE_SHARE_READ and
FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the
sidecar open blocks deletion, and that reader is the workload this
change exists to protect. Propagating would fail stores during exactly
the overlap being fixed. The next store of the key retries.
A test pins it, using a non-empty directory at the sidecar path to make
os.Remove fail with something other than not-exist on any platform.
* Fail the concurrency test if its writer stops
The writer returned silently when Store failed, so the test could pass
with no concurrent writes at all. Its error is now reported, and the
test also checks that at least one write completed.
Reporting it showed the writer had been dying on Windows at its first
collision: Go opens files without FILE_SHARE_DELETE, so a reader holding
the file open makes the writer's rename fail with access denied. The
test now skips there, since it cannot contend a writer with readers on
that platform.
* Clear legacy sidecars for keys fileblob escapes
legacySidecarPath declined any key filepath.Localize rejects, which on
Windows is every key with a colon: OCI digests and Debian epochs. Their
sidecars were never cleared there, and a truncated one kept failing
reads, since fileblob still reads a sidecar it finds under metadata=skip.
fileblob hex-escapes such characters on the way to disk. The path is now
derived the same way, so the sidecar is looked for where fileblob wrote
it. Localize still validates the escaped form, which keeps the removal
inside the cache directory.
* Drop stale comment about declining colon keys on Windows
623ff3e made legacySidecarPath escape keys the way fileblob does, so
colon-bearing keys are now cleared on Windows and the OCI and Debian
rows in TestStoreClearsLegacyAttrsSidecar prove it. The comment
described the behaviour before that commit.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Add GCS storage backend with Workload Identity support
Register gocloud.dev/blob/gcsblob so gs:// URLs are accepted as a storage
backend. Authentication uses Application Default Credentials, which makes
GKE Workload Identity work out of the box; signed URLs (direct_serve)
fall back to the IAM Credentials signBlob API when no private key is
available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Replace gcsblob with lightweight GCS backend
* Extract GCS client into standalone module
---------
Co-authored-by: Anthony A. <github@anthony-arnaud.fr>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
When storage.direct_serve is enabled and the backend supports it (S3,
Azure), cached artifact downloads return a 302 redirect to a presigned
URL instead of streaming bytes through the proxy. Falls back to
streaming when the backend can't sign (fileblob, local filesystem) or
signing fails.
Adds the azureblob driver so azblob:// storage URLs work.
Cache-hit accounting already happened before io.Copy so redirects are
counted correctly; the metrics calls are pulled into a helper so both
paths share them.
Closes#96
`fileblob` creates temp files in `os.TempDir()` (`/tmp`) by default,
then uses `os.Rename` to move them to the final path. When the storage
directory is on a different filesystem (e.g. a Docker volume mount at
`/data`), the rename fails with "invalid cross-device link".
Set `no_tmp_dir=true` on file:// bucket URLs so fileblob creates temp
files next to the final destination instead.
Fixes#65
* Fix all golangci-lint issues across the codebase
Resolve 77 lint issues reported by golangci-lint with gocritic, gocognit,
gocyclo, maintidx, dupl, mnd, unparam, ireturn, goconst, and errcheck
enabled. Net reduction of ~175 lines through shared helpers and
deduplication.
* Suppress staticcheck SA1019 for intentional deprecated field usage
The Storage.Path field is deprecated but still read for backwards
compatibility with existing configs that haven't migrated to the URL field.
