mirror of
https://github.com/git-pkgs/proxy.git
synced 2026-09-16 07:42:05 -04:00
| Filename | Latest commit message | Latest commit date |
|---|---|---|
0089917485* Stop writing fileblob's .attrs sidecar
fileblob stores blob metadata in an ".attrs" file per object and rewrites
it with os.Create, truncating in place outside the atomic rename that
protects the blob. A read overlapping a write decodes a partial file and
fails with "opening reader: EOF", served as a 502. One writer against
four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob
is most exposed to it, rewriting a key on every refresh while readers
are served from it.
Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is
imported only by internal/storage, Store sets no ContentType, and
Attributes is used only for Size, which comes from os.Stat. A missing
sidecar already defaults cleanly, so "metadata=skip" removes the hazard
rather than locking around it, and saves a write per store.
* Clear .attrs sidecars left by earlier versions
metadata=skip stops fileblob rewriting sidecars but does not delete ones
already on disk, so a sidecar left partial by an interrupted write now
fails every read of its key for good. Before, a later store repaired it
by rewriting.
Store therefore removes the sidecar for the key it writes. Removal is
atomic where the rewrite was not, so a concurrent reader gets the whole
old file or nothing. Delete already removes sidecars, so the two paths
drain a cache between them.
Deriving that path is necessary because fileblob's key escaping is
unexported. It is the identity for a plain key and parts from one only
for keys that are not valid local paths, which is what filepath.Localize
rejects. That also keeps the removal inside the cache directory: without
it a key holding ".." resolves outside.
The clearing test runs one key per storage path the proxy builds, seeded
through a bucket that still writes sidecars so the path under test is
fileblob's own.
* Explain why failed sidecar cleanup does not fail the store
Move the removal into clearLegacySidecar and say why its error is
dropped rather than returned. A failed removal leaves exactly the state
this change inherited, while failing the write would turn a cleanup miss
into a failed request.
Windows makes that concrete: Go opens files with FILE_SHARE_READ and
FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the
sidecar open blocks deletion, and that reader is the workload this
change exists to protect. Propagating would fail stores during exactly
the overlap being fixed. The next store of the key retries.
A test pins it, using a non-empty directory at the sidecar path to make
os.Remove fail with something other than not-exist on any platform.
* Fail the concurrency test if its writer stops
The writer returned silently when Store failed, so the test could pass
with no concurrent writes at all. Its error is now reported, and the
test also checks that at least one write completed.
Reporting it showed the writer had been dying on Windows at its first
collision: Go opens files without FILE_SHARE_DELETE, so a reader holding
the file open makes the writer's rename fail with access denied. The
test now skips there, since it cannot contend a writer with readers on
that platform.
* Clear legacy sidecars for keys fileblob escapes
legacySidecarPath declined any key filepath.Localize rejects, which on
Windows is every key with a colon: OCI digests and Debian epochs. Their
sidecars were never cleared there, and a truncated one kept failing
reads, since fileblob still reads a sidecar it finds under metadata=skip.
fileblob hex-escapes such characters on the way to disk. The path is now
derived the same way, so the sidecar is looked for where fileblob wrote
it. Localize still validates the escaped form, which keeps the removal
inside the cache directory.
* Drop stale comment about declining colon keys on Windows
|
||
| .. | ||
| blob.go | ||
| blob_test.go | ||
| gcs.go | ||
| gcs_test.go | ||
| storage.go | ||
| storage_test.go | ||