Compare commits

..
Author SHA1 Message Date
maximilien
99923a8718 Merge pull request 'fix reflected xss when returning errors on web endpoint' (#1472) from escape-errors-v1 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1472
2026-06-11 06:51:01 +00:00
trinity-1686a
0d0068317d fix reflected xss when returning errors on web endpoint
Reported by Filippo Decortes from ares-cyber.ai
2026-06-10 19:43:51 +02:00
Alex Auvolat
b6b18427a5 use optimization level 3 and thin LTO for release builds (#1405)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1405
Co-authored-by: Alex Auvolat <lx@deuxfleurs.fr>
Co-committed-by: Alex Auvolat <lx@deuxfleurs.fr>
2026-04-16 08:47:02 +00:00
Gauthier Zirnhelt
9987166b2b Fix the LifecycleWorker being uncooperative (#1396)
## Summary

This PR ensures that the `LifecycleWorker` yields at least once to the Tokio scheduler in between each batch of 100 objects.

## Problem being solved

I'm administrating a Garage cluster which has been experiencing timeouts on all endpoints while the lifecycle worker is running at midnight UTC : `Ping timeout` error messages and even requests eventually failing due to `Could not reach quorum ...`.

I have found that this happens while the lifecycle worker is working on a big bucket (containing millions of objects) with a lifecycle rule that applies to very few objects.
The `process_object()` function does not hit any `await`:
- `last_bucket` is always the same, so the `bucket_table` is not read asynchronously
- no transaction is made on the `object_table` because my lifecycle rule (almost) never applies to any object

The first commit in this PR adds an executable which reproduces the problem that I've been experiencing in a self-contained way : the lifecycle worker starves the Tokio scheduler so much that no other task is able to run (or very rarely).
To run it : `cargo run -p garage_model --bin lifecycle-starvation-test`.
This commit can be dropped post-review, as it's only useful to demonstrate the starvation.

The error messages completely stopped after adding the extra yield to the nodes of my cluster.
The duration of the lifecycle worker task does not appear to have changed at all from what I can see (looking at the timestamps produced either by the self-contained binary or by each of my nodes with the `Lifecycle worker finished` message).

## Note

An other potential fix would have been to force the `WorkerProcessor` to yield before re-enqueuing a busy task, but this would have affected all Garage workers even though it's only the `LifecycleWorker` being uncooperative.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1396
Reviewed-by: Alex <lx@deuxfleurs.fr>
Co-authored-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
Co-committed-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
2026-04-15 09:56:24 +00:00
trinity-1686a
b72b090a09 fix silent write errors (#1358)
fix #1355

some write errors are not reported when calling write_all. That's notably the case of ENOSPC on small buffers (1MiB).
on ext4, the error is catched when calling flush(). This is hopefully the case on most local filesystems, though afaik this assumption doesn't hold for NFS

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1358
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-committed-by: trinity-1686a <trinity@deuxfleurs.fr>
2026-02-21 07:21:24 +00:00
Armael
8551aefed4 Fix: correctly parse CORS website configuration with no rules (#1320)
When sending a website config with an empty list of CORS rules, garage currently incorrectly refuses it with error message "Invalid XML: missing field `CORSRule`".
This fix the issue by following the documentation of quick-xml related to serde field parameters for this specific scenario:  https://docs.rs/quick-xml/latest/quick_xml/de/#sequences-xsall-and-xssequence-xml-schema-types .

(I've based this PR on main-v1 because we want it for deuxfleurs' deployment.)

Co-authored-by: Armaël Guéneau <armael.gueneau@ens-lyon.org>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1320
Co-authored-by: Armael <armael@noreply.localhost>
Co-committed-by: Armael <armael@noreply.localhost>
2026-02-07 13:11:20 +00:00
Alex Auvolat
47bf5d9fb0 bump version to v1.3.1 2026-01-24 13:01:27 +01:00
Alex Auvolat
5df37dae5e update cargo dependencies in main-v1 (#1299)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1299
Co-authored-by: Alex Auvolat <lx@deuxfleurs.fr>
Co-committed-by: Alex Auvolat <lx@deuxfleurs.fr>
2026-01-24 11:59:01 +00:00
Alex
44af0bdab3 Merge pull request 'Backport #1283 and #1290 to main-v1' (#1297) from backports-v1 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1297
2026-01-24 11:34:28 +00:00
rmoff
a7d6620e18 Fix typo in error message 2026-01-24 12:21:45 +01:00
Joe Anderson
8eb12755e4 Allow bucket to be missing from presigned post params 2026-01-24 12:21:25 +01:00
256 changed files with 6183 additions and 18362 deletions

View file

@ -16,16 +16,6 @@ steps:
commands:
- nix-build -j4 --attr flakePackages.fmt
- name: check typos
image: nixpkgs/nix:nixos-24.05
commands:
- nix-shell --attr ci --run typos
- name: check lints with clippy
image: nixpkgs/nix:nixos-24.05
commands:
- nix-build -j4 --attr flakePackages.clippy
- name: build
image: nixpkgs/nix:nixos-24.05
commands:

View file

@ -38,15 +38,7 @@ steps:
- matrix:
ARCH: i386
- name: upgrade tests from v1.0.0
image: nixpkgs/nix:nixos-24.05
commands:
- nix-shell --attr ci --run "./script/test-upgrade.sh v1.0.0 x86_64-unknown-linux-musl" || (cat /tmp/garage.log; false)
when:
- matrix:
ARCH: amd64
- name: upgrade tests from v0.8.4
- name: upgrade tests
image: nixpkgs/nix:nixos-24.05
commands:
- nix-shell --attr ci --run "./script/test-upgrade.sh v0.8.4 x86_64-unknown-linux-musl" || (cat /tmp/garage.log; false)

154
Cargo.lock generated
View file

@ -301,9 +301,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.102.0"
version = "1.120.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75ddb925e840f49446aa6338b67abdbec04b4ebf923b7da038ec4c35afb916cd"
checksum = "06673901e961f20fa8d7da907da48f7ad6c1b383e3726c22bd418900f015abe1"
dependencies = [
"aws-credential-types",
"aws-runtime",
@ -313,6 +313,7 @@ dependencies = [
"aws-smithy-eventstream",
"aws-smithy-http",
"aws-smithy-json",
"aws-smithy-observability",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
@ -369,9 +370,9 @@ dependencies = [
[[package]]
name = "aws-smithy-checksums"
version = "0.63.6"
version = "0.63.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9054b4cc5eda331cde3096b1576dec45365c5cbbca61d1fffa5f236e251dfce7"
checksum = "23374b9170cbbcc6f5df8dc5ebb9b6c5c28a3c8f599f0e8b8b10eb6f4a5c6e74"
dependencies = [
"aws-smithy-http",
"aws-smithy-types",
@ -865,15 +866,41 @@ dependencies = [
]
[[package]]
name = "crc-fast"
version = "1.10.0"
name = "crc"
version = "3.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5"
checksum = "9710d3b3739c2e349eb44fe848ad0b7c8cb1e42bd87ee49371df2f7acaf3e675"
dependencies = [
"crc-catalog",
]
[[package]]
name = "crc-catalog"
version = "2.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5"
[[package]]
name = "crc-fast"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2fd92aca2c6001b1bf5ba0ff84ee74ec8501b52bbef0cac80bf25a6c1d87a83d"
dependencies = [
"crc",
"digest",
"rustversion",
"spin 0.10.0",
]
[[package]]
name = "crc32c"
version = "0.6.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47"
dependencies = [
"rustc_version",
]
[[package]]
name = "crc32fast"
version = "1.5.0"
@ -1174,6 +1201,12 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "foldhash"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
[[package]]
name = "form_urlencoded"
version = "1.2.2"
@ -1278,7 +1311,7 @@ dependencies = [
[[package]]
name = "garage"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"assert-json-diff",
"async-trait",
@ -1289,7 +1322,7 @@ dependencies = [
"bytes",
"bytesize",
"chrono",
"crc-fast",
"crc32fast",
"format_table",
"futures",
"garage_api_admin",
@ -1318,6 +1351,7 @@ dependencies = [
"opentelemetry-otlp",
"opentelemetry-prometheus",
"parse_duration",
"serde",
"serde_json",
"sha1",
"sha2",
@ -1329,21 +1363,16 @@ dependencies = [
"tracing",
"tracing-journald",
"tracing-subscriber",
"utoipa",
]
[[package]]
name = "garage_api_admin"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"argon2",
"async-trait",
"bytesize",
"chrono",
"format_table",
"futures",
"garage_api_common",
"garage_block",
"garage_model",
"garage_rpc",
"garage_table",
@ -1353,7 +1382,6 @@ dependencies = [
"hyper 1.8.1",
"opentelemetry",
"opentelemetry-prometheus",
"paste",
"prometheus",
"serde",
"serde_json",
@ -1361,17 +1389,17 @@ dependencies = [
"tokio",
"tracing",
"url",
"utoipa",
]
[[package]]
name = "garage_api_common"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"base64 0.21.7",
"bytes",
"chrono",
"crc-fast",
"crc32c",
"crc32fast",
"crypto-common",
"futures",
"garage_model",
@ -1399,7 +1427,7 @@ dependencies = [
[[package]]
name = "garage_api_k2v"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"base64 0.21.7",
"futures",
@ -1422,14 +1450,15 @@ dependencies = [
[[package]]
name = "garage_api_s3"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"aes-gcm",
"async-compression",
"base64 0.21.7",
"bytes",
"chrono",
"crc-fast",
"crc32c",
"crc32fast",
"form_urlencoded",
"futures",
"garage_api_common",
@ -1440,7 +1469,6 @@ dependencies = [
"garage_table",
"garage_util",
"hex",
"hmac",
"http 1.4.0",
"http-body-util",
"http-range",
@ -1467,7 +1495,7 @@ dependencies = [
[[package]]
name = "garage_block"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"arc-swap",
"async-compression",
@ -1478,6 +1506,7 @@ dependencies = [
"garage_db",
"garage_net",
"garage_rpc",
"garage_table",
"garage_util",
"hex",
"opentelemetry",
@ -1491,7 +1520,7 @@ dependencies = [
[[package]]
name = "garage_db"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"fjall",
"heed",
@ -1506,9 +1535,8 @@ dependencies = [
[[package]]
name = "garage_model"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"argon2",
"async-trait",
"base64 0.21.7",
"blake2",
@ -1534,7 +1562,7 @@ dependencies = [
[[package]]
name = "garage_net"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"arc-swap",
"bytes",
@ -1559,7 +1587,7 @@ dependencies = [
[[package]]
name = "garage_rpc"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"arc-swap",
"async-trait",
@ -1591,7 +1619,7 @@ dependencies = [
[[package]]
name = "garage_table"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"arc-swap",
"async-trait",
@ -1612,7 +1640,7 @@ dependencies = [
[[package]]
name = "garage_util"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"arc-swap",
"async-trait",
@ -1644,13 +1672,14 @@ dependencies = [
[[package]]
name = "garage_web"
version = "2.2.0"
version = "1.3.1"
dependencies = [
"garage_api_common",
"garage_api_s3",
"garage_model",
"garage_table",
"garage_util",
"html-escape",
"http 1.4.0",
"http-body-util",
"hyper 1.8.1",
@ -1806,9 +1835,7 @@ version = "0.15.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
"foldhash 0.1.5",
]
[[package]]
@ -1816,6 +1843,11 @@ name = "hashbrown"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash 0.2.0",
]
[[package]]
name = "hashlink"
@ -1918,6 +1950,15 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "html-escape"
version = "0.2.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d1ad449764d627e22bfd7cd5e8868264fc9236e07c752972b4080cd351cb476"
dependencies = [
"utf8-width",
]
[[package]]
name = "http"
version = "0.2.12"
@ -2273,8 +2314,6 @@ checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017"
dependencies = [
"equivalent",
"hashbrown 0.16.1",
"serde",
"serde_core",
]
[[package]]
@ -2653,11 +2692,11 @@ checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lru"
version = "0.12.5"
version = "0.16.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38"
checksum = "a1dc47f592c06f33f8e3aea9591776ec7c9f9e4124778ff8a3c3b87159f7e593"
dependencies = [
"hashbrown 0.15.5",
"hashbrown 0.16.1",
]
[[package]]
@ -3079,12 +3118,6 @@ dependencies = [
"subtle",
]
[[package]]
name = "paste"
version = "1.0.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
[[package]]
name = "path-absolutize"
version = "3.1.1"
@ -4797,6 +4830,12 @@ dependencies = [
"serde",
]
[[package]]
name = "utf8-width"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1292c0d970b54115d14f2492fe0170adf21d68a1de108eebc51c1df4f346a091"
[[package]]
name = "utf8_iter"
version = "1.0.4"
@ -4809,29 +4848,6 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "utoipa"
version = "5.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2fcc29c80c21c31608227e0912b2d7fddba57ad76b606890627ba8ee7964e993"
dependencies = [
"indexmap 2.13.0",
"serde",
"serde_json",
"utoipa-gen",
]
[[package]]
name = "utoipa-gen"
version = "5.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d79d08d92ab8af4c5e8a6da20c47ae3f61a0f1dabc1997cdf2d082b757ca08b"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.114",
]
[[package]]
name = "uuid"
version = "1.4.1"

View file

@ -24,22 +24,22 @@ default-members = ["src/garage"]
# Internal Garage crates
format_table = { version = "0.1.1", path = "src/format-table" }
garage_api_common = { version = "2.2.0", path = "src/api/common" }
garage_api_admin = { version = "2.2.0", path = "src/api/admin" }
garage_api_s3 = { version = "2.2.0", path = "src/api/s3" }
garage_api_k2v = { version = "2.2.0", path = "src/api/k2v" }
garage_block = { version = "2.2.0", path = "src/block" }
garage_db = { version = "2.2.0", path = "src/db", default-features = false }
garage_model = { version = "2.2.0", path = "src/model", default-features = false }
garage_net = { version = "2.2.0", path = "src/net" }
garage_rpc = { version = "2.2.0", path = "src/rpc" }
garage_table = { version = "2.2.0", path = "src/table" }
garage_util = { version = "2.2.0", path = "src/util" }
garage_web = { version = "2.2.0", path = "src/web" }
garage_api_common = { version = "1.3.1", path = "src/api/common" }
garage_api_admin = { version = "1.3.1", path = "src/api/admin" }
garage_api_s3 = { version = "1.3.1", path = "src/api/s3" }
garage_api_k2v = { version = "1.3.1", path = "src/api/k2v" }
garage_block = { version = "1.3.1", path = "src/block" }
garage_db = { version = "1.3.1", path = "src/db", default-features = false }
garage_model = { version = "1.3.1", path = "src/model", default-features = false }
garage_net = { version = "1.3.1", path = "src/net" }
garage_rpc = { version = "1.3.1", path = "src/rpc" }
garage_table = { version = "1.3.1", path = "src/table" }
garage_util = { version = "1.3.1", path = "src/util" }
garage_web = { version = "1.3.1", path = "src/web" }
k2v-client = { version = "0.0.4", path = "src/k2v-client" }
# External crates from crates.io
arc-swap = "1.1"
arc-swap = "1.0"
argon2 = "0.5"
async-trait = "0.1.7"
backtrace = "0.3"
@ -48,13 +48,15 @@ blake2 = "0.10"
bytes = "1.0"
bytesize = "1.1"
cfg-if = "1.0"
chrono = { version = "0.4", features = ["serde"] }
crc-fast = "1.6"
chrono = "0.4"
crc32fast = "1.4"
crc32c = "0.6"
crypto-common = "0.1"
gethostname = "0.4"
git-version = "0.3.4"
hex = "0.4"
hexdump = "0.1"
html-escape = "0.2.13"
hmac = "0.12"
itertools = "0.12"
ipnet = "2.9.0"
@ -65,7 +67,6 @@ nix = { version = "0.29", default-features = false, features = ["fs"] }
nom = "7.1"
parking_lot = "0.12"
parse_duration = "2.1"
paste = "1.0"
pin-project = "1.0.12"
pnet_datalink = "0.34"
rand = "0.8"
@ -95,13 +96,12 @@ fjall = "2.4"
async-compression = { version = "0.4", features = ["tokio", "zstd"] }
zstd = { version = "0.13", default-features = false }
quick-xml = { version = "0.26", features = ["serialize"] }
quick-xml = { version = "0.26", features = [ "serialize" ] }
rmp-serde = "1.1.2"
serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
serde_bytes = "0.11"
serde_json = "1.0"
toml = { version = "0.8", default-features = false, features = ["parse"] }
utoipa = { version = "5.3.1", features = ["chrono"] }
# newer version requires rust edition 2021
k8s-openapi = { version = "0.21", features = ["v1_24"] }
@ -115,7 +115,7 @@ httpdate = "1.0"
http-range = "0.1"
http-body-util = "0.1"
hyper = { version = "1.0", default-features = false }
hyper-util = { version = "0.1", features = ["full"] }
hyper-util = { version = "0.1", features = [ "full" ] }
multer = "3.0"
percent-encoding = "2.2"
roxmltree = "0.19"
@ -123,11 +123,11 @@ url = "2.3"
futures = "0.3"
futures-util = "0.3"
tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
tokio = { version = "1.0", default-features = false, features = ["net", "rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
tokio-util = { version = "0.7", features = ["compat", "io"] }
tokio-stream = { version = "0.1", features = ["net"] }
opentelemetry = { version = "0.17", features = ["rt-tokio", "metrics", "trace"] }
opentelemetry = { version = "0.17", features = [ "rt-tokio", "metrics", "trace" ] }
opentelemetry-prometheus = "0.10"
opentelemetry-otlp = "0.10"
opentelemetry-contrib = "0.9"
@ -147,12 +147,8 @@ aws-smithy-runtime = { version = "1.8", default-features = false, features = ["t
aws-sdk-config = { version = "1.62", default-features = false }
aws-sdk-s3 = { version = "1.79", default-features = false, features = ["rt-tokio"] }
[profile.dev]
#lto = "thin" # disabled for now, adds 2-4 min to each CI build
lto = "off"
[profile.release]
lto = true
codegen-units = 1
lto = "thin"
codegen-units = 16
opt-level = 3
strip = true
strip = "debuginfo"

View file

@ -1,7 +1,7 @@
<!DOCTYPE html>
<html>
<head>
<title>Garage administration API v0</title>
<title>Garage Adminstration API v0</title>
<!-- needed for adaptive design -->
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">

View file

@ -3,10 +3,10 @@ info:
version: v0.8.0
title: Garage Administration API v0+garage-v0.8.0
description: |
Administrate your Garage cluster programmatically, including status, layout, keys, buckets, and maintenance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionally, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
Administrate your Garage cluster programatically, including status, layout, keys, buckets, and maintainance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionnaly, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
/status:
get:
tags:

View file

@ -1,7 +1,7 @@
<!DOCTYPE html>
<html>
<head>
<title>Garage administration API v1</title>
<title>Garage Adminstration API v0</title>
<!-- needed for adaptive design -->
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">

View file

@ -3,10 +3,10 @@ info:
version: v0.9.0
title: Garage Administration API v0+garage-v0.9.0
description: |
Administrate your Garage cluster programmatically, including status, layout, keys, buckets, and maintenance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionally, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
Administrate your Garage cluster programatically, including status, layout, keys, buckets, and maintainance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionnaly, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
/health:
get:
tags:
@ -440,7 +440,7 @@ paths:
- "false"
example: "true"
required: false
description: "Whether or not the secret key should be returned in the response"
description: "Wether or not the secret key should be returned in the response"
responses:
'500':
description: "The server can not handle your request. Check your connectivity with the rest of the cluster."

View file

@ -1,24 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>Garage administration API v2</title>
<!-- needed for adaptive design -->
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link href="./css/redoc.css" rel="stylesheet">
<!--
Redoc doesn't change outer page styles
-->
<style>
body {
margin: 0;
padding: 0;
}
</style>
</head>
<body>
<redoc spec-url='./garage-admin-v2.json'></redoc>
<script src="./redoc.standalone.js"> </script>
</body>
</html>

File diff suppressed because it is too large Load diff

View file

@ -51,4 +51,4 @@ We are currently building this SDK for [Python](@/documentation/build/python.md#
More information:
- [In the reference manual](@/documentation/reference-manual/admin-api.md)
- [Full specification](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html)
- [Full specifiction](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html)

View file

@ -5,13 +5,13 @@ weight = 99
## S3
If you are developing a new application, you may want to use Garage to store your user's media.
If you are developping a new application, you may want to use Garage to store your user's media.
The S3 API that Garage uses is a standard REST API, so as long as you can make HTTP requests,
you can query it. You can check the [S3 REST API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) from Amazon to learn more.
Developing your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already available.
Developping your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already avalaible.
Some of them are maintained by Amazon, some by Minio, others by the community.

View file

@ -23,7 +23,7 @@ To configure S3-compatible software to interact with Garage,
you will need the following parameters:
- An **API endpoint**: this corresponds to the HTTP or HTTPS address
used to contact the Garage server. When running Garage locally this will usually
used to contact the Garage server. When runing Garage locally this will usually
be `http://127.0.0.1:3900`. In a real-world setting, you would usually have a reverse-proxy
that adds TLS support and makes your Garage server available under a public hostname
such as `https://garage.example.com`.

View file

@ -12,9 +12,8 @@ In this section, we cover the following web applications:
| [Mastodon](#mastodon) | ✅ | Natively supported |
| [Matrix](#matrix) | ✅ | Tested with `synapse-s3-storage-provider` |
| [ejabberd](#ejabberd) | ✅ | `mod_s3_upload` |
| [Ente](#ente) | ✅ | Natively supported |
| [Pixelfed](#pixelfed) | ❓ | Natively supported |
| [Pleroma](#pleroma) | ✅ | Natively supported |
| [Pixelfed](#pixelfed) | ✅ | Natively supported |
| [Pleroma](#pleroma) | ❓ | Not yet tested |
| [Lemmy](#lemmy) | ✅ | Supported with pict-rs |
| [Funkwhale](#funkwhale) | ❓ | Not yet tested |
| [Misskey](#misskey) | ❓ | Not yet tested |
@ -54,7 +53,7 @@ garage bucket allow nextcloud --read --write --key nextcloud-key
Now edit your Nextcloud configuration file to enable object storage.
On my installation, the config. file is located at the following path: `/var/www/nextcloud/config/config.php`.
We will add a new root key to the `$CONFIG` dictionary named `objectstore`:
We will add a new root key to the `$CONFIG` dictionnary named `objectstore`:
```php
<?php
@ -413,7 +412,7 @@ mc mirror --newer-than "3h" ./public/system/ garage/mastodon-data
## Matrix
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developed by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developped by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
### synapse-s3-storage-provider (synapse only)
@ -450,7 +449,7 @@ media_storage_providers:
Note that uploaded media will also be stored locally and this behavior can not be deactivated, it is even required for
some operations like resizing images.
In fact, your local filesystem is considered as a cache but without any automated way to garbage collect it.
In fact, your local filesysem is considered as a cache but without any automated way to garbage collect it.
We can build our garbage collector with `s3_media_upload`, a tool provided with the module.
If you installed the module with the command provided before, you should be able to bring it in your path:
@ -568,186 +567,13 @@ The module can then be configured with:
Other configuration options can be found in the
[configuration YAML file](https://github.com/processone/ejabberd-contrib/blob/master/mod_s3_upload/conf/mod_s3_upload.yml).
## Ente
Ente is an alternative for Google Photos and Apple Photos. It [can be selfhosted](https://help.ente.io/self-hosting/) and is working fine with Garage as of May 2024.
As a first step we need to create a bucket and a key for Ente:
```bash
garage bucket create ente
garage key create ente-key
# For the CORS setup to work, the key needs to be --owner as well, at least temporarily.
garage bucket allow ente --read --write --owner --key ente-key
```
We also need to setup some CORS rules to allow the Ente frontend to access the bucket:
```bash
export CORS='{"CORSRules":[{"AllowedHeaders":["*"],"AllowedMethods":["GET", "PUT", "POST", "DELETE"],"AllowedOrigins":["*"], "ExposeHeaders":["ETag"]}]}'
aws s3api put-bucket-cors --bucket ente --cors-configuration $CORS
```
Now we need to configure ente-server to use our bucket. This is explained [in the Ente S3 documentation](https://help.ente.io/self-hosting/guides/external-s3).
Prepare a configuration file for ente's backend as `museum.yaml`:
```yaml
credentials-file: /credentials.yaml
apps:
public-albums: https://albums.example.tld # If you want to use the share album feature
internal:
hardcoded-ott:
local-domain-suffix: "@example.com" # Your domain
local-domain-value: 123456 # Custom One-Time Password since we are not sending mail by default
key:
# WARNING -- You MUST CHANGE the values below
# Someone has made an image that can do it for you : https://github.com/EdyTheCow/ente-selfhost/blob/main/images/ente-server-tools/Dockerfile
# Simply build it yourself or run docker run --rm ghcr.io/edythecow/ente-server-tools go run tools/gen-random-keys/main.go
encryption: yvmG/RnzKrbCb9L3mgsmoxXr9H7i2Z4qlbT0mL3ln4w= # CHANGE THIS VALUE
hash: KXYiG07wC7GIgvCSdg+WmyWdXDAn6XKYJtp/wkEU7x573+byBRAYtpTP0wwvi8i/4l37uicX1dVTUzwH3sLZyw== # CHANGE THIS VALUE
jwt:
secret: i2DecQmfGreG6q1vBj5tCokhlN41gcfS2cjOs9Po-u8= # CHANGE THIS VALUE
```
The full configuration file can be found [here](https://github.com/ente-io/ente/blob/main/server/configurations/local.yaml)
Then prepare a credentials file as `credentials.yaml`
```yaml
db:
host: postgres
port: 5432
name: <ente_db_name>
user: <pguser>
password: <pgpass>
s3:
# Override the primary and secondary hot storage. The commented out values
# are the defaults.
#
hot_storage:
primary: b2-eu-cen
# secondary: wasabi-eu-central-2-v3
# If true, enable some workarounds to allow us to use a local minio instance
# for object storage.
#
# 1. Disable SSL.
# 2. Use "path" style S3 URLs (see `use_path_style_urls` below).
# 3. Directly download the file during replication instead of going via the
# Cloudflare worker.
# 4. Do not specify storage classes when uploading objects (since minio does
# not support them, specifically it doesn't support GLACIER).
are_local_buckets: true
# To use "path" style S3 URLs instead of DNS-based bucket access
# default to true if you set "are_local_buckets: true"
# use_path_style_urls: true
b2-eu-cen: # Don't change this key, it is hardcoded
key: <keyID>
secret: <keySecret>
endpoint: garage:3900 # publicly accessible endpoint of your garage instance
region: garage
bucket: <yourbucketName>
use_path_style: true
# you can specify secondary locations, names are hardcoded as well
# wasabi-eu-central-2-v3:
# scw-eu-fr-v3:
# and you can also specify a bucket to be used for embeddings, preview etc..
# default to the first bucket
# derived-storage: wasabi-eu-central-2-derived
```
Finally you can run it with Docker :
```bash
docker run -d --name ente-server --restart unless-stopped -v /path/to/museum.yaml:/museum.yaml -v /path/to/credentials.yaml:/credentials.yaml -p 8080:8080 ghcr.io/ente-io/ente-server
```
For more information on deployment you can check the [ente documentation](https://help.ente.io/self-hosting/)
## Pixelfed
[Pixelfed Technical Documentation > Configuration](https://docs.pixelfed.org/technical-documentation/env.html#filesystem)
## Pleroma
### Creating your bucket
This is the usual Garage setup:
```bash
garage key new --name pleroma-key
garage bucket create pleroma
garage bucket allow pleroma --read --write --owner --key pleroma-key
```
We also need to expose these buckets publicly to serve their content to users:
```bash
garage bucket website --allow pleroma
```
Note the Key ID and Secret Key.
### Configure Pleroma
Update your Pleroma configuration like that in `/etc/pleroma/config.exs`.
```
config :pleroma, Pleroma.Upload,
uploader: Pleroma.Uploaders.S3,
base_url: "https://pleroma.garage.example.tld"
config :ex_aws, :s3,
access_key_id: "GW...",
secret_access_key: "XXX",
region: "garage",
host: "api.garage.example.tld"
```
And restart Pleroma.
You can found more information in [Pleroma Documentation > Pleroma.Uploaders.S3](https://docs-develop.pleroma.social/backend/configuration/cheatsheet/#pleromauploaderss3)
### Migrating your data
Pleroma have an internal migration tool that can encounter some fatal error
```
** (EXIT from #PID<0.98.0>) an exception was raised:
** (File.Error) could not stream "/var/lib/pleroma/uploads/09/f8": illegal operation on a directory
(elixir 1.17.3) lib/file/stream.ex:100: anonymous fn/3 in Enumerable.File.Stream.reduce/3
(elixir 1.17.3) lib/stream.ex:1675: anonymous fn/5 in Stream.resource/3
(elixir 1.17.3) lib/stream.ex:1891: Enumerable.Stream.do_each/4
(elixir 1.17.3) lib/task/supervised.ex:370: Task.Supervised.stream_reduce/7
(elixir 1.17.3) lib/enum.ex:4423: Enum.map/2
(ex_aws_s3 2.5.8) lib/ex_aws/s3/upload.ex:141: ExAws.Operation.ExAws.S3.Upload.perform/2
(pleroma 2.10.0) lib/pleroma/uploaders/s3.ex:60: Pleroma.Uploaders.S3.put_file/1
(pleroma 2.10.0) lib/pleroma/uploaders/uploader.ex:49: Pleroma.Uploaders.Uploader.put_file/2
```
So, use [your best tool](https://garagehq.deuxfleurs.fr/documentation/connect/cli/) to sync `/var/lib/pleroma/uploads/` in your S3.
Then, to avoid some non existent problem (just in case of), run this command
```bash
while true
do
rm -vr $(./bin/pleroma_ctl uploads migrate_local S3 2>&1 | grep "could not stream" | awk -F '"' '{print $2}')
sleep 5
done
```
If you have many files, stop this command sometime and the command bellow (interactive) to delete local
file after upload. Then restart the loop.
```bash
./bin/pleroma_ctl uploads migrate_local S3 --delete
```
And *voilà*
[Pleroma Documentation > Pleroma.Uploaders.S3](https://docs-develop.pleroma.social/backend/configuration/cheatsheet/#pleromauploaderss3)
## Lemmy

View file

@ -207,13 +207,3 @@ $ plakar at @garageS3 ls
```
More information in Plakar documentation: https://www.plakar.io/docs/main/quickstart/
## Synology HyperBackup
HyperBackup can be configured to upload backups to garage using a custom S3 destination. However, the HyperBackup client hardcodes the `us-east-1` region that is a critical input to the v4 signature process. If garage is not set to `us-east-1`, HyperBackup will recognize available buckets, but fail during the final setup stage.
In garage.toml:
```toml
[s3_api]
s3_region = "us-east-1"
```

View file

@ -41,7 +41,7 @@ Some commands:
# list buckets
mc ls garage/
# list objects in a bucket
# list objets in a bucket
mc ls garage/my_files
# copy from your filesystem to garage
@ -149,15 +149,6 @@ rclone help
This will tremendously accelerate operations such as `rclone sync` or `rclone ncdu` by reducing the number
of ListObjects calls that are made.
**Garage behind Cloudflare proxy:** when running Garage behind Cloudflare proxy, you might see `Response: error 403 Forbidden, Forbidden: Invalid signature` error in your garage logs or `AccessDenied: Forbidden: Invalid signature` error in rclone logs. Try adding `--s3-sign-accept-encoding=false` flag to your rclone command and see if the issue is resolved.
```bash
# this throws an error
rclone lsd garage:
# this should work
rclone lsd --s3-sign-accept-encoding=false garage:
```
## `s3cmd`
@ -218,7 +209,7 @@ Within Cyberduck, a
available within the `Preferences -> Profiles` section. This can enabled and
then connections to Garage may be configured.
### Instructions for the CLI
### Instuctions for the CLI
To configure duck (Cyberduck's CLI tool), start by creating its folder hierarchy:
@ -323,3 +314,4 @@ ls
```
And through the web interface at http://[::1]:8080/web/client

View file

@ -201,9 +201,11 @@ on the binary cache, the client will download the result from the cache instead
### Channels
Channels additionally serve Nix definitions, ie. a `.nix` file referencing
Channels additionnaly serve Nix definitions, ie. a `.nix` file referencing
all the derivations you want to serve.
## Gitlab
*External link:* [Gitlab Documentation > Object storage](https://docs.gitlab.com/ee/administration/object_storage.html)

View file

@ -8,12 +8,12 @@ have published Ansible roles. We list them and compare them below.
## Comparison of Ansible roles
| Feature | [ansible-role-garage](#zorun-ansible-role-garage) | [garage-docker-ansible-deploy](#moan0s-garage-docker-ansible-deploy) | [eddster2309 ansible-role-garage](#eddster2309-ansible-role-garage) |
| Feature | [ansible-role-garage](#zorun-ansible-role-garage) | [garage-docker-ansible-deploy](#moan0s-garage-docker-ansible-deploy) | [eddster ansible-role-garage](#eddster-ansible-role-garage) |
|------------------------------------|---------------------------------------------|---------------------------------------------------------------|---------------------------------|
| **Runtime** | Systemd | Docker | Systemd |
| **Target OS** | Any Linux | Any Linux | Any Linux |
| **Architecture** | amd64, arm64, i686 | amd64, arm64 | arm64, arm, 386, amd64 |
| **Additional software** | None | Traefik | Nginx and Keepalived (optional) |
| **Additional software** | None | Traefik | Ngnix and Keepalived (optional) |
| **Automatic node connection** | ❌ | ✅ | ✅ |
| **Layout management** | ❌ | ✅ | ✅ |
| **Manage buckets & keys** | ❌ | ✅ (basic) | ✅ |

View file

@ -29,10 +29,6 @@ it's stable).
Garage is available in the official repositories under [extra](https://archlinux.org/packages/extra/x86_64/garage).
```bash
pacman -S garage
```
## FreeBSD
```bash
@ -44,9 +40,3 @@ pkg install garage
```bash
nix-shell -p garage
```
## conda-forge
```bash
pixi global install garage
```

View file

@ -33,7 +33,7 @@ by adding encryption at different levels.
We would be very curious to know your needs and thougs about ideas such as
encryption practices and things like key management, as we want Garage to be a
serious base platform for the development of secure, encrypted applications.
serious base platform for the developpment of secure, encrypted applications.
Do not hesitate to come talk to us if you have any thoughts or questions on the
subject.
@ -59,7 +59,7 @@ For standard S3 API requests, Garage does not encrypt data at rest by itself.
For the most generic at rest encryption of data, we recommend setting up your
storage partitions on encrypted LUKS devices.
If you are developing your own client software that makes use of S3 storage,
If you are developping your own client software that makes use of S3 storage,
we recommend implementing data encryption directly on the client side and never
transmitting plaintext data to Garage. This makes it easy to use an external
untrusted storage provider if necessary.
@ -108,14 +108,14 @@ Protects against the following threats:
- Stolen HDD
Crucially, does not protect against malicious sysadmins or remote attackers that
Crucially, does not protect againt malicious sysadmins or remote attackers that
might gain access to your servers.
Methods include full-disk encryption with tools such as LUKS.
## Encrypting data on the client side
Protects against the following threats:
Protects againt the following threats:
- A honest-but-curious administrator
- A malicious administrator that tries to corrupt your data

Some files were not shown because too many files have changed in this diff Show more