Compare commits

...
Author SHA1 Message Date
maximilien
99923a8718 Merge pull request 'fix reflected xss when returning errors on web endpoint' (#1472) from escape-errors-v1 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1472
2026-06-11 06:51:01 +00:00
trinity-1686a
0d0068317d fix reflected xss when returning errors on web endpoint
Reported by Filippo Decortes from ares-cyber.ai
2026-06-10 19:43:51 +02:00
Alex Auvolat
b6b18427a5 use optimization level 3 and thin LTO for release builds (#1405)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1405
Co-authored-by: Alex Auvolat <lx@deuxfleurs.fr>
Co-committed-by: Alex Auvolat <lx@deuxfleurs.fr>
2026-04-16 08:47:02 +00:00
Gauthier Zirnhelt
9987166b2b Fix the LifecycleWorker being uncooperative (#1396)
## Summary

This PR ensures that the `LifecycleWorker` yields at least once to the Tokio scheduler in between each batch of 100 objects.

## Problem being solved

I'm administrating a Garage cluster which has been experiencing timeouts on all endpoints while the lifecycle worker is running at midnight UTC : `Ping timeout` error messages and even requests eventually failing due to `Could not reach quorum ...`.

I have found that this happens while the lifecycle worker is working on a big bucket (containing millions of objects) with a lifecycle rule that applies to very few objects.
The `process_object()` function does not hit any `await`:
- `last_bucket` is always the same, so the `bucket_table` is not read asynchronously
- no transaction is made on the `object_table` because my lifecycle rule (almost) never applies to any object

The first commit in this PR adds an executable which reproduces the problem that I've been experiencing in a self-contained way : the lifecycle worker starves the Tokio scheduler so much that no other task is able to run (or very rarely).
To run it : `cargo run -p garage_model --bin lifecycle-starvation-test`.
This commit can be dropped post-review, as it's only useful to demonstrate the starvation.

The error messages completely stopped after adding the extra yield to the nodes of my cluster.
The duration of the lifecycle worker task does not appear to have changed at all from what I can see (looking at the timestamps produced either by the self-contained binary or by each of my nodes with the `Lifecycle worker finished` message).

## Note

An other potential fix would have been to force the `WorkerProcessor` to yield before re-enqueuing a busy task, but this would have affected all Garage workers even though it's only the `LifecycleWorker` being uncooperative.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1396
Reviewed-by: Alex <lx@deuxfleurs.fr>
Co-authored-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
Co-committed-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
2026-04-15 09:56:24 +00:00
trinity-1686a
b72b090a09 fix silent write errors (#1358)
fix #1355

some write errors are not reported when calling write_all. That's notably the case of ENOSPC on small buffers (1MiB).
on ext4, the error is catched when calling flush(). This is hopefully the case on most local filesystems, though afaik this assumption doesn't hold for NFS

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1358
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-committed-by: trinity-1686a <trinity@deuxfleurs.fr>
2026-02-21 07:21:24 +00:00
Armael
8551aefed4 Fix: correctly parse CORS website configuration with no rules (#1320)
When sending a website config with an empty list of CORS rules, garage currently incorrectly refuses it with error message "Invalid XML: missing field `CORSRule`".
This fix the issue by following the documentation of quick-xml related to serde field parameters for this specific scenario:  https://docs.rs/quick-xml/latest/quick_xml/de/#sequences-xsall-and-xssequence-xml-schema-types .

(I've based this PR on main-v1 because we want it for deuxfleurs' deployment.)

Co-authored-by: Armaël Guéneau <armael.gueneau@ens-lyon.org>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1320
Co-authored-by: Armael <armael@noreply.localhost>
Co-committed-by: Armael <armael@noreply.localhost>
2026-02-07 13:11:20 +00:00
Alex Auvolat
47bf5d9fb0 bump version to v1.3.1 2026-01-24 13:01:27 +01:00
Alex Auvolat
5df37dae5e update cargo dependencies in main-v1 (#1299)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1299
Co-authored-by: Alex Auvolat <lx@deuxfleurs.fr>
Co-committed-by: Alex Auvolat <lx@deuxfleurs.fr>
2026-01-24 11:59:01 +00:00
Alex
44af0bdab3 Merge pull request 'Backport #1283 and #1290 to main-v1' (#1297) from backports-v1 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1297
2026-01-24 11:34:28 +00:00
rmoff
a7d6620e18 Fix typo in error message 2026-01-24 12:21:45 +01:00
Joe Anderson
8eb12755e4 Allow bucket to be missing from presigned post params 2026-01-24 12:21:25 +01:00
maximilien
c685a2cbaf Merge pull request 'Update doc/book/cookbook/binary-packages.md' (#1269) from nmstoker/garage:main-v1 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1269
2025-12-21 21:12:15 +00:00
maximilien
969f42a970 Merge pull request 'feat: add service annotations' (#1264) from deimosfr/garage:feat/add_helm_svc_annotations into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1264
Reviewed-by: maximilien <git@mricher.fr>
2025-12-21 21:00:00 +00:00
nmstoker
424d4f8d4d Update doc/book/cookbook/binary-packages.md
Correct the Arch Linux link as garage is now available in the official repos under extra, and no longer in AUR.
2025-12-20 13:16:38 +00:00
Pierre Mavro
bf5290036f
feat: add service annotations 2025-12-18 18:12:22 +01:00
Alex
4efc8bac07 Merge pull request 'Add the parameter, which replaces . This is to accommodate different storage media such as HDD and NVMe.' (#1251) from perrynzhou/garage:dev into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1251
Reviewed-by: Alex <lx@deuxfleurs.fr>
2025-12-17 10:05:49 +00:00
perrynzhou
f3dcc39903 Merge branch 'main-v1' into dev 2025-12-17 10:05:19 +00:00
maximilien
43e02920c2 Merge pull request 'docs: fix typo in doc/book/cookbook/kubernetes.md' (#1259) from simonpasquier/garage:fix-typo into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1259
Reviewed-by: maximilien <me@mricher.fr>
2025-12-17 07:09:59 +00:00
Simon Pasquier
dcc2fe4ac5
docs: fix typo in doc/book/cookbook/kubernetes.md 2025-12-16 10:16:44 +01:00
perrynzhou@gmail.com
e3a5ec6ef6 rename put_blocks_max_parallel to block_max_concurrent_writes_per_request and update configuration.md 2025-12-12 07:09:38 +08:00
perrynzhou@gmail.com
4d124e1c76 Add the parameter, which replaces . This is to accommodate different storage media such as HDD and NVMe. 2025-12-10 06:43:51 +08:00
Alex
d769a7be5d Merge pull request 'Update rust toolchain to 1.91.0' (#1233) from toolchain-update into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1233
2025-11-25 09:58:17 +00:00
Alex Auvolat
511cf0c6ec disable awscli checksumming in ci scripts
required because garage.deuxfleurs.fr is still running v1.x
2025-11-24 18:37:34 +01:00
Alex Auvolat
95693d45b2 run cargo fmt as a nix derivation 2025-11-24 18:09:53 +01:00
Alex Auvolat
ca296477f3 disable checksums in aws cli (todo: revert in main-v2) 2025-11-24 17:58:57 +01:00
Alex Auvolat
ca3b4a050d update nixos image used in woodpecker ci 2025-11-24 17:35:51 +01:00
Alex Auvolat
a057ab23ea Update rust toolchain 2025-11-24 11:09:46 +01:00
Alex
58bc65b9a8 Merge pull request 'migrate to this error, garage-v1' (#1218) from thiserror into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1218
Reviewed-by: Alex <lx@deuxfleurs.fr>
2025-11-12 08:05:32 +00:00
trinity-1686a
ac851d6dee fmt 2025-11-01 18:04:54 +01:00
trinity-1686a
eac2aa6fe4 Merge pull request 'fix: default config path changed for alpine binary' (#1204) from berndsen-io/garage:fix-alpine-docs into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1204
Reviewed-by: trinity-1686a <trinity@deuxfleurs.fr>
2025-11-01 16:43:32 +00:00
trinity-1686a
1e0201ada2 Merge pull request 'Update link to signature v2.' (#1211) from teo-tsirpanis/garage:sigv2-docs into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1211
Reviewed-by: trinity-1686a <trinity@deuxfleurs.fr>
2025-11-01 16:43:05 +00:00
trinity-1686a
82297371bf migrate to this error
it doesn't generate a bazillion warning at compile time
2025-11-01 17:20:39 +01:00
teo-tsirpanis
174f4f01a8 Update link to signature v2. 2025-10-26 15:54:08 +00:00
fgberry
1aac7b4875 chore: spacing 2025-10-24 11:25:33 +02:00
fgberry
b43c58cbe5 fix: default config path changed for alpine binary 2025-10-24 11:22:32 +02:00
Alex
9481ac428e Merge pull request 'sigv4: don't enforce x-amz-content-sha256 to be in signed headers list (fix #770)' (#1195) from fix-770 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1195
2025-10-14 09:34:35 +00:00
Alex Auvolat
1c29d04cc5 sigv4: don't enforce x-amz-content-sha256 to be in signed headers list (fix #770)
From the following page:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

> In both cases, because the x-amz-content-sha256 header value is already
> part of your HashedPayload, you are not required to include the
> x-amz-content-sha256 header as a canonical header.
2025-10-14 11:18:25 +02:00
Alex
b48a8eaa1f Merge pull request 'properly handle precondition time equal to object time' (#1193) from precondition-ms into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1193
2025-10-10 19:41:06 +00:00
trinity-1686a
42fd8583bd properly handle precondition time equal to object time 2025-10-08 17:54:22 +02:00
Alex
236af3a958 Merge pull request 'Garage v1.3.0' (#1166) from rel-v1.3.0 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1166
2025-09-14 21:26:21 +00:00
Alex Auvolat
4b1fdbef55 bump version to v1.3.0 2025-09-14 21:36:33 +02:00
Alex Auvolat
0f1b488be0 fix rust warnings 2025-09-14 21:25:37 +02:00
Alex
0bbf63ee0e Merge pull request 'update rusqlite and snapshot using VACUUM INTO' (#1164) from update-rusqlite into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1164
2025-09-14 18:28:01 +00:00
Alex
879d941d7b Merge pull request 'add garage repair clear-resync-queue (fix #1151)' (#1165) from clear-resync-queue into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1165
2025-09-14 17:50:41 +00:00
Alex Auvolat
d726cf0299 add garage repair clear-resync-queue (fix #1151) 2025-09-14 19:34:44 +02:00
Alex
0c7aeab6f8 Merge pull request 'garage_db: fix error handling logic (fix #1138)' (#1163) from fix-1138 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1163
2025-09-14 17:26:08 +00:00
Alex Auvolat
5687fc0375 update rusqlite and snapshot using VACUUM INTO 2025-09-14 19:22:36 +02:00
Alex
97f1e9ab52 Merge pull request 'Add Plakar documentation (backup tools)' (#1119) from Lapineige/garage:Plakar_support into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1119
2025-09-14 16:08:36 +00:00
Lapineige
60b1d78b56 Add Plakar documentation 2025-09-14 18:07:49 +02:00
Alex Auvolat
4c895a7186 garage_db: fix error handling logic (fix #1138) 2025-09-14 18:03:31 +02:00
Alex
c3b5cbf212 Merge pull request 'fix panic when cluster_layout cannot be saved (fix #1150)' (#1158) from fix-1150 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1158
2025-09-13 15:58:52 +00:00
Alex
57a467b5c0 Merge pull request 'Block manager: limit simultaneous block reads from disk' (#1157) from block-max-simultaneous-reads into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1157
2025-09-13 15:53:24 +00:00
Alex Auvolat
6cf6db5c61 fix panic when cluster_layout cannot be saved (fix #1150) 2025-09-13 17:49:25 +02:00
Alex Auvolat
d5a57e3e13 block: read_block: don't add not found blocks to resync queue 2025-09-13 17:38:23 +02:00
Alex Auvolat
5cf354acb4 block: maximum number of simultaneous reads 2025-09-13 17:38:06 +02:00
Alex
2b007ddea3 Merge pull request 'woodpecker: require the nix=enabled label' (#1152) from woodpecker-nix-flag into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1152
2025-09-04 09:10:10 +00:00
Alex Auvolat
c8599a8636 woodpecker: require the nix=enabled label 2025-09-04 11:06:46 +02:00
Alex
0b901bf291 Merge pull request 'garage_db: reduce frequency of sqlite snapshot progress log (fix #1129)' (#1146) from fix-1129 into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1146
2025-08-27 22:26:32 +00:00
Alex Auvolat
c8c20d6f47 garage_db: reduce frequency of sqlite snapshot progress log (fix #1129) 2025-08-28 00:07:35 +02:00
Alex
e5db610e4c Merge pull request 'K2V client: allow custom HTTP client' (#731) from k2v/shared_http_client into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/731
Reviewed-by: maximilien <me@mricher.fr>
2025-08-27 21:21:09 +00:00
Alex
65c6f8adea Merge pull request 'garage_db: refactor open function' (#1142) from factor-db-open into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1142
2025-08-27 21:10:59 +00:00
Alex Auvolat
54b9bf02a3 garage_db: refactor open function 2025-08-27 23:03:09 +02:00
Alex
469153233f Merge pull request 'garage_db: rename len to approximate_len as it is used for stats only' (#1141) from db-approximate-len into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1141
2025-08-27 20:44:50 +00:00
Alex Auvolat
90bba5889a garage_db: rename len to approximate_len as it is used for stats only 2025-08-27 21:23:45 +02:00
Alex
a64b567d43 Merge pull request 'Add experimental support for Fjall DB engine' (#906) from withings/garage:feat/fjall-db-engine into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/906
2025-08-27 19:09:40 +00:00
Alex Auvolat
6ea86db8cd document fjall db engine, remove flakey metadata_fsync implementation 2025-08-27 20:22:41 +02:00
Alex Auvolat
aa69c06f2b fix potential race condition and naming bug in fjall adapter 2025-08-27 20:22:38 +02:00
Alex Auvolat
a6c6c44310 nix: build and test fjall feature 2025-08-27 18:54:42 +02:00
Julien Kritter
96d7713915 Add support for an LSM-tree-based backend with Fjall 2025-08-27 18:54:34 +02:00
Alex
d64498c3d3 Merge pull request 'log access keys' (#1122) from 1686a/log-access-key into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1122
2025-08-27 16:18:16 +00:00
trinity-1686a
b340599e68 log access keys 2025-08-03 15:30:56 +02:00
Alex
5448012b27 Merge pull request 'Pixelfed_support' (#1118) from Lapineige/garage:Pixelfed_support into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1118
2025-08-02 15:03:57 +00:00
Alex
ce34d11a65 Merge pull request 'don't die on SIGHUP' (#1121) from 1686a/handle-sighup into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1121
2025-08-02 14:53:58 +00:00
Alex
8cb7623ebd Merge pull request 'handle ECONNABORTED' (#1120) from 1686a/handle-econnaborted into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1120
2025-08-02 14:53:45 +00:00
trinity-1686a
5469c95877 handle ECONNABORTED 2025-08-02 13:14:01 +02:00
trinity-1686a
f930c6f643 don't die on SIGHUP 2025-08-02 13:09:33 +02:00
Alex
afcb22bf16 Merge pull request 'Fix typo in peertube buckets names' (#1117) from Lapineige/garage:main into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1117
2025-08-02 08:27:01 +00:00
Lapineige
cc29a40d51 Actualiser doc/book/connect/apps/index.md 2025-08-01 21:35:15 +00:00
Lapineige
0f3f180c3e Merge branch 'main-v1' into main 2025-08-01 21:33:58 +00:00
Lapineige
70cf6004ae Fix typo in peertube buckets names 2025-08-01 21:32:59 +00:00
Alex
c7571ff89b Merge pull request 'Fix some unsoundness in lmdb adapter unsafe' (#1099) from krtab/garage:fix_some_ub into main-v1
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1099
2025-07-31 19:38:23 +00:00
Arthur Carcano
1b42919bf7 Fix some unsoundness in lmdb adapter unsafe 2025-07-25 23:33:51 +02:00
Alex
3f4ab3a4a3 Merge pull request 'Garage v1.2.0' (#1068) from rel-1.2.0 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1068
2025-06-13 16:12:29 +00:00
Alex Auvolat
3a4afc04a9 cargo: update crossbeam-channel to avoid yanked version 2025-06-13 17:22:47 +02:00
Alex Auvolat
fbf03e9378 bump version to v1.2.0 2025-06-13 14:21:28 +02:00
Alex
9eb07d4c7b Merge pull request 'cli: mark block refs as deleted in garage block purge (fix #1055)' (#1067) from fix-1055 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1067
2025-06-13 11:53:41 +00:00
Alex Auvolat
85ee4f5d8c cli: mark block refs as deleted in garage block purge 2025-06-13 13:52:02 +02:00
Alex
328072d122 Merge pull request 'put web error in a basic webpage' (#1064) from trinity-1686a/garage:1686a/non-xml-web-error into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1064
2025-06-12 06:06:38 +00:00
trinity-1686a
26bc807905 put web error in a basic webpage
before, it was a plain string, with an xml content type

this caused browsers to show very ugly and meaningless pages
2025-06-10 22:23:06 +02:00
Alex
a9f5f242b2 Merge pull request 'feat: add log to journald feature' (#1056) from ragazenta/garage:feat/tracing-journald into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1056
2025-06-10 18:38:23 +00:00
maximilien
ae98abca5c Merge pull request 'Add eddster2309/ansible-role-garage as deployment option' (#1057) from eddster2309/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1057
Reviewed-by: maximilien <me@mricher.fr>
2025-06-08 11:56:31 +00:00
eddster2309
adfa44ad70 Add architecture support 2025-06-03 09:22:43 +00:00
eddster2309
47143b88ad Add eddster2309/ansible-role-garage as deployment option 2025-06-03 09:15:57 +00:00
Renjaya Raga Zenta
8843aa92fa
feat: add log to journald feature
The systemd-journald is used in most major Linux distros that use systemd.
This enables logging using the systemd-journald native protocol, instead
of just writing to stderr.
2025-06-02 11:55:27 +07:00
Alex
b601b3e46d Merge pull request 'documentation: Minor doc change to clarify why the capacity does not matter and how the zone name is used' (#1051) from ddxv/garage:docs-quick-start into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1051
2025-05-30 16:26:19 +00:00
Alex
a19d2f16e2 Merge pull request 'api: s3: implement get bucket acl' (#1045) from ragazenta/garage:feat/dummy-acl into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1045
2025-05-30 16:25:04 +00:00
trinity-1686a
fc8fc60f6d emit internal error when we detect race condition (#1053) (fix #1050)
i went with a `500`/`InternalError`/`Please try again.` because that is something i've seen AWS S3 report while developing other software, and i'm not convinced all clients would understand a 409 conflict properly (GET don't usually conflict)

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1053
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-committed-by: trinity-1686a <trinity@deuxfleurs.fr>
2025-05-30 16:24:12 +00:00
Alex
77079a1498 Merge pull request '[1.1.x] speed up UploadPartCopy' (#1047) from yuka/garage:uploadpartcopy-v1 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1047
2025-05-30 16:22:35 +00:00
James O'Claire
2a4f729b57 Minor doc change to clarify why the capacity does not matter and how the zone name is used 2025-05-28 09:49:50 +08:00
Renjaya Raga Zenta
1b042e379e
api: s3: implement get bucket acl 2025-05-26 09:43:15 +07:00
Yureka
ffbce0f689 speed up UploadPartCopy
(cherry picked from commit db54bf96c7)
2025-05-23 20:36:32 +02:00
Alex
37e5621dde Merge pull request 'documentation updates' (#1046) from doc-updates into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1046
2025-05-23 15:05:19 +00:00
Alex Auvolat
6529ff379a documentation updates 2025-05-23 17:02:23 +02:00
Alex
a8d73682a4 Merge pull request 'more resilience to inconsistent alias states' (#989) from fix-bucket-aliases into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/989
2025-05-22 17:41:42 +00:00
Alex Auvolat
8654eb19bf implement repair procedure to fix inconsistent bucket aliases 2025-05-22 19:34:38 +02:00
maximilien
54ea412188 Merge pull request 'Add kubernetes CRD' (#994) from babykart/garage:k8s-crd into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/994
2025-05-22 17:15:56 +00:00
Alex Auvolat
2ade8c86f6 more resilience to inconsistent alias states 2025-05-22 19:12:05 +02:00
babykart
b15e2cbb6c Update Kubernetes cookbook
Signed-off-by: babykart <babykart@gmail.com>
2025-05-22 17:11:14 +00:00
babykart
0fd1b7342b Add Kubernetes CRD and the related kustomization
Signed-off-by: babykart <babykart@gmail.com>
2025-05-22 17:11:14 +00:00
Alex
be16bc7a05 Merge pull request 'Fix behavior of CopyObject wrt x-amz-website-redirect-location' (#1037) from Armael/garage:copy-website-redirect into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1037
2025-05-22 13:57:28 +00:00
Alex
bfaa1ca6b7 Merge pull request 'api: lifecycle: 404 if missing lifecycle config' (#1043) from ragazenta/garage:no-lifecycle-response into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1043
2025-05-22 13:56:52 +00:00
Alex
de8eeab4ad Merge pull request 'optionally support puny code (fix #273)' (#1042) from trinity-1686a/garage:1686a/punnycode into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1042
2025-05-22 12:49:10 +00:00
Renjaya Raga Zenta
ae3f7ee76c
api: lifecycle: 404 if missing lifecycle config 2025-05-22 19:33:54 +07:00
trinity-1686a
2dc3a6dbbe document allow_punycode configuration option 2025-05-22 14:08:06 +02:00
Armaël Guéneau
c6bc3f229b Fix behavior of CopyObject wrt x-amz-website-redirect-location 2025-05-22 14:03:11 +02:00
trinity-1686a
bba9202f31 add test for punycode 2025-05-19 20:36:03 +02:00
trinity-1686a
a605a80806 support punnycode in api/web endpoint 2025-05-19 18:11:55 +02:00
trinity-1686a
539af12d21 allow punnycode in bucket name 2025-05-19 18:07:04 +02:00
Alex
a2a9e3cec4 Merge pull request 'doc: Add systemd example to increase file descriptors limit' (#1023) from baptiste/garage:systemd_openfiles into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1023
2025-05-09 10:34:07 +00:00
Baptiste Jonglez
14274bc13c doc: Add systemd example to increase file descriptors limit 2025-05-08 10:27:53 +02:00
Alex
bf4691d98a Merge pull request 'Fix #1007: hint that region can be changed depending on cluster config' (#1015) from garage-1007-update-region-in-doc into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1015
2025-04-24 07:41:32 +00:00
Maximilien R.
ad151cb1dc Fix #1007: hint that region can be changed depending on cluster config 2025-04-23 23:30:16 +02:00
babykart
3c20984a08 helm-chart: Cosmetic changes
Signed-off-by: babykart <babykart@gmail.com>
2025-04-21 10:04:53 +00:00
babykart
e6e4e051a1 helm-chart: Add metadata_auto_snapshot_interval
Signed-off-by: babykart <babykart@gmail.com>
2025-04-21 10:04:53 +00:00
babykart
9b38cba6f3 helm-chart: Add livenessProbe & readinessProbe
Signed-off-by: babykart <babykart@gmail.com>
2025-04-21 10:04:53 +00:00
Alex
4ef954d176 Merge pull request 'Fix Docker run volume mappings' (#1012) from Zoob/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1012
2025-04-19 20:05:16 +00:00
Zoob
02498a93d0 doc: fix Docker run volume mappings 2025-04-19 18:46:36 +00:00
Alex
4caad5425d Merge pull request 'metadata: Create compact LMDB snapshots' (#1008) from baptiste/garage:lmdb_compact_snapshot into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1008
2025-04-17 19:26:32 +00:00
Baptiste Jonglez
9ec3f8cc3c metadata: Create compact LMDB snapshots
See #1006

LMDB files never shrink, so we can end up with a large database that
contains a smaller amount of actual data.

Compacting the snapshots is an easy win: it will write faster to disk,
take less space, and if needed you can reimport an already-compacted
snapshot as the main database.
2025-04-12 23:18:50 +02:00
Alex
14d2f2b18d Merge pull request 'update cargo dependencies' (#992) from update-deps into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/992
2025-03-21 09:06:06 +00:00
Alex Auvolat
a7d845a999 change aws-sdk features to avoid using aws-lc which doesn't compile on i686/arm 2025-03-20 17:05:43 +01:00
Alex Auvolat
dd20e5d22a update cargo dependencies 2025-03-20 13:36:01 +01:00
maximilien
6906a4ff12 Merge pull request 'doc: add instructions on how to increase PVC size' (#987) from Joker9944/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/987
Reviewed-by: maximilien <me@mricher.fr>
2025-03-17 20:32:31 +00:00
Joker9944
9053782d71
doc: add instructions on how to increase PVC size 2025-03-15 00:32:18 +01:00
Alex
c96be1a9a8 Merge pull request 'doc/upgrading: slightly more precise wording' (#981) from Armael/garage:doc-upgrading into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/981
2025-03-07 15:16:24 +00:00
maximilien
98e56490a1 Merge pull request 'helm-chart: Fix headless service' (#976) from babykart/garage:headless-svc into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/976
2025-03-07 12:17:20 +00:00
Armaël Guéneau
e791ccec8f doc/upgrading: slightly more precise wording 2025-03-07 12:27:21 +01:00
maximilien
d605c4fed1 Explicitely set ClusterIP on headless service type
Signed-off-by: maximilien <maximilien@deuxfleurs.fr>
2025-03-07 09:17:05 +00:00
babykart
0ce5f7eb00
helm-chart: Fix headless service
Signed-off-by: babykart <babykart@gmail.com>
2025-03-05 20:26:12 +01:00
Alex
516255321f Merge pull request 'doc: fix version number in quick start' (#974) from fix-quickstart into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/974
2025-03-05 11:07:27 +00:00
Alex Auvolat
f3b05ff771 doc: fix version number in quick start 2025-03-05 12:06:05 +01:00
Alex
e254cc20e5 Merge pull request 'Garage v1.1.0' (#968) from rel-1.1 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/968
2025-03-05 10:56:34 +00:00
Alex Auvolat
12f15c4c2b fix readme paths in cargo.toml for new crates 2025-03-05 11:00:19 +01:00
Alex Auvolat
42c5d02cdf doc: fix "since vX.X.X" in multiple places 2025-03-05 10:19:51 +01:00
Alex Auvolat
4689b10448 bump version to v1.1.0 2025-03-05 10:19:51 +01:00
Alex
156b10ee65 Merge pull request 'admin api definition: fix globalAlias query parameter name (related: #971)' (#973) from admin-sdk-fix into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/973
2025-03-05 09:19:30 +00:00
Alex Auvolat
8647ebf003 admin api definition: fix globalAlias query parameter name (related: #971) 2025-03-05 10:16:36 +01:00
maximilien
67d7c0769b Merge pull request 'Add headless service for statefulSet serviceName' (#970) from babykart/garage:helm-headless-svc into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/970
Reviewed-by: maximilien <me@mricher.fr>
2025-03-05 08:59:36 +00:00
babykart
09ed5ab8cc
Fix documentation link
Signed-off-by: babykart <babykart@gmail.com>
2025-02-23 15:55:01 +01:00
babykart
a0ea28b0da
Add headless service for statefulSet serviceName
Signed-off-by: babykart <babykart@gmail.com>
2025-02-23 15:45:55 +01:00
Alex
c5237c31e7 Merge pull request 'Implement all HTTP preconditions in GetObject/HeadObject' (#967) from fix-804 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/967
2025-02-19 17:31:26 +00:00
Alex Auvolat
f87943a39d tests: add test for http preconditions 2025-02-19 18:26:03 +01:00
Alex Auvolat
c0846c56fe api: unify http precondition handling 2025-02-19 18:14:27 +01:00
Alex
1cb0ae10a8 Merge pull request 'fix crash in layout computation when changing all nodes of a zone to gateway mode' (#937) from baptiste/garage:fix_layout_crash into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/937
Reviewed-by: Alex <lx@deuxfleurs.fr>
2025-02-19 17:09:10 +00:00
Alex Auvolat
1a8f74fc94 api: GetObject: implement if-match and if-unmodified-since 2025-02-19 17:26:29 +01:00
Alex
2191620af5 Merge pull request 'web: implement x-amz-website-redirect-location' (#966) from redirect-location-header into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/966
2025-02-19 16:10:04 +00:00
Alex Auvolat
bf27a3ec98 web: implement x-amz-website-redirect-location 2025-02-19 17:04:10 +01:00
Alex
f64ec6e542 Merge pull request 'implement STREAMING-*-PAYLOAD-TRAILER' (#960) from fix-824 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/960
2025-02-19 09:59:32 +00:00
Alex Auvolat
6d38907dac test: verify saved checksums in streaming putobject tests 2025-02-18 22:02:04 +01:00
Alex Auvolat
cfe8e8d45c api: PutObject: save trailer checksum in metadata 2025-02-18 21:56:32 +01:00
Alex Auvolat
f6e805e7db api: various fixes 2025-02-18 21:47:53 +01:00
Alex Auvolat
45e10e55f9 update aws-sdk-s3 in tests and fix wrong checksumming behavior in GetObject 2025-02-18 15:33:42 +01:00
Alex Auvolat
730bfee753 api: validate trailing checksum + add test for unsigned-paylad-trailer 2025-02-18 15:33:42 +01:00
Alex Auvolat
ccab0e4ae5 api: fix optional \n after trailer checksum header 2025-02-18 15:33:42 +01:00
Alex Auvolat
abb60dcf7e api: remove content-encoding: aws-chunked for streaming payload 2025-02-18 15:33:42 +01:00
Alex Auvolat
f8b0817ddc api: streaming signature: fix trailer parsing 2025-02-18 12:00:41 +01:00
Alex Auvolat
21c0dda16a api: refactor: move checksumming code around again 2025-02-17 20:11:06 +01:00
Alex Auvolat
658541d812 api: use checksumming in api_common::signature for put/putpart 2025-02-17 19:54:25 +01:00
Alex Auvolat
c5df820e2c api: start refactor of signature to calculate checksums earlier 2025-02-17 18:47:06 +01:00
Alex Auvolat
a04d6cd5b8 api: streaming: parse unsigned streaming bodies and payload trailers 2025-02-17 16:23:24 +01:00
Alex Auvolat
44a896f9b5 api: add logic to parse x-amz-content-sha256 2025-02-16 18:25:35 +01:00
Alex Auvolat
cee7560fc1 api: refactor: move checksum algorithms to common 2025-02-16 17:25:55 +01:00
Alex Auvolat
2f0c5ca220 signature: refactor: move constant defs to mod.rs 2025-02-16 16:34:18 +01:00
Alex
859b38b0d2 Merge pull request 'fix compilation warnings' (#959) from fixes into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/959
2025-02-14 17:32:30 +00:00
Alex Auvolat
2729a71d9d fix warning in garage test 2025-02-14 18:27:00 +01:00
Alex Auvolat
c9d00f5f7b garage_api_s3: remove unused field in ListPartsQuery 2025-02-14 18:25:23 +01:00
Alex
89c944ebd6 Merge pull request 's3api: return Location in CompleteMultipartUpload (fix #852)' (#958) from fix-852 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/958
2025-02-14 17:16:58 +00:00
Alex Auvolat
24470377c9 garage_model: fix warning about dead code 2025-02-14 18:12:14 +01:00
Alex Auvolat
5b26545abf fix deprecated uses of chrono in lifecycle worker 2025-02-14 18:08:23 +01:00
Alex Auvolat
9c7e3c7bde remove cargo build options in makefile to avoid mistakes 2025-02-14 18:06:07 +01:00
Alex Auvolat
165f9316e2 s3api: return Location in CompleteMultipartUpload (fix #852)
NB. The location returned is not guaranteed to work in all cases.
This already fixes the parse issue in #852.
2025-02-14 18:05:07 +01:00
Alex
a94adf804f Merge pull request 'block manager: avoid deadlock in fix_block_location (fix #845)' (#957) from fix-845 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/957
2025-02-14 16:53:01 +00:00
Alex Auvolat
e4c9a8cd53 block manager: avoid deadlock in fix_block_location (fix #845) 2025-02-14 17:41:50 +01:00
Alex
9312c6bbcb Merge pull request 'Store data blocks only on nodes in the latest cluster layout version (fix #815)' (#956) from fix-815 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/956
2025-02-14 15:53:16 +00:00
Alex Auvolat
fdf4dad728 block resync: avoid saving blocks to draining nodes 2025-02-14 16:45:55 +01:00
Alex Auvolat
6820b69f30 block manager: improve read strategy to find blocks faster 2025-02-14 16:45:55 +01:00
Alex Auvolat
d0104b9f9b block manager: write blocks only to currently active layout version (fix #815)
avoid wastefully writing blocks to nodes that will discard them as soon
as the layout migration is finished
2025-02-14 16:45:55 +01:00
Alex
3fe8db9e52 Merge pull request 'web_server.rs: Added bucket domain to observability' (#608) from jpds/garage:domain-web-requests into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/608
2025-02-14 14:26:08 +00:00
Alex
627a37fe9f Merge pull request 's3 api: parse x-id query parameter and warn of any inconsistency (fix #822)' (#954) from fix-822 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/954
2025-02-14 14:07:01 +00:00
Alex Auvolat
2f55889835 add configuration option to enable/disable monitoring bucket in web metrics 2025-02-14 14:59:00 +01:00
Jonathan Davies
8b9cc5ca3f web_server.rs: Added bucket domain to observability. 2025-02-14 14:36:20 +01:00
Alex
a1533d2919 Merge pull request 'cli: return info of all nodes when doing garage stats -a (fix #814)' (#953) from fix-814 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/953
2025-02-14 13:31:42 +00:00
Alex Auvolat
c1b39d9ba1 s3 api: parse x-id query parameter and warn of any inconsistency (fix #822) 2025-02-14 14:30:58 +01:00
Alex Auvolat
d84308c413 cli: return info of all nodes when doing garage stats -a (fix #814) 2025-02-14 14:11:41 +01:00
Alex
63f20bdeab Merge pull request 'db-snapshot: Add error handling to metadata snapshot creation' (#930) from handle_snapshot_errors into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/930
Reviewed-by: Armael <armael@noreply.localhost>
2025-02-14 11:52:58 +00:00
Baptiste Jonglez
a2e134f036 db-snapshot: propagate any node snapshot error through RPC call
In particular, it means that "garage meta snapshot --all" will get an exit
code of 1 if any node fails to snapshot.

This makes sure that any external tool trying to snapshot nodes (e.g. from
cron) will be aware of the failure.

Fix #920
2025-02-07 00:29:43 +01:00
Baptiste Jonglez
06aa4b604f db-snapshot: Fix error reporting when using "garage meta snapshot --all"
Snapshot errors on remote nodes were not reported at all.

We now get proper error output such as:

    0fa0f35be69528ab  error: Internal error: DB error: LMDB: No space left on device (os error 28)
    88d92e2971d14bae  ok

Fix #920
2025-02-07 00:18:01 +01:00
Alex
d3226bfa91 Merge pull request 'remove uses of #[async_trait]' (#952) from remove-async-trait into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/952
2025-02-05 19:52:00 +00:00
Alex Auvolat
af67626ab2 remove async_trait for TableRepair 2025-02-05 20:45:07 +01:00
Alex Auvolat
5475da8ea8 remove async_trait used in generic_server.rs 2025-02-05 20:31:34 +01:00
Alex Auvolat
620dc58560 remove async_trait for traits declared in garage_net 2025-02-05 20:22:16 +01:00
Alex
47e87c8739 Merge pull request 'upgrade Rust compiler and Cargo dependencies' (#951) from nix-crane into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/951
2025-02-03 17:49:00 +00:00
Alex Auvolat
34599bff51 update all Cargo dependencies except AWS crates and their dependencies 2025-02-03 17:46:54 +01:00
Alex Auvolat
ec1a475923 build with rust 1.82.0 2025-02-03 17:46:48 +01:00
Alex
b9df2d1ad1 Merge pull request 'compile with crane' (#950) from nix-crane into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/950
2025-02-03 15:54:54 +00:00
Alex Auvolat
390a5d97fe nix, ci: build with Crane
This removes our dependency on cargo2nix, which was causing us some
issues. Whereas cargo2nix creates one Nix derivation per crate, Crane
uses only two derivations:

1. Build dependencies only
2. Build the final binary

This means that during the second step, no caching can be done. For
instance, if we do a change in garage_model, we need to recompile all of
the Garage crates including those that do not depend on garage_model.
On the upside, this allows all of the Garage crates to be built at once
using cargo build logic, which is optimized for high parallelism and
better pipelining between all of the steps of the build. All in all,
this makes most builds faster than cargo2nix.

A few other changes have been made to the build scripts and CI:

- Unit tests are now run within a Nix derivation. In fact, we have
  different derivations to run the tests using LMDB and Sqlite as
  metadata db engines.

- For debug builds, most CI steps now run in parallel (with the notable
  exception of the smoke test that runs after the build, which is
  inevitable).

- We no longer pass the GIT_VERSION argument when building debug builds
  and running the tests. This means that dev binaries and test
  binaries don't know the exact version of Garage they are from. That
  shouldn't be an issue in most cases.

- The not-dynamic.sh scripts has been fixed to fail if the file does not
  exist.
2025-02-03 16:39:50 +01:00
Alex
4dc2bc337f Merge pull request 'woodpecker: use parallel nix-build in debug builds' (#949) from nix-parallel into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/949
2025-02-01 18:58:15 +00:00
Alex Auvolat
5dd2791981 woodpecker: use parallel nix-build in debug builds 2025-02-01 19:48:01 +01:00
Alex
d601f31186 Merge pull request 'split garage_api in garage_api_{common,s3,k2v,admin}' (#947) from split-garage-api into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/947
2025-02-01 17:48:25 +00:00
Alex Auvolat
e4de7bdfd5 fix ci for more test crates 2025-01-31 19:21:36 +01:00
Alex Auvolat
d18c5ad0ff fix tests 2025-01-31 19:12:51 +01:00
Alex Auvolat
3d5e9a027e cargo defs: simplify and fix descriptions 2025-01-31 18:54:29 +01:00
Alex Auvolat
f4ca7758b4 update cargo.nix 2025-01-31 18:48:07 +01:00
Alex Auvolat
4563313f87 use cargo-shear to remove many unused dependencies between crates 2025-01-31 18:47:30 +01:00
Alex Auvolat
afa28706e5 split s3/cors.rs into also common/cors.rs 2025-01-31 18:42:14 +01:00
Alex Auvolat
84f1db91c4 fix things up 2025-01-31 18:34:57 +01:00
Alex Auvolat
9fa20d45be wip: split garage_api into garage_api_{common,s3,k2v,admin} 2025-01-31 18:18:29 +01:00
Alex
9330fd79d3 Merge pull request 'table::insert_many: avoid failure with zero items (fix #915)' (#946) from fix-915 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/946
2025-01-31 13:10:54 +00:00
Alex Auvolat
83f6928ff7 table::insert_many: avoid failure with zero items (fix #915) 2025-01-30 18:06:47 +01:00
Alex
ab71544499 Merge pull request 'api: better handling of helper errors to distinguish error codes' (#942) from fix-getkeyinfo-404 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/942
2025-01-29 18:25:44 +00:00
Alex
991edbe02c Merge pull request 'Update doc/book/connect/repositories.md' (#941) from yatesco/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/941
2025-01-29 18:18:59 +00:00
Alex Auvolat
9f3c7c3720 api: better handling of helper errors to distinguish error codes 2025-01-29 19:14:34 +01:00
yatesco
bfde9152b8 Update doc/book/operations/multi-hdd.md
trivial spelling mistake
2025-01-29 13:40:41 +00:00
yatesco
7bb042f0b7 Update doc/book/connect/repositories.md
trivial spelling mistake
2025-01-29 13:34:35 +00:00
Alex
a1d081ee84 Merge pull request 's3 api: make x-amz-meta-* headers lowercase (fix #844)' (#938) from fix-844 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/938
2025-01-27 19:32:19 +00:00
Alex Auvolat
e8fa89e834 s3 api: make x-amz-meta-* headers lowercase (fix #844) 2025-01-27 19:58:06 +01:00
Alex
beedc9fd11 Merge pull request 'snapshot: sqlite: use a subdirectory for consistency with LMDB' (#932) from baptiste/garage:snapshot_consistency_sqlite into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/932
2025-01-27 18:50:11 +00:00
Baptiste Jonglez
6d798c640f WIP: fix crash in layout computation when changing all nodes of a zone to gateway mode
This change is probably not a proper fix, somebody with more expertise on
this code should look at it.

Here is how to reproduce the crash:

- start with a layout with two zones
- move all nodes of a zone to gateway mode: `garage layout assign fea54bcc081f318 -g`
- `garage layout show` will panic with a backtrace

Fortunately, the crash is only on the RPC client side, not on the Garage
server itself, and `garage layout revert` still works to go back to the
previous state.

As far as I can tell, this bug is present since Garage 0.9.0 which
includes the new layout assignation algorithm:

  https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/296
2025-01-27 19:33:57 +01:00
Alex
d4e3e60920 Merge pull request 'update nix crate to 0.29 and libc to 0.2.169' (#931) from neuschaefer/garage:nix into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/931
2025-01-27 18:09:51 +00:00
Baptiste Jonglez
43402c9619 snapshot: sqlite: use a subdirectory for consistency with LMDB
Currently, taking a snapshot of the metadata database with sqlite creates
a sqlite file without extension with the following format:

    snapshots/2025-01-26T15:29:17Z

This makes it hard to understand what kind of data this is, and is not
consistent with LMDB:

    snapshots/2025-01-26T15:29:17Z/data.mdb

With this change, we now get a directory with a single db.sqlite file:

    snapshots/2025-01-26T15:29:17Z/db.sqlite
2025-01-27 19:06:52 +01:00
Alex
efa6f3d85e Merge pull request 'db-snapshot: allow to set directory where snapshots are stored' (#933) from baptiste/garage:configure_metadata_snapshots_dir into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/933
2025-01-27 18:04:05 +00:00
Alex Auvolat
74a1b49b13 Update Cargo.nix 2025-01-27 18:37:05 +01:00
J. Neuschäfer
23d57b89dc update nix crate to 0.29 and libc to 0.2.169 2025-01-27 18:37:05 +01:00
Alex
5e3e1f4453 Merge pull request 'fix problems with CI doing work multiple times' (#936) from woodpecker-simplify into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/936
2025-01-27 17:36:27 +00:00
Baptiste Jonglez
59c153d280 db-snapshot: allow to set directory where snapshots are stored
Fix #926
2025-01-27 18:33:55 +01:00
Alex Auvolat
bb3e0f7d22 nix CI: reduce redundant work 2025-01-27 18:09:51 +01:00
Alex
0156e40c9d Merge pull request 'ci: fix woodpecker definitions to comply with woodpecker 3' (#935) from woodpecker3 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/935
Reviewed-by: maximilien <me@mricher.fr>
2025-01-27 12:03:46 +00:00
Alex Auvolat
f6f88065ad ci: fix woodpecker definitions to comply with woodpecker 3 2025-01-27 12:06:31 +01:00
Alex
591bd808ec Merge pull request 'doc: Fix Nix devenv setup' (#927) from fix_devenv into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/927
2025-01-23 10:20:04 +00:00
maximilien
294cb99409 Merge pull request 'Fix all typos' (#928) from majst01/garage:fix-typos into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/928
Reviewed-by: maximilien <me@mricher.fr>
2025-01-17 06:06:14 +00:00
Stefan Majer
2eb9fcae20 Fix all typos 2025-01-16 13:22:00 +01:00
Baptiste Jonglez
58b9eb46fc doc: Fix Nix devenv setup
This is a hotfix to fix the doc for the current setup, see #868 for
possible future directions.
2025-01-16 10:00:12 +01:00
maximilien
255b01b626 Merge pull request 'Helm chart: Add garage.existingConfigmap and replace garage.garage.toml with garage.garageTomlString' (#923) from jessebot/garage:allow-existing-configmap into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/923
Reviewed-by: maximilien <me@mricher.fr>
2025-01-15 23:53:25 +00:00
Maximilien R.
58a765c51f Minor rewording, add some more hints 2025-01-15 23:51:07 +00:00
jessebot
1c431b8457 Add garage.existingConfigmap and replace garage.garage.toml with garage.garageTomlString
also moves all gotemplating back to configmap

and adds autogenerated docs via helm-docs

Signed-off-by: jessebot <jessebot@linux.com>
2025-01-15 23:51:07 +00:00
Alex
39ac034de5 Merge pull request 'update toolchain' (#924) from nix-update into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/924
2025-01-13 10:19:53 +00:00
Alex Auvolat
8ddb0dd485 nix build: switch to upstream cargo2nix (branch release-0.11.0) 2025-01-12 18:16:23 +01:00
Alex Auvolat
83887a8519 nix build: remove clippy build env that doesn't work 2025-01-12 17:51:33 +01:00
Alex Auvolat
0a15db6960 nix build: update rustc to v1.78 2025-01-12 17:37:36 +01:00
Alex Auvolat
295237476e fix formatting to comply with latest rustfmt 2025-01-12 17:36:25 +01:00
Alex Auvolat
9d83605736 flake: update versions of nixpkgs and rust-overlay 2025-01-12 17:34:04 +01:00
maximilien
4b1a7fb5e3 Merge pull request 'The version flag is required when applying a layout' (#921) from update-quickstart-docs-layout-apply into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/921
2025-01-09 00:41:35 +00:00
fabientot
b6aaebaf4c The version flag is required when applying a layout
I followed the documentation and got an error if the layout's version was not specified 

```
garage layout apply

Error: Internal error:
Please pass the new layout version number to ensure that you are writing the correct version of the cluster layout.
To know the correct value of the new layout version, invoke `garage layout show` and review the proposed changes.
```

This fixes that
2025-01-08 20:30:09 +00:00
Alex
7bbc8fec50 Merge pull request 'Fix #907' (#917) from vk/garage:fix_907 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/917
2025-01-04 16:07:40 +00:00
Vedad KAJTAZ
6689800986 Formatting with 2025-01-04 16:52:23 +01:00
Alex
d2246baab7 Merge pull request 'update flake.lock' (#918) from update-flake into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/918
2025-01-04 15:43:41 +00:00
Alex Auvolat
afac1d4d4a update flake.lock 2025-01-04 16:29:42 +01:00
Vedad KAJTAZ
6ca99fd02c formatting 2025-01-04 14:46:42 +01:00
Vedad KAJTAZ
b568bb863d Fix #907 2025-01-04 12:50:10 +01:00
Alex
b8f301a61d Merge pull request 'woodpecker: use modern syntax for secrets (removes warning)' (#912) from woodpecker-fix-warnings into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/912
2024-12-23 17:41:15 +00:00
Alex Auvolat
428ad2075d
woodpecker: use modern syntax for secrets (removes warning) 2024-12-23 18:00:22 +01:00
maximilien
3661a597fa Merge pull request 'feat: add use_local_tz configuration' (#908) from ragazenta/garage:feat/local-timezone into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/908
Reviewed-by: maximilien <me@mricher.fr>
2024-12-01 13:23:24 +00:00
Renjaya Raga Zenta
0fd3c0e794
doc: add use_local_tz configuration 2024-11-25 10:35:00 +07:00
Renjaya Raga Zenta
4c1bf42192
feat: add use_local_tz configuration
Used in lifecycle_worker to determine midnight time
2024-11-23 05:51:12 +07:00
maximilien
906c8708fd Merge pull request 'add extraVolumes and extraVolumeMounts to helm chart' (#896) from eugene-davis/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/896
Reviewed-by: maximilien <me@mricher.fr>
2024-11-19 22:23:13 +00:00
Alex
747889a096 Merge pull request 'Update Python SDK documentation' (#887) from cryptolukas/garage:fix-python-sdk-docs into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/887
2024-11-19 09:15:03 +00:00
Alex
feb09a4bc6 Merge pull request 'doc: update mastodon media header pruning section' (#888) from teutat3s/garage:doc-update-mastodon-media into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/888
2024-11-19 09:14:34 +00:00
maximilien
aa8bc6aa88 Merge pull request 'doc: add Triplebit's use-case' (#901) from jonah/garage:triplebit into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/901
Reviewed-by: maximilien <me@mricher.fr>
2024-11-17 13:43:49 +00:00
Jonah Aragon
aba7902995
doc: add Triplebit's use-case 2024-11-15 16:27:46 -06:00
Alex
78de7b5bde Merge pull request 'fix bit/byte inversion in rpc secret error message' (#898) from trinity-1686a/garage:rpc-comment into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/898
2024-11-07 11:11:12 +00:00
trinity-1686a
9bd9e392ba fix bit/byte inversion in rpc secret error message 2024-11-07 00:29:26 +01:00
Eugene Davis
116ad479a8
add extraVolumes and extraVolumeMounts to helm chart 2024-10-26 21:14:08 +02:00
teutat3s
b6a58c5c16
doc: update mastodon media header pruning section
This is now possible since the upstream issue has been resolved.
https://github.com/mastodon/mastodon/issues/9567
2024-10-17 20:59:21 +02:00
Matthias Doering
2b0bfa9b18 the old value do not work out of the box 2024-10-14 17:20:26 +02:00
Alex
a18b3f0d1f Merge pull request 'Garage v1.0.1' (#881) from rel-v1.0.1 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/881
2024-09-22 13:02:02 +00:00
Alex Auvolat
7a143f46fc
Bump to version 1.0.1 2024-09-22 14:25:32 +02:00
Alex
c731f0291a Merge pull request 'fix logic in garage layout skip-dead-nodes + fix typo (fix #879)' (#880) from fix-skip-dead-nodes into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/880
2024-09-22 12:01:49 +00:00
Alex Auvolat
34453bc9c2
fix logic in garage layout skip-dead-nodes + fix typo (fix #879) 2024-09-22 13:47:27 +02:00
Alex
6da1353541 Merge pull request 'Don't fetch old values in cross-partition transactional inserts' (#877) from withings/garage:perf/kv/insert-no-return-cross-partition into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/877
2024-09-14 15:57:27 +00:00
Julien Kritter
bd71728874
Tests: don't expect old value after transactional insert 2024-09-12 10:50:53 +02:00
Julien Kritter
51ced60366
Don't fetch old values in cross-partition transactional inserts 2024-09-12 10:26:28 +02:00
Alex
586957b4b7 Merge pull request 'KV: don't retrieve values for write ops' (#873) from marvinj97/garage:perf/kv/insert-no-return into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/873
Reviewed-by: Alex <alex@adnab.me>
2024-09-10 09:06:29 +00:00
Alex
8d2bb4afeb Merge pull request 'Typo' (#875) from faust/garage:doc2 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/875
2024-09-10 09:03:31 +00:00
Faustin Lammler
c26f32b769
Typo
And remove trailing white space.
2024-09-10 09:34:59 +02:00
marvin-j97
8062ec7b4b test: fix db tests 2024-09-04 19:24:36 +02:00
marvin-j97
eb416a02fb dont assert deletion count in sqlite KV adapter 2024-09-04 18:51:51 +02:00
marvin-j97
74363c9060 perf(kv): dont retrieve values for write ops
see https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/851
2024-09-04 18:45:17 +02:00
Alex
615698df7d Merge pull request 'update compiler to Rust 1.77' (#866) from rust-1.77 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/866
2024-08-26 19:08:00 +00:00
Alex Auvolat
7061fa5a56
use rust 1.77 in nix/compile.nix 2024-08-26 19:19:16 +02:00
Alex Auvolat
8881930cdd
update nixpkgs and rust-overlay sources in flake.nix 2024-08-26 19:19:16 +02:00
Alex
52f6c0760b Merge pull request 'update crate time (fix #849)' (#865) from update-time into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/865
2024-08-26 16:20:04 +00:00
Alex Auvolat
5b0602c7e9
update crate time (fix #849) 2024-08-26 18:11:21 +02:00
Alex
182b2af7e5 Merge pull request 'api servers: kill opened connections after SIGINT after 10s deadline (fix #806)' (#864) from exit-deadline into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/864
2024-08-25 18:34:55 +00:00
Alex Auvolat
baf32c9575
api servers: kill opened connections after SIGINT after 10s deadline (fix #806) 2024-08-25 20:04:56 +02:00
Alex
3dda1ee4f6 Merge pull request 'fix build when lmdb feature is disabled (fix #800)' (#863) from fix-800 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/863
2024-08-25 10:00:40 +00:00
Alex Auvolat
aa7ce9e97c
fix build when lmdb feature is disabled (fix #800) 2024-08-25 11:42:37 +02:00
Alex
8d62616ec0 Merge pull request 'layout: discard old info when it is completely out-of-date (fix #841)' (#861) from fix-841 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/861
2024-08-24 11:12:39 +00:00
Alex
bd6fe72c06 Merge pull request 'Quick start: mention Docker (replace #803)' (#862) from dougreeder into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/862
2024-08-24 11:07:46 +00:00
Alex Auvolat
4c9e8ef625
doc: clarify quick start on using docker 2024-08-24 13:07:02 +02:00
Alex
3e711bc110 Merge pull request 'don't modify postobject request before validating policy' (#850) from trinity-1686a/garage:fix-acl-postobject into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/850
2024-08-24 10:49:14 +00:00
Alex Auvolat
7fb66b4944
layout: discard old info when it is completely out-of-date (fix #841) 2024-08-24 12:38:56 +02:00
Quentin
679ae8bcbb Merge pull request 'Set "no read ahead" on LMDB to improve performances when "LMDB size" > "RAM size"' (#855) from fix-lmdb-no-read-ahead into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/855
Reviewed-by: Alex <alex@adnab.me>
2024-08-18 12:25:35 +00:00
Quentin Dufour
2a93ad0c84
force flag "no read ahead" on LMDB 2024-08-17 21:17:15 +02:00
trinity-1686a
f190032589 don't modify postobject request before validating policy 2024-08-10 20:10:47 +02:00
maximilien
3a87bd1370 Merge pull request 'Improve error message for malformed RPC secret key' (#846) from improve-secret-error-message into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/846
Reviewed-by: Quentin <quentin@dufour.io>
2024-08-09 06:47:11 +00:00
Maximilien R.
9302cd42f0 Improve error message for malformed RPC secret key 2024-08-08 23:05:24 +00:00
kosssi
060ad0da32 docs: Update LMDB website 2024-08-06 21:47:14 +00:00
maximilien
a5ed1161c6 Merge pull request 'Add environment variable dict to helm chart.' (#843) from Benjamin/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/843
Reviewed-by: maximilien <me@mricher.fr>
2024-08-06 21:45:35 +00:00
Benjamin von Mossner
222674432b This commit adds an environment dict to garage helm chart. Using it, env variables can be set into the garage container environment, useful to set eg. GARAGE_ADMIN_TOKEN or GARAGE_METRICS_TOKEN 2024-07-25 11:42:13 +02:00
Alex
070a8ad110 Merge pull request 'doc: fix typo' (#831) from Armael/garage:typo into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/831
2024-06-18 12:40:32 +00:00
Alex
770384cae1 Merge pull request 'add rpc_public_addr_subnet config option' (#817) from flokli/garage:rpc_public_addr_subnet into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/817
Reviewed-by: Alex <alex@adnab.me>
2024-06-18 12:40:07 +00:00
Florian Klink
a0f6bc5b7f add rpc_public_addr_subnet config option
In case `rpc_public_addr` is not set, but autodiscovery is used, this
allows filtering the list of automatically discovered IPs to a specific
subnet.

For example, if nodes should pick *their* IP inside a specific subnet,
but you don't want to explicitly write the IP down (as it's dynamic, or
you want to share configs across nodes), you can use this option.
2024-06-05 08:41:36 +02:00
Armaël Guéneau
88c734bbd9 typo 2024-06-04 15:34:02 +02:00
maximilien
d38509ef4b Merge pull request 'adding the ability to change the default podManagementPolicy for StatefulSets' (#823) from bodaciousbiscuits/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/823
Reviewed-by: maximilien <me@mricher.fr>
2024-05-25 18:35:53 +00:00
bodaciousbiscuits
39b37833c5 adding the ability to change the default podManagementPolicy for StatefulSets 2024-05-19 21:31:19 -05:00
maximilien
a2c1de646b Merge pull request 'cli: clarify garage block is node-local' (#813) from flokli/garage:block-node-local into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/813
2024-05-12 08:53:26 +00:00
Florian Klink
15847a636a cli: clarify garage block is node-local
Prevents some of the confusion from
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/810.
2024-05-07 07:42:33 +00:00
Alex
123d3e1f04 Merge pull request 'flake.nix: add rust-analyzer to devShells.full, expose devShells.full in shell.nix' (#816) from flokli/garage:shell-fixes into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/816
2024-04-23 18:54:26 +00:00
Florian Klink
a6e4b96ca9 shell.nix: expose devShellFull
This allows accessing devShells.full from shell.nix.
2024-04-23 11:59:37 +03:00
Florian Klink
b442b0e35e devShells.full: add rust-analyzer 2024-04-23 11:59:37 +03:00
P. Douglas Reeder
0c3b198b22 Improves Quick Start for users not using Linux 2024-04-10 16:42:10 -04:00
Alex
33c2086d9e Merge pull request '[fix-doc] fix broken references in documentation' (#802) from fix-doc into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/802
2024-04-10 15:49:03 +00:00
Alex Auvolat
5ad1e55ccf [fix-doc] fix broken references in documentation 2024-04-10 17:47:34 +02:00
Alex
1779fd40c0 Merge pull request 'Garage v1.0' (#683) from next-0.10 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/683
2024-04-10 15:23:12 +00:00
Alex Auvolat
ff093ddbb8
Merge branch 'main' into next-0.10 2024-04-10 14:38:14 +02:00
Alex Auvolat
90e3c2af91
[next-0.10] small updates to mention Garage v0.9.4 2024-04-10 14:35:30 +02:00
Alex
b47706809c Merge pull request 'fix typo in doc' (#799) from fix-typo into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/799
2024-04-08 15:09:27 +00:00
Alex Auvolat
126e0f47a3 fix typo in doc 2024-04-08 17:08:44 +02:00
Alex
738bb2f09c Merge pull request 'Garage v0.9.4' (#798) from rel-0.9.4 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/798
2024-04-08 09:57:03 +00:00
Alex Auvolat
7dd7cb5759
[rel-0.9.4] upgrade version to v0.9.4 2024-04-08 11:18:19 +02:00
Alex
8b663d8c5b Merge pull request 'jepsen testing of Garage v1.0.0-rc1' (#796) from jepsen-1.0rc1 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/796
2024-04-05 21:05:58 +00:00
Alex Auvolat
c051db8204 jepsen testing of Garage v1.0.0-rc1 2024-04-05 22:57:00 +02:00
Alex Auvolat
50669b3e76
[next-0.10] bump helm chart version 2024-04-03 14:19:59 +02:00
Alex
e5838b4837 Merge pull request '[doc-units] document how interval value is parsed' (#795) from doc-units into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/795
2024-04-03 12:15:18 +00:00
Alex Auvolat
87dfaf2eb9
[doc-units] document how interval value is parsed 2024-04-03 14:14:13 +02:00
Alex Auvolat
554437254e
[next-0.10] Add migration guide for v1.0 2024-03-28 18:45:06 +01:00
Alex Auvolat
afad62939e
[next-0.10] bump version number to 1.0 2024-03-28 15:19:44 +01:00
Alex Auvolat
8bfc16ba7d
Merge branch 'main' into next-0.10 2024-03-28 15:01:05 +01:00
Alex
ecf641d88c Merge pull request 'Fix unbounded buffering when one node has slower network' (#792) from fix-buffering into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/792
2024-03-28 12:40:27 +00:00
Alex
75cd14926d Merge pull request 'CI: properly cleanup between garage integration tests' (#793) from fix-ci into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/793
2024-03-28 12:37:18 +00:00
Alex Auvolat
e1dc84e123
[fix-ci] CI: properly cleanup between garage integration tests 2024-03-28 13:08:42 +01:00
Alex Auvolat
85f580cbde
[fix-buffering] change request sending strategy and fix priorities
remove LAS, priorize new requests but otherwise just do standard queuing
2024-03-27 16:22:40 +01:00
Alex Auvolat
0d3e285d13
[fix-buffering] implement block_ram_buffer_max to avoid excessive RAM usage 2024-03-27 16:22:40 +01:00
Alex Auvolat
25c196f34d
[next-0.10] admin api: fix logic in get cluster status 2024-03-27 13:55:49 +01:00
Alex Auvolat
4eba32f29f
[next-0.10] layout helper: rename & clarify updates to update trackers 2024-03-27 13:47:06 +01:00
Alex Auvolat
32f1786f9f
[next-0.10] cache layout check result 2024-03-27 13:37:20 +01:00
Alex Auvolat
01a0bd5410
[next-0.10] remove impl Deref for LayoutHelper 2024-03-27 13:32:13 +01:00
Alex Auvolat
c0eeb0b0f3
[next-0.10] fixes to k2v rpc + comment fixes 2024-03-27 10:44:03 +01:00
Alex Auvolat
51d11b4b26
[next-0.10] doc: 2 changes
- rewrite section on encryption to mention SSE-C
- change to real-world to make it closer to main branch
2024-03-27 10:10:45 +01:00
Alex Auvolat
f7cd4eb600
Merge branch 'main' into next-0.10 2024-03-26 16:34:40 +01:00
Alex
95eb8808e8 Merge pull request 'Disable more K2V tests' (#791) from disable-k2v-test into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/791
2024-03-26 15:33:46 +00:00
Alex Auvolat
e0a4fc097a
[disable-k2v-test] remove obsolete k2v test script 2024-03-26 16:24:44 +01:00
Alex Auvolat
73551e9a2d
[disable-k2v-test] disable the other k2v poll test 2024-03-26 16:24:26 +01:00
Alex
80f81fa6f3 Merge pull request '[disable-k2v-test] disable tests::k2v::test_poll_item as it is not 100% reliable' (#789) from disable-k2v-test into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/789
2024-03-26 14:57:46 +00:00
Alex Auvolat
f267609343
[disable-k2v-test] disable tests::k2v::test_poll_item as it is not 100% reliable 2024-03-26 15:39:17 +01:00
Alex
cdde0f19ee Merge pull request 'checksum algorithms' (#787) from s3-checksum into next-0.10
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/787
2024-03-26 14:24:58 +00:00
Alex Auvolat
74949c69cb
[s3-checksum] implement x-amz-checksum-* headers 2024-03-26 15:01:34 +01:00
Alex
7e0107c47d Merge pull request 'Fixes to garage_net peering manager' (#786) from net-fixes into next-0.10
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/786
2024-03-21 10:26:36 +00:00
Alex Auvolat
3844110cd0
[net-fixes] netapp peer exchange: send only currently connected address 2024-03-21 10:50:44 +01:00
Alex Auvolat
961b4f9af3
[net-fixes] fix issues with local peer address (fix #761) 2024-03-21 10:45:34 +01:00
Alex Auvolat
5225a81dee
[net-fixes] peering: only count node IDs and not addresses in hash 2024-03-21 09:47:04 +01:00
Alex
e835196940 Merge pull request 'Add marker files in data directories (fix #601)' (#785) from check-data-dir into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/785
2024-03-20 16:53:47 +00:00
Alex Auvolat
ba33bb31f1
[check-data-dir] add marker files in data directories (fix #601) 2024-03-20 15:20:25 +01:00
Alex
30abf7e086 Merge pull request 'Add support to logging to syslog (based on patch by @jirutka)' (#784) from syslog into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/784
2024-03-20 13:54:18 +00:00
Alex Auvolat
84018be862
[syslog] warning when syslog support is not enabled 2024-03-20 14:39:04 +01:00
Alex Auvolat
091e693670
[syslog] document environment variables 2024-03-20 14:39:04 +01:00
Alex Auvolat
fe8a7819fa
[syslog] Add support to logging to syslog
Original patch by Jakub Jirutka for Alpine Linux port.
2024-03-20 14:22:18 +01:00
Alex Auvolat
ce69dc302c
Merge branch 'main' into next-0.10 2024-03-19 17:17:46 +01:00
Alex
26310f3242 Merge pull request 'CLI: allow manipulating buckets by prefixes of their full IDs' (#783) from bucket-id-prefix into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/783
2024-03-19 16:17:16 +00:00
Alex
65853a4863 Merge pull request 'block refcount repair' (#782) from block-ref-repair into next-0.10
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/782
2024-03-19 15:59:19 +00:00
Alex Auvolat
783b586de9
[bucket-id-prefix] CLI: allow manipulating buckets by prefixes of their full IDs 2024-03-19 16:57:51 +01:00
Alex Auvolat
3eab639c14
[block-ref-repair] mention garage block repair-rc in documentation 2024-03-19 16:24:34 +01:00
Alex Auvolat
3165ab926c
[block-ref-repair] rename rc's rc field to rc_table 2024-03-19 16:20:22 +01:00
Alex Auvolat
dc0b78cdb8
[block-ref-repair] Block refcount recalculation and repair
- We always recalculate the reference count of a block before deleting
  it locally, to make sure that it is indeed zero.

- If we had to fetch a remote block but we were not able to get it,
  check that refcount is indeed > 0.

- Repair procedure that checks everything
2024-03-19 16:20:22 +01:00
Alex
693b89b94b Merge pull request 'Update WinSCP link in documentation' (#781) from stefano/garage:main into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/781
2024-03-19 09:34:16 +00:00
stefano
cf344d73d5 Update WinSCP link in documentation
Update link to new wiki location. See Deuxfleurs/garage#780
2024-03-19 09:21:50 +00:00
Alex Auvolat
0038ca8a78
Merge branch 'main' into next-0.10 2024-03-18 20:19:30 +01:00
Alex
1a0bffae34 Merge pull request 'Use connection pooling in sqlite backend' (#779) from sqlite-r2d2 into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/779
2024-03-18 19:02:36 +00:00
Alex Auvolat
b55f52a9b7
[sqlite-r2d2] run integration test with all db engines 2024-03-18 18:31:35 +01:00
Alex Auvolat
e8f9718ccd
[sqlite-r2d2] implement connection pooling in sqlite backend 2024-03-18 18:05:25 +01:00
Alex
fd2e19bf1b Merge pull request 'metadata db snapshotting' (#775) from db-snapshot into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/775
2024-03-15 13:17:53 +00:00
Alex Auvolat
8cf3d24875
[db-snapshot] documentation for metadata db snapshots 2024-03-15 13:51:31 +01:00
Alex Auvolat
a68c37555d
[db-snapshot] add garage meta snapshot cli operation 2024-03-15 13:51:31 +01:00
Alex Auvolat
1e42808a59
[db-snapshot] implement meta_auto_snapshot_interval 2024-03-15 13:51:31 +01:00
Alex Auvolat
8dff278b72
[db-snapshot] Implement db snapshotting logic in garage_db 2024-03-15 10:57:22 +01:00
Alex
a80ce6ab5a Merge pull request 'disable_scrub configuration option' (#774) from disable-scrub into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/774
2024-03-15 09:22:33 +00:00
Alex Auvolat
990205dc3b
[disable-scrub] document disable_scrub config option 2024-03-14 17:01:16 +01:00
Alex Auvolat
7c86ff6c37
[disable-scrub] implement a disable_scrub configuration option 2024-03-14 17:01:16 +01:00
Alex
62b01d8705 Merge pull request 'Doc: be slightly more critical of LMDB' (#773) from doc-updates into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/773
2024-03-14 15:43:30 +00:00
Alex Auvolat
422d45b659
[doc-updates] from source: fix default feature list 2024-03-14 16:35:15 +01:00
Alex Auvolat
a7dddebedd
[doc-updates] doc: be slightly more critical of LMDB 2024-03-14 16:31:22 +01:00
Quentin Dufour
8b35a946d9
Allow external HTTP client 2024-02-23 17:09:47 +01:00
221 changed files with 10461 additions and 11951 deletions

View file

@ -1,3 +0,0 @@
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

View file

@ -1,3 +1,6 @@
labels:
nix: "enabled"
when: when:
event: event:
- push - push
@ -9,37 +12,33 @@ when:
steps: steps:
- name: check formatting - name: check formatting
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-shell --attr devShell --run "cargo fmt -- --check" - nix-build -j4 --attr flakePackages.fmt
- name: build - name: build
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-build --no-build-output --attr clippy.amd64 --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA} - nix-build -j4 --attr flakePackages.dev
- name: unit + func tests - name: unit + func tests (lmdb)
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
environment:
GARAGE_TEST_INTEGRATION_EXE: result-bin/bin/garage
GARAGE_TEST_INTEGRATION_PATH: tmp-garage-integration
commands: commands:
- nix-build --no-build-output --attr clippy.amd64 --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA} - nix-build -j4 --attr flakePackages.tests-lmdb
- nix-build --no-build-output --attr test.amd64
- ./result/bin/garage_db-* - name: unit + func tests (sqlite)
- ./result/bin/garage_api-* image: nixpkgs/nix:nixos-24.05
- ./result/bin/garage_model-* commands:
- ./result/bin/garage_rpc-* - nix-build -j4 --attr flakePackages.tests-sqlite
- ./result/bin/garage_table-*
- ./result/bin/garage_util-* - name: unit + func tests (fjall)
- ./result/bin/garage_web-* image: nixpkgs/nix:nixos-24.05
- ./result/bin/garage-* commands:
- ./result/bin/integration-* || (cat tmp-garage-integration/stderr.log; false) - nix-build -j4 --attr flakePackages.tests-fjall
- rm result
- rm -rv tmp-garage-integration
- name: integration tests - name: integration tests
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-build --no-build-output --attr clippy.amd64 --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA} - nix-build -j4 --attr flakePackages.dev
- nix-shell --attr ci --run ./script/test-smoke.sh || (cat /tmp/garage.log; false) - nix-shell --attr ci --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)
depends_on: [ build ]

View file

@ -1,3 +1,6 @@
labels:
nix: "enabled"
when: when:
event: event:
- deployment - deployment
@ -8,20 +11,21 @@ depends_on:
steps: steps:
- name: refresh-index - name: refresh-index
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
secrets: environment:
- source: garagehq_aws_access_key_id AWS_ACCESS_KEY_ID:
target: AWS_ACCESS_KEY_ID from_secret: garagehq_aws_access_key_id
- source: garagehq_aws_secret_access_key AWS_SECRET_ACCESS_KEY:
target: AWS_SECRET_ACCESS_KEY from_secret: garagehq_aws_secret_access_key
commands: commands:
- mkdir -p /etc/nix && cp nix/nix.conf /etc/nix/nix.conf - mkdir -p /etc/nix && cp nix/nix.conf /etc/nix/nix.conf
- nix-shell --attr ci --run "refresh_index" - nix-shell --attr ci --run "refresh_index"
- name: multiarch-docker - name: multiarch-docker
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
secrets: environment:
- docker_auth DOCKER_AUTH:
from_secret: docker_auth
commands: commands:
- mkdir -p /root/.docker - mkdir -p /root/.docker
- echo $DOCKER_AUTH > /root/.docker/config.json - echo $DOCKER_AUTH > /root/.docker/config.json

View file

@ -1,3 +1,6 @@
labels:
nix: "enabled"
when: when:
event: event:
- deployment - deployment
@ -16,18 +19,17 @@ matrix:
steps: steps:
- name: build - name: build
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-build --no-build-output --attr pkgs.${ARCH}.release --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA} - nix-build --attr releasePackages.${ARCH} --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA}
- name: check is static binary - name: check is static binary
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-build --no-build-output --attr pkgs.${ARCH}.release --argstr git_version ${CI_COMMIT_TAG:-$CI_COMMIT_SHA} - nix-shell --attr ci --run "./script/not-dynamic.sh result/bin/garage"
- nix-shell --attr ci --run "./script/not-dynamic.sh result-bin/bin/garage"
- name: integration tests - name: integration tests
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-shell --attr ci --run ./script/test-smoke.sh || (cat /tmp/garage.log; false) - nix-shell --attr ci --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)
when: when:
@ -37,7 +39,7 @@ steps:
ARCH: i386 ARCH: i386
- name: upgrade tests - name: upgrade tests
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
commands: commands:
- nix-shell --attr ci --run "./script/test-upgrade.sh v0.8.4 x86_64-unknown-linux-musl" || (cat /tmp/garage.log; false) - nix-shell --attr ci --run "./script/test-upgrade.sh v0.8.4 x86_64-unknown-linux-musl" || (cat /tmp/garage.log; false)
when: when:
@ -45,24 +47,23 @@ steps:
ARCH: amd64 ARCH: amd64
- name: push static binary - name: push static binary
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
environment: environment:
TARGET: "${TARGET}" TARGET: "${TARGET}"
secrets: AWS_ACCESS_KEY_ID:
- source: garagehq_aws_access_key_id from_secret: garagehq_aws_access_key_id
target: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY:
- source: garagehq_aws_secret_access_key from_secret: garagehq_aws_secret_access_key
target: AWS_SECRET_ACCESS_KEY
commands: commands:
- nix-shell --attr ci --run "to_s3" - nix-shell --attr ci --run "to_s3"
- name: docker build and publish - name: docker build and publish
image: nixpkgs/nix:nixos-22.05 image: nixpkgs/nix:nixos-24.05
environment: environment:
DOCKER_PLATFORM: "linux/${ARCH}" DOCKER_PLATFORM: "linux/${ARCH}"
CONTAINER_NAME: "dxflrs/${ARCH}_garage" CONTAINER_NAME: "dxflrs/${ARCH}_garage"
secrets: DOCKER_AUTH:
- docker_auth from_secret: docker_auth
commands: commands:
- mkdir -p /root/.docker - mkdir -p /root/.docker
- echo $DOCKER_AUTH > /root/.docker/config.json - echo $DOCKER_AUTH > /root/.docker/config.json

2988
Cargo.lock generated

File diff suppressed because it is too large Load diff

7038
Cargo.nix

File diff suppressed because it is too large Load diff

View file

@ -8,7 +8,10 @@ members = [
"src/table", "src/table",
"src/block", "src/block",
"src/model", "src/model",
"src/api", "src/api/common",
"src/api/s3",
"src/api/k2v",
"src/api/admin",
"src/web", "src/web",
"src/garage", "src/garage",
"src/k2v-client", "src/k2v-client",
@ -21,15 +24,18 @@ default-members = ["src/garage"]
# Internal Garage crates # Internal Garage crates
format_table = { version = "0.1.1", path = "src/format-table" } format_table = { version = "0.1.1", path = "src/format-table" }
garage_api = { version = "0.10.0", path = "src/api" } garage_api_common = { version = "1.3.1", path = "src/api/common" }
garage_block = { version = "0.10.0", path = "src/block" } garage_api_admin = { version = "1.3.1", path = "src/api/admin" }
garage_db = { version = "0.10.0", path = "src/db", default-features = false } garage_api_s3 = { version = "1.3.1", path = "src/api/s3" }
garage_model = { version = "0.10.0", path = "src/model", default-features = false } garage_api_k2v = { version = "1.3.1", path = "src/api/k2v" }
garage_net = { version = "0.10.0", path = "src/net" } garage_block = { version = "1.3.1", path = "src/block" }
garage_rpc = { version = "0.10.0", path = "src/rpc" } garage_db = { version = "1.3.1", path = "src/db", default-features = false }
garage_table = { version = "0.10.0", path = "src/table" } garage_model = { version = "1.3.1", path = "src/model", default-features = false }
garage_util = { version = "0.10.0", path = "src/util" } garage_net = { version = "1.3.1", path = "src/net" }
garage_web = { version = "0.10.0", path = "src/web" } garage_rpc = { version = "1.3.1", path = "src/rpc" }
garage_table = { version = "1.3.1", path = "src/table" }
garage_util = { version = "1.3.1", path = "src/util" }
garage_web = { version = "1.3.1", path = "src/web" }
k2v-client = { version = "0.0.4", path = "src/k2v-client" } k2v-client = { version = "0.0.4", path = "src/k2v-client" }
# External crates from crates.io # External crates from crates.io
@ -43,25 +49,28 @@ bytes = "1.0"
bytesize = "1.1" bytesize = "1.1"
cfg-if = "1.0" cfg-if = "1.0"
chrono = "0.4" chrono = "0.4"
crc32fast = "1.4"
crc32c = "0.6"
crypto-common = "0.1" crypto-common = "0.1"
digest = "0.10"
err-derive = "0.3"
gethostname = "0.4" gethostname = "0.4"
git-version = "0.3.4" git-version = "0.3.4"
hex = "0.4" hex = "0.4"
hexdump = "0.1" hexdump = "0.1"
html-escape = "0.2.13"
hmac = "0.12" hmac = "0.12"
idna = "0.5"
itertools = "0.12" itertools = "0.12"
ipnet = "2.9.0"
lazy_static = "1.4" lazy_static = "1.4"
md-5 = "0.10" md-5 = "0.10"
mktemp = "0.5" mktemp = "0.5"
nix = { version = "0.27", default-features = false, features = ["fs"] } nix = { version = "0.29", default-features = false, features = ["fs"] }
nom = "7.1" nom = "7.1"
parking_lot = "0.12"
parse_duration = "2.1" parse_duration = "2.1"
pin-project = "1.0.12" pin-project = "1.0.12"
pnet_datalink = "0.34" pnet_datalink = "0.34"
rand = "0.8" rand = "0.8"
sha1 = "0.10"
sha2 = "0.10" sha2 = "0.10"
timeago = { version = "0.4", default-features = false } timeago = { version = "0.4", default-features = false }
xxhash-rust = { version = "0.8", default-features = false, features = ["xxh3"] } xxhash-rust = { version = "0.8", default-features = false, features = ["xxh3"] }
@ -73,11 +82,16 @@ kuska-handshake = { version = "0.2.0", features = ["default", "async_std"] }
clap = { version = "4.1", features = ["derive", "env"] } clap = { version = "4.1", features = ["derive", "env"] }
pretty_env_logger = "0.5" pretty_env_logger = "0.5"
structopt = { version = "0.3", default-features = false } structopt = { version = "0.3", default-features = false }
syslog-tracing = "0.3"
tracing = "0.1" tracing = "0.1"
tracing-journald = "0.3.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
heed = { version = "0.11", default-features = false, features = ["lmdb"] } heed = { version = "0.11", default-features = false, features = ["lmdb"] }
rusqlite = "0.30.0" rusqlite = "0.37"
r2d2 = "0.8"
r2d2_sqlite = "0.31"
fjall = "2.4"
async-compression = { version = "0.4", features = ["tokio", "zstd"] } async-compression = { version = "0.4", features = ["tokio", "zstd"] }
zstd = { version = "0.13", default-features = false } zstd = { version = "0.13", default-features = false }
@ -120,26 +134,21 @@ opentelemetry-contrib = "0.9"
prometheus = "0.13" prometheus = "0.13"
# used by the k2v-client crate only # used by the k2v-client crate only
aws-sigv4 = { version = "1.1" } aws-sigv4 = { version = "1.1", default-features = false }
hyper-rustls = { version = "0.26", features = ["http2"] } hyper-rustls = { version = "0.26", default-features = false, features = ["http1", "http2", "ring", "rustls-native-certs"] }
log = "0.4" log = "0.4"
thiserror = "1.0" thiserror = "2.0"
# ---- used only as build / dev dependencies ---- # ---- used only as build / dev dependencies ----
assert-json-diff = "2.0" assert-json-diff = "2.0"
rustc_version = "0.4.0" rustc_version = "0.4.0"
static_init = "1.0" static_init = "1.0"
aws-smithy-runtime = { version = "1.8", default-features = false, features = ["tls-rustls"] }
aws-config = "1.1.4" aws-sdk-config = { version = "1.62", default-features = false }
aws-sdk-config = "1.13" aws-sdk-s3 = { version = "1.79", default-features = false, features = ["rt-tokio"] }
aws-sdk-s3 = "1.14"
[profile.dev]
#lto = "thin" # disabled for now, adds 2-4 min to each CI build
lto = "off"
[profile.release] [profile.release]
lto = true lto = "thin"
codegen-units = 1 codegen-units = 16
opt-level = "s" opt-level = 3
strip = true strip = "debuginfo"

View file

@ -3,5 +3,5 @@ FROM scratch
ENV RUST_BACKTRACE=1 ENV RUST_BACKTRACE=1
ENV RUST_LOG=garage=info ENV RUST_LOG=garage=info
COPY result-bin/bin/garage / COPY result/bin/garage /
CMD [ "/garage", "server"] CMD [ "/garage", "server"]

View file

@ -1,13 +1,8 @@
.PHONY: doc all release shell run1 run2 run3 .PHONY: doc all run1 run2 run3
all: all:
clear; cargo build clear
cargo build
release:
nix-build --attr pkgs.amd64.release --no-build-output
shell:
nix-shell
# ---- # ----

View file

@ -3,53 +3,22 @@
with import ./nix/common.nix; with import ./nix/common.nix;
let let
pkgs = import pkgsSrc { }; pkgs = import nixpkgs { };
compile = import ./nix/compile.nix; compile = import ./nix/compile.nix;
build_debug_and_release = (target: { build_release = target: (compile {
debug = (compile { inherit target system git_version nixpkgs;
inherit system target git_version pkgsSrc cargo2nixOverlay; crane = flake.inputs.crane;
release = false; rust-overlay = flake.inputs.rust-overlay;
}).workspace.garage { compileMode = "build"; }; release = true;
}).garage;
release = (compile {
inherit system target git_version pkgsSrc cargo2nixOverlay;
release = true;
}).workspace.garage { compileMode = "build"; };
});
test = (rustPkgs:
pkgs.symlinkJoin {
name = "garage-tests";
paths =
builtins.map (key: rustPkgs.workspace.${key} { compileMode = "test"; })
(builtins.attrNames rustPkgs.workspace);
});
in { in {
pkgs = { releasePackages = {
amd64 = build_debug_and_release "x86_64-unknown-linux-musl"; amd64 = build_release "x86_64-unknown-linux-musl";
i386 = build_debug_and_release "i686-unknown-linux-musl"; i386 = build_release "i686-unknown-linux-musl";
arm64 = build_debug_and_release "aarch64-unknown-linux-musl"; arm64 = build_release "aarch64-unknown-linux-musl";
arm = build_debug_and_release "armv6l-unknown-linux-musleabihf"; arm = build_release "armv6l-unknown-linux-musleabihf";
};
test = {
amd64 = test (compile {
inherit system git_version pkgsSrc cargo2nixOverlay;
target = "x86_64-unknown-linux-musl";
features = [
"garage/bundled-libs"
"garage/k2v"
"garage/lmdb"
"garage/sqlite"
];
});
};
clippy = {
amd64 = (compile {
inherit system git_version pkgsSrc cargo2nixOverlay;
target = "x86_64-unknown-linux-musl";
compiler = "clippy";
}).workspace.garage { compileMode = "build"; };
}; };
flakePackages = flake.packages.${system};
} }

View file

@ -687,7 +687,7 @@ paths:
operationId: "GetBucketInfo" operationId: "GetBucketInfo"
summary: "Get a bucket" summary: "Get a bucket"
description: | description: |
Given a bucket identifier (`id`) or a global alias (`alias`), get its information. Given a bucket identifier (`id`) or a global alias (`globalAlias`), get its information.
It includes its aliases, its web configuration, keys that have some permissions It includes its aliases, its web configuration, keys that have some permissions
on it, some statistics (number of objects, size), number of dangling multipart uploads, on it, some statistics (number of objects, size), number of dangling multipart uploads,
and its quotas (if any). and its quotas (if any).
@ -701,7 +701,7 @@ paths:
example: "b4018dc61b27ccb5c64ec1b24f53454bbbd180697c758c4d47a22a8921864a87" example: "b4018dc61b27ccb5c64ec1b24f53454bbbd180697c758c4d47a22a8921864a87"
schema: schema:
type: string type: string
- name: alias - name: globalAlias
in: query in: query
description: | description: |
The exact global alias of one of the existing buckets. The exact global alias of one of the existing buckets.

View file

@ -23,7 +23,7 @@ client = minio.Minio(
"GKyourapikey", "GKyourapikey",
"abcd[...]1234", "abcd[...]1234",
# Force the region, this is specific to garage # Force the region, this is specific to garage
region="region", region="garage",
) )
``` ```

View file

@ -12,7 +12,7 @@ In this section, we cover the following web applications:
| [Mastodon](#mastodon) | ✅ | Natively supported | | [Mastodon](#mastodon) | ✅ | Natively supported |
| [Matrix](#matrix) | ✅ | Tested with `synapse-s3-storage-provider` | | [Matrix](#matrix) | ✅ | Tested with `synapse-s3-storage-provider` |
| [ejabberd](#ejabberd) | ✅ | `mod_s3_upload` | | [ejabberd](#ejabberd) | ✅ | `mod_s3_upload` |
| [Pixelfed](#pixelfed) | ❓ | Not yet tested | | [Pixelfed](#pixelfed) | ✅ | Natively supported |
| [Pleroma](#pleroma) | ❓ | Not yet tested | | [Pleroma](#pleroma) | ❓ | Not yet tested |
| [Lemmy](#lemmy) | ✅ | Supported with pict-rs | | [Lemmy](#lemmy) | ✅ | Supported with pict-rs |
| [Funkwhale](#funkwhale) | ❓ | Not yet tested | | [Funkwhale](#funkwhale) | ❓ | Not yet tested |
@ -69,7 +69,7 @@ $CONFIG = array(
'hostname' => '127.0.0.1', // Can also be a domain name, eg. garage.example.com 'hostname' => '127.0.0.1', // Can also be a domain name, eg. garage.example.com
'port' => 3900, // Put your reverse proxy port or your S3 API port 'port' => 3900, // Put your reverse proxy port or your S3 API port
'use_ssl' => false, // Set it to true if you have a TLS enabled reverse proxy 'use_ssl' => false, // Set it to true if you have a TLS enabled reverse proxy
'region' => 'garage', // Garage has only one region named "garage" 'region' => 'garage', // Garage default region is named "garage", edit according to your cluster config
'use_path_style' => true // Garage supports only path style, must be set to true 'use_path_style' => true // Garage supports only path style, must be set to true
], ],
], ],
@ -135,7 +135,7 @@ bucket but doesn't also know the secret encryption key.
*Click on the picture to zoom* *Click on the picture to zoom*
Add a new external storage. Put what you want in "folder name" (eg. "shared"). Select "Amazon S3". Keep "Access Key" for the Authentication field. Add a new external storage. Put what you want in "folder name" (eg. "shared"). Select "Amazon S3". Keep "Access Key" for the Authentication field.
In Configuration, put your bucket name (eg. nextcloud), the host (eg. 127.0.0.1), the port (eg. 3900 or 443), the region (garage). Tick the SSL box if you have put an HTTPS proxy in front of garage. You must tick the "Path access" box and you must leave the "Legacy authentication (v2)" box empty. Put your Key ID (eg. GK...) and your Secret Key in the last two input boxes. Finally click on the tick symbol on the right of your screen. In Configuration, put your bucket name (eg. nextcloud), the host (eg. 127.0.0.1), the port (eg. 3900 or 443), the region ("garage" if you use the default, or the one your configured in your `garage.toml`). Tick the SSL box if you have put an HTTPS proxy in front of garage. You must tick the "Path access" box and you must leave the "Legacy authentication (v2)" box empty. Put your Key ID (eg. GK...) and your Secret Key in the last two input boxes. Finally click on the tick symbol on the right of your screen.
Now go to your "Files" app and a new "linked folder" has appeared with the name you chose earlier (eg. "shared"). Now go to your "Files" app and a new "linked folder" has appeared with the name you chose earlier (eg. "shared").
@ -191,10 +191,10 @@ garage key create peertube-key
Keep the Key ID and the Secret key in a pad, they will be needed later. Keep the Key ID and the Secret key in a pad, they will be needed later.
We need two buckets, one for normal videos (named peertube-video) and one for webtorrent videos (named peertube-playlist). We need two buckets, one for normal videos (named peertube-videos) and one for webtorrent videos (named peertube-playlists).
```bash ```bash
garage bucket create peertube-videos garage bucket create peertube-videos
garage bucket create peertube-playlist garage bucket create peertube-playlists
``` ```
Now we allow our key to read and write on these buckets: Now we allow our key to read and write on these buckets:
@ -238,7 +238,7 @@ object_storage:
# Put localhost only if you have a garage instance running on that node # Put localhost only if you have a garage instance running on that node
endpoint: 'http://localhost:3900' # or "garage.example.com" if you have TLS on port 443 endpoint: 'http://localhost:3900' # or "garage.example.com" if you have TLS on port 443
# Garage supports only one region for now, named garage # Garage default region is named "garage", edit according to your config
region: 'garage' region: 'garage'
credentials: credentials:
@ -253,7 +253,7 @@ object_storage:
proxify_private_files: false proxify_private_files: false
streaming_playlists: streaming_playlists:
bucket_name: 'peertube-playlist' bucket_name: 'peertube-playlists'
# Keep it empty for our example # Keep it empty for our example
prefix: '' prefix: ''
@ -335,6 +335,7 @@ From the [official Mastodon documentation](https://docs.joinmastodon.org/admin/t
```bash ```bash
$ RAILS_ENV=production bin/tootctl media remove --days 3 $ RAILS_ENV=production bin/tootctl media remove --days 3
$ RAILS_ENV=production bin/tootctl media remove --days 15 --prune-profiles
$ RAILS_ENV=production bin/tootctl media remove-orphans $ RAILS_ENV=production bin/tootctl media remove-orphans
$ RAILS_ENV=production bin/tootctl preview_cards remove --days 15 $ RAILS_ENV=production bin/tootctl preview_cards remove --days 15
``` ```
@ -353,8 +354,6 @@ Imports: 1.7 KB
Settings: 0 Bytes Settings: 0 Bytes
``` ```
Unfortunately, [old avatars and headers cannot currently be cleaned up](https://github.com/mastodon/mastodon/issues/9567).
### Migrating your data ### Migrating your data
Data migration should be done with an efficient S3 client. Data migration should be done with an efficient S3 client.
@ -442,7 +441,7 @@ media_storage_providers:
store_synchronous: True # do we want to wait that the file has been written before returning? store_synchronous: True # do we want to wait that the file has been written before returning?
config: config:
bucket: matrix # the name of our bucket, we chose matrix earlier bucket: matrix # the name of our bucket, we chose matrix earlier
region_name: garage # only "garage" is supported for the region field region_name: garage # "garage" by default, edit according to your cluster config
endpoint_url: http://localhost:3900 # the path to the S3 endpoint endpoint_url: http://localhost:3900 # the path to the S3 endpoint
access_key_id: "GKxxx" # your Key ID access_key_id: "GKxxx" # your Key ID
secret_access_key: "xxxx" # your Secret Key secret_access_key: "xxxx" # your Secret Key

View file

@ -161,3 +161,49 @@ kopia repository validate-provider
You can then run all the standard kopia commands: `kopia snapshot create`, `kopia mount`... You can then run all the standard kopia commands: `kopia snapshot create`, `kopia mount`...
Everything should work out-of-the-box. Everything should work out-of-the-box.
## Plakar
Create your key and bucket on Garage server:
```bash
garage key create my-plakar-key
garage bucket create plakar-backups
garage bucket allow plakar-backups --read --write --key my-plakar-key
…
```
On Plakar server, add your Garage as a storage location:
```bash
plakar store add garageS3 s3://my-garage.tld/plakar-backups \
region=garage # Or as you've specified in garage.toml \
access_key=<Key ID from "garage key info my-plakar-key"> \
secret_access_key=<Secret key from "garage key info my-plakar-key">
```
Then create the repository.
```bash
plakar at @garageS3 create -plaintext # Unencrypted
# or
plakar at @garageS3 create #encrypted
```
If you encrypt your backups (Plakar default), you will need to define a strong passphrase. Do not forget to save your password safely. It will be needed to decrypt your backups.
After the repository has been created, check that everything works as expected (that might give an empty result as no file has been added yet, but no error message):
```bash
plakar at @garageS3 check
```
Now that everything is configure, you can use Garage as your backups storage. For instance sync it with a local backup storage:
```bash
$ plakar at ~/backups sync to @garageS3
```
Or list the S3 storage content:
```bash
$ plakar at @garageS3 ls
```
More information in Plakar documentation: https://www.plakar.io/docs/main/quickstart/

View file

@ -259,7 +259,7 @@ duck --delete garage:/my-files/an-object.txt
## WinSCP (libs3) {#winscp} ## WinSCP (libs3) {#winscp}
*You can find instructions on how to use the GUI in french [in our wiki](https://wiki.deuxfleurs.fr/fr/Guide/Garage/WinSCP).* *You can find instructions on how to use the GUI in french [in our wiki](https://guide.deuxfleurs.fr/prise_en_main/winscp/).*
How to use `winscp.com`, the CLI interface of WinSCP: How to use `winscp.com`, the CLI interface of WinSCP:

View file

@ -17,7 +17,7 @@ Garage can also help you serve this content.
## Gitea ## Gitea
You can use Garage with Gitea to store your [git LFS](https://git-lfs.github.com/) data, your users' avatar, and their attachements. You can use Garage with Gitea to store your [git LFS](https://git-lfs.github.com/) data, your users' avatar, and their attachments.
You can configure a different target for each data type (check `[lfs]` and `[attachment]` sections of the Gitea documentation) and you can provide a default one through the `[storage]` section. You can configure a different target for each data type (check `[lfs]` and `[attachment]` sections of the Gitea documentation) and you can provide a default one through the `[storage]` section.
Let's start by creating a key and a bucket (your key id and secret will be needed later, keep them somewhere): Let's start by creating a key and a bucket (your key id and secret will be needed later, keep them somewhere):

View file

@ -8,18 +8,18 @@ have published Ansible roles. We list them and compare them below.
## Comparison of Ansible roles ## Comparison of Ansible roles
| Feature | [ansible-role-garage](#zorun-ansible-role-garage) | [garage-docker-ansible-deploy](#moan0s-garage-docker-ansible-deploy) | | Feature | [ansible-role-garage](#zorun-ansible-role-garage) | [garage-docker-ansible-deploy](#moan0s-garage-docker-ansible-deploy) | [eddster ansible-role-garage](#eddster-ansible-role-garage) |
|------------------------------------|---------------------------------------------|---------------------------------------------------------------| |------------------------------------|---------------------------------------------|---------------------------------------------------------------|---------------------------------|
| **Runtime** | Systemd | Docker | | **Runtime** | Systemd | Docker | Systemd |
| **Target OS** | Any Linux | Any Linux | | **Target OS** | Any Linux | Any Linux | Any Linux |
| **Architecture** | amd64, arm64, i686 | amd64, arm64 | | **Architecture** | amd64, arm64, i686 | amd64, arm64 | arm64, arm, 386, amd64 |
| **Additional software** | None | Traefik | | **Additional software** | None | Traefik | Ngnix and Keepalived (optional) |
| **Automatic node connection** | ❌ | ✅ | | **Automatic node connection** | ❌ | ✅ | ✅ |
| **Layout management** | ❌ | ✅ | | **Layout management** | ❌ | ✅ | ✅ |
| **Manage buckets & keys** | ❌ | ✅ (basic) | | **Manage buckets & keys** | ❌ | ✅ (basic) | ✅ |
| **Allow custom Garage config** | ✅ | ❌ | | **Allow custom Garage config** | ✅ | ❌ | ❌ |
| **Facilitate Garage upgrades** | ✅ | ❌ | | **Facilitate Garage upgrades** | ✅ | ❌ | ✅ |
| **Multiple instances on one host** | ✅ | ✅ | | **Multiple instances on one host** | ✅ | ✅ | ❌ |
## zorun/ansible-role-garage ## zorun/ansible-role-garage
@ -49,3 +49,15 @@ structured DNS names, etc).
As a result, this role makes it easier to start with Garage on Ansible, As a result, this role makes it easier to start with Garage on Ansible,
but is less flexible. but is less flexible.
## eddster2309/ansible-role-garage
[Source code](https://github.com/eddster2309/ansible-role-garage), [Ansible galaxy](https://galaxy.ansible.com/ui/standalone/roles/eddster2309/garage/)
This role is a opinionated but customisable role using the official Garage
static binaries and only requires Systemd. As such it should work on any
Linux based host. It includes all the nesscary configuration to
automatically setup a clustered Garage deployment. Most Garage
configuration options are exposed through Ansible variables so while you
can't provide a custom config you can get very close. It can optionally
installed a HA nginx deployment with Keepalived.

View file

@ -15,9 +15,10 @@ Alpine Linux repositories (available since v3.17):
apk add garage apk add garage
``` ```
The default configuration file is installed to `/etc/garage.toml`. You can run The default configuration file is installed to `/etc/garage/garage.toml`. You can run
Garage using: `rc-service garage start`. If you don't specify `rpc_secret`, it Garage using: `rc-service garage start`.
will be automatically replaced with a random string on the first start.
If you don't specify `rpc_secret`, it will be automatically replaced with a random string on the first start.
Please note that this package is built without Consul discovery, Kubernetes Please note that this package is built without Consul discovery, Kubernetes
discovery, OpenTelemetry exporter, and K2V features (K2V will be enabled once discovery, OpenTelemetry exporter, and K2V features (K2V will be enabled once
@ -26,7 +27,7 @@ it's stable).
## Arch Linux ## Arch Linux
Garage is available in the [AUR](https://aur.archlinux.org/packages/garage). Garage is available in the official repositories under [extra](https://archlinux.org/packages/extra/x86_64/garage).
## FreeBSD ## FreeBSD

View file

@ -53,20 +53,43 @@ and that's also why your nodes have super long identifiers.
Adding TLS support built into Garage is not currently planned. Adding TLS support built into Garage is not currently planned.
## Garage stores data in plain text on the filesystem ## Garage stores data in plain text on the filesystem or encrypted using customer keys (SSE-C)
Garage does not handle data encryption at rest by itself, and instead delegates For standard S3 API requests, Garage does not encrypt data at rest by itself.
to the user to add encryption, either at the storage layer (LUKS, etc) or on For the most generic at rest encryption of data, we recommend setting up your
the client side (or both). There are no current plans to add data encryption storage partitions on encrypted LUKS devices.
directly in Garage.
Implementing data encryption directly in Garage might make things simpler for If you are developping your own client software that makes use of S3 storage,
end users, but also raises many more questions, especially around key we recommend implementing data encryption directly on the client side and never
management: for encryption of data, where could Garage get the encryption keys transmitting plaintext data to Garage. This makes it easy to use an external
from ? If we encrypt data but keep the keys in a plaintext file next to them, untrusted storage provider if necessary.
it's useless. We probably don't want to have to manage secrets in garage as it
would be very hard to do in a secure way. Maybe integrate with an external Garage does support [SSE-C
system such as Hashicorp Vault? encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html),
an encryption mode of Amazon S3 where data is encrypted at rest using
encryption keys given by the client. The encryption keys are passed to the
server in a header in each request, to encrypt or decrypt data at the moment of
reading or writing. The server discards the key as soon as it has finished
using it for the request. This mode allows the data to be encrypted at rest by
Garage itself, but it requires support in the client software. It is also not
adapted to a model where the server is not trusted or assumed to be
compromised, as the server can easily know the encryption keys. Note however
that when using SSE-C encryption, the only Garage node that knows the
encryption key passed in a given request is the node to which the request is
directed (which can be a gateway node), so it is easy to have untrusted nodes
in the cluster as long as S3 API requests containing SSE-C encryption keys are
not directed to them.
Implementing automatic data encryption directly in Garage without client-side
management of keys (something like
[SSE-S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html))
could make things simpler for end users that don't want to setup LUKS, but also
raises many more questions, especially around key management: for encryption of
data, where could Garage get the encryption keys from? If we encrypt data but
keep the keys in a plaintext file next to them, it's useless. We probably don't
want to have to manage secrets in Garage as it would be very hard to do in a
secure way. At the time of speaking, there are no plans to implement this in
Garage.
# Adding data encryption using external tools # Adding data encryption using external tools

View file

@ -90,5 +90,6 @@ The following feature flags are available in v0.8.0:
| `kubernetes-discovery` | optional | Enable automatic registration and discovery<br>of cluster nodes through the Kubernetes API | | `kubernetes-discovery` | optional | Enable automatic registration and discovery<br>of cluster nodes through the Kubernetes API |
| `metrics` | *by default* | Enable collection of metrics in Prometheus format on the admin API | | `metrics` | *by default* | Enable collection of metrics in Prometheus format on the admin API |
| `telemetry-otlp` | optional | Enable collection of execution traces using OpenTelemetry | | `telemetry-otlp` | optional | Enable collection of execution traces using OpenTelemetry |
| `syslog` | optional | Enable logging to Syslog |
| `lmdb` | *by default* | Enable using LMDB to store Garage's metadata | | `lmdb` | *by default* | Enable using LMDB to store Garage's metadata |
| `sqlite` | optional | Enable using Sqlite3 to store Garage's metadata | | `sqlite` | *by default* | Enable using Sqlite3 to store Garage's metadata |

Some files were not shown because too many files have changed in this diff Show more