mirrors/garage
1
0
Fork 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2026-05-15 05:36:53 -04:00
Code Issues Releases Packages Wiki Activity
1,187 commits 30 branches 89 tags 51 MiB
Commit graph

1,187 commits

This branch
This branch
All branches
Author SHA1 Message Date
Alex Auvolat
dd5304f6fc
 		Replace logging crate pretty_env_logger by tracing_subscriber::fmt 2022-08-31 14:24:41 +02:00
Alex Auvolat
322dafc761
 		Try to fix clippy 2022-08-29 17:32:45 +02:00
Alex Auvolat
5d065b8a0f
 		cargo2nix fix to fetchCrateGit 2022-08-29 17:24:53 +02:00
Alex Auvolat
52749e28f7
 		Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-29 16:48:43 +02:00
Alex Auvolat
4da67b0035
 		Update drone signature 2022-08-29 16:48:31 +02:00
Alex Auvolat
1921f4f7e6
 		Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-29 16:45:05 +02:00
Alex Auvolat
ebc20a8798
 		Merge branch 'main' into lx-perf-improvements 2022-08-29 16:44:13 +02:00
Quentin Dufour
532eca7ff9
 		Add some documentation for Caddy 2022-08-12 10:33:41 +02:00
Quentin Dufour
2c7bae935a
 		Configure structopt to report the right version v0.7.2_ci-test-2 
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
 2022-08-11 10:21:45 +02:00
Quentin Dufour
8cd02639dc
 		drone: set TARGET env as needed by "to_s3" func 2022-08-03 11:19:26 +02:00
Alex Auvolat
e935861854
 		Factor out node request order selection logic & use in manager 2022-07-29 12:25:03 +02:00
Alex Auvolat
f0ee3056d3
 		Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
126b037307
 		update netapp 2022-07-29 12:25:03 +02:00
Alex Auvolat
33750c04ed
 		Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
68087ee13d
 		Fix clippy 2022-07-29 12:25:03 +02:00
Alex Auvolat
605a630333
 		Use streaming in block manager 2022-07-29 12:25:02 +02:00
Alex Auvolat
a35d4da721
 		update netapp to 0.5 2022-07-29 12:25:02 +02:00
Alex Auvolat
8e7e680afe
 		First adaptation to WIP netapp with streaming body 2022-07-29 12:25:02 +02:00
Alex Auvolat
16f6a1a65d
 		fix clippy 2022-07-29 12:24:49 +02:00
Alex Auvolat
ad35b18bb1
 		Faster chunker 2022-07-29 12:24:49 +02:00
Alex Auvolat
49154a78d8
 		Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex Auvolat
ff4771c36a
 		cargo fmt 2022-07-29 12:24:48 +02:00
Alex Auvolat
381eb9a5a1
 		Fix tests 2022-07-29 12:24:48 +02:00
Alex Auvolat
2cad656a03
 		More make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
0176da3ad2
 		Make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
40150527b8
 		Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex Auvolat
2f111e6b3d
 		Performance improvements: 
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
 2022-07-29 12:24:48 +02:00
Alex Auvolat
1b2e1296eb
 		Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
Quentin Dufour
a184f0d0b5
 		Migrate to nix-daemon builders 2022-07-29 08:37:33 +02:00
Quentin Dufour
fcb04843f7
 		Run clippy in nix, leveraging nix caching ability 2022-07-26 18:27:52 +02:00
Quentin Dufour
5fb8584247
 		Refactor default.nix to follow Nix Flakes patterns 2022-07-26 18:27:52 +02:00
Quentin Dufour
96561c48a1
 		Bump Nix image to 22.05 2022-07-26 18:27:52 +02:00
Quentin Dufour
a49d0ea19f
 		Fix: compile aarch64+armv6 as static binaries 2022-07-26 18:27:51 +02:00
Quentin Dufour
9c9e483375
 		Put log-lines in nix.conf 2022-07-26 18:27:51 +02:00
Quentin Dufour
76cb34a0ae
 		Fail if compiled binary is dynamic 2022-07-26 18:27:46 +02:00
Alex Auvolat
ac03fa7937
 		Uniformize tracing::* imports (hopefully fixes 32-bit build) 2022-07-15 18:31:19 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex Auvolat
aab34bfe54
 		add delays in k2v test_items_and_indices 2022-07-08 10:41:57 +02:00
Alex
fe3fa83de7 Publish k2v-client crate to crates.io (#337) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/337
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 18:27:25 +02:00
Alex Auvolat
b6d59ec19a
 		Fix poll item when item didn't change 2022-07-04 14:00:02 +02:00
Alex
0850bac874 Add poll command to k2v-cli (#335) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/335
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 12:45:32 +02:00
Alex
b74b533b7b Fix typo 2022-06-29 11:50:51 +02:00
Alex
996f2a6d58 Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/333
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-23 14:28:40 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00
Quentin Dufour
d544a0e0e0
 		Send CORS headers for all requests 2022-06-13 10:19:52 +02:00
Alex Auvolat
138e13071b
 		Fix garage_db build on 32-bit systems 2022-06-09 14:55:20 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Simon C
7eed3ceda9 docs: Add Trafik reverse proxy documentation 2022-06-07 16:16:52 +02:00
Simon C
4b8f48f3c5 docs: Fix title level 2022-06-07 13:32:52 +02:00
Simon C
7d3b5585f1 docs: Add link to facilitate navigation in the documentation 2022-06-07 09:38:59 +02:00