mirrors/garage
1
0
Fork 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2026-05-15 13:46:54 -04:00
Code Issues Releases Packages Wiki Activity
2,773 commits 30 branches 89 tags 51 MiB
Commit graph

1,598 commits

Author SHA1 Message Date
Alex Auvolat
7b474855e3
 		Make background runner terminate correctly 2022-05-17 11:38:31 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat
def78c5e6f
 		Update netapp to 0.4.4, fix #300 2022-05-09 12:08:47 +02:00
Alex
277a20ec44 Fix layout show to not show changes when there are no changes (#297) 
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-09 11:14:55 +02:00
Alex Auvolat
617f28bfa4
 		Correct small formatting issue 2022-05-05 14:21:57 +02:00
Mendes
948ff93cf1 Corrected the warnings and errors issued by cargo clippy 2022-05-01 16:05:39 +02:00
Alex Auvolat
2aeaddd5e2
 		Apply cargo fmt 2022-05-01 09:57:05 +02:00
Alex Auvolat
c1d1646c4d
 		Change the way new layout assignations are computed. 
The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.

This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>
 2022-05-01 09:54:19 +02:00
Alex Auvolat
c9ef3e461b
 		fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat
c93008d333
 		Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat
e5341ca47b
 		Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom
a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Alex Auvolat
5d404dcd54
 		Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
cb5836d53c Bring maximum exponential backoff time down from 16h to 1h 2022-04-07 11:49:29 +02:00
Alex Auvolat
0091002ef2
 		New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex Auvolat
8f9cf3a5d1
 		fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat
913f7754bb
 		Add blocks in errored state to garage stats 2022-03-28 15:47:23 +02:00
Alex Auvolat
42dde54126
 		Log admin GET requests at debug level instead of info 
to reduce noise in logs
 2022-03-28 15:46:52 +02:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
509d256c58
 		Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat
2814d41842
 		Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex Auvolat
7e0e2ffda2
 		Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat
413ab0eaed
 		Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat
3dc9214172
 		Add lots of comments on how the resync queue works 
(I don't really want to change/refactor that code though)
 2022-03-23 10:25:39 +01:00
Alex Auvolat
077dd1cde9
 		Clippy 2022-03-23 10:25:39 +01:00
Alex Auvolat
e480aaf338
 		Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex Auvolat
8fd6745745
 		Move block RC code to separate rc.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c3982a90b6
 		Move DataBlock out of manager.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c1d9854d2c
 		Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a
8565f7dc31 cleanup 2022-03-23 10:22:37 +01:00
trinity-1686a
8db6b84559 add test for create bucket and put website with streaming signature 2022-03-23 10:22:37 +01:00
trinity-1686a
1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
KokaKiwi
e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
KokaKiwi
98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00
Alex Auvolat
db46cdef79
 		Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex Auvolat
ba6b56ae68
 		Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex Auvolat
0af314b295
 		Add comment for fsync 2022-03-14 11:54:00 +01:00
Alex Auvolat
d78bf379fb
 		Fix resync queue to not drop items 2022-03-14 11:51:37 +01:00
Alex Auvolat
f7e6f4616f
 		Spawn a single resync worker 2022-03-14 11:51:37 +01:00
Alex Auvolat
dc5ec4ecf9
 		Add appropriate fsync() calls in write_block 
to ensure that data is persisted properly
 2022-03-14 11:51:32 +01:00
Alex Auvolat
fe62d01b7e
 		Implement exponential backoff for resync retries 2022-03-14 11:41:20 +01:00
Alex Auvolat
9b2b531f4d
 		Make admin server optional 2022-03-14 10:54:25 +01:00
Alex Auvolat
2377a92f6b
 		Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex Auvolat
203e8d2c34
 		Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat
f869ca625d
 		Add spans to table calls, change span names in RPC 2022-03-14 10:54:12 +01:00
Alex Auvolat
0cc31ee169
 		add missing netapp telemetry feature 2022-03-14 10:54:11 +01:00
Alex Auvolat
dc8d0496cc
 		Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex Auvolat
d9a35359bf
 		Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex Auvolat
2a5609b292
 		Add metrics to API endpoint 2022-03-14 10:53:36 +01:00