mirrors/garage
1
0
Fork 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2026-05-15 13:46:54 -04:00
Code Issues Releases Packages Wiki Activity
1,992 commits 30 branches 89 tags 51 MiB
Commit graph

1,179 commits

Author SHA1 Message Date
trinity-1686a
f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
Jill
84613e66a2
 		garage(tests): Remove RNG stuff 2022-02-11 10:50:55 +01:00
Jill
c8b30ebc79
 		garage(tests): Remove superfluous test 2022-02-10 17:55:51 +01:00
Jill
d7decda3f4
 		garage(tests): Add random suffix to created buckets. 2022-02-10 17:55:50 +01:00
Jill
cd13ea461b
 		garage(tests): Add some unsafe-usage doc and tweaks 2022-02-10 17:55:50 +01:00
Quentin Dufour
5d19f3d2d7
 		Add integration tests to Drone 2022-02-10 17:55:50 +01:00
Jill
3baa841d6f
 		tests: Fix garage integration test 2022-02-10 17:55:49 +01:00
Jill
dd407e7041
 		tests: Add garage integration tests (base) 2022-02-10 17:55:49 +01:00
Alex Auvolat
af261e1789 Fix a bug when a migration is followed by a rebalance 
Nodes would stabilize on different encoding formats for the values,
some having the pre-migration format and some having the post-migration
format. This would be reflected in the Merkle trees never converging
and thus having an infinite resync loop.
 2022-02-10 17:38:27 +01:00
Alex Auvolat
ae2f32baf1
 		Hide deleted key in bucket info (fix #211) 2022-02-02 17:12:48 +01:00
Alex Auvolat
c99f55c420
 		Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
Alex Auvolat
acdf893362
 		Fix partnumber 2022-01-25 12:25:23 +01:00
Alex Auvolat
338b1b83ee
 		Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
Alex Auvolat
6dab836f3a
 		Multipart improvements 
- support part_number for HeadObject
- add checks in complete_multipart_upload
 2022-01-24 21:04:40 +01:00
Alex Auvolat
513a6b15f9
 		Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00
Alex Auvolat
ea7fb901eb
 		Implement {Put,Get,Delete}BucketCors and CORS in general 
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
 2022-01-24 11:58:00 +01:00
Trinity Pointard
820924534a use clamp instead of min(max()) 2022-01-24 11:56:59 +01:00
Quentin Dufour
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
Quentin Dufour
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
trinity-1686a
e55fa38c99 Add date verification to presigned urls (#196) 
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-01-18 12:22:31 +01:00
trinity-1686a
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious 2022-01-17 15:50:24 +01:00
Jill
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193) 
Closes #164.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
 2022-01-17 11:18:40 +01:00
Jill
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) 
Closes #64.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
 2022-01-17 10:55:31 +01:00
Alex Auvolat
d4dd2e2640
 		Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
Alex Auvolat
9eb211948e
 		Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
Alex Auvolat
3ea8ca1b9e
 		Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
Alex Auvolat
f7349f4005
 		Add quotes in returned etags 2022-01-13 14:03:33 +01:00
Alex Auvolat
1ee8f596ee
 		Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
Alex Auvolat
6617a72220
 		Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Alex Auvolat
3770a34e3d
 		Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
Quentin
b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Alex Auvolat
3586c7257c
 		Clarify error message 2022-01-10 14:10:04 +01:00
Alex Auvolat
17ea28a438
 		Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
Alex Auvolat
7ee11f0eb6
 		Fix unit tests 2022-01-05 17:34:48 +01:00
Alex Auvolat
168a90dfb5
 		Fix some error codes 2022-01-05 17:07:36 +01:00
Alex Auvolat
fb1e31add0
 		Small CLI changes 2022-01-05 16:28:46 +01:00
Alex Auvolat
135858d067
 		Implement DeleteBucket 2022-01-05 16:28:19 +01:00
Alex Auvolat
8395030e48
 		Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex Auvolat
9431090b1e
 		Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
Alex Auvolat
677ab60cc1
 		Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex Auvolat
df35feba18
 		New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex Auvolat
1bcd6fabbd
 		New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex Auvolat
ba7f268b99
 		Rename and change query filters 2022-01-04 12:52:46 +01:00
Alex Auvolat
e59c23a69d
 		Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
Alex Auvolat
2140cd7205
 		Remove website redirects 2022-01-04 12:52:46 +01:00
Alex Auvolat
beeef4758e
 		Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex Auvolat
d8ab5bdc3e
 		New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex Auvolat
c7d5c73244
 		Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
Alex Auvolat
b76d0580a0
 		Fix forgotten flag 2022-01-04 12:47:28 +01:00
Alex Auvolat
87121dce9d
 		New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00