Alexander Neumann
28a4a35625
Allow migrate to run althoug check failed
2017-07-02 10:29:41 +02:00
Alexander Neumann
c9cc724b31
s3: Split Create() from Open()
2017-06-17 22:15:58 +02:00
Alexander Neumann
0d3674245b
Merge pull request #1043 from restic/fix-gcs
...
s3: Fix GCS
2017-06-17 10:35:10 +02:00
Alexander Neumann
82b21cdf4a
Merge pull request #1027 from restic/s3-set-retry
...
s3: Allow setting the number of retries for minio-go
2017-06-17 10:34:36 +02:00
Alexander Neumann
6a948d5afd
s3: Fix backend for Google Cloud Storage
2017-06-16 11:25:06 +02:00
Alexander Neumann
ea66ae0811
s3: Fix IsNotExist()
2017-06-16 10:54:46 +02:00
Alexander Neumann
98ae7b1210
s3: Save config in backend
2017-06-15 16:41:09 +02:00
Alexander Neumann
51877cecf7
s3: Prevent closing of the reader for GCS
2017-06-15 16:39:42 +02:00
Alexander Neumann
9053b2000b
s3: Delete ignores error if the object doesn't exist
2017-06-15 16:27:19 +02:00
Alexander Neumann
cf65893c4b
s3: Allow setting the number of retries for minio-go
...
https://github.com/restic/restic/issues/1013#issuecomment-307883970
2017-06-12 21:09:37 +02:00
Alexander Neumann
73b296918b
s3: Reorder debug messages
...
This way the semaphore token acquisition can be observed in the debug
log.
2017-06-11 20:49:53 +02:00
Alexander Neumann
08e1d9ffad
s3: Switch back to high-level API, limit connections
2017-06-11 20:48:42 +02:00
Alexander Neumann
a4e8dc3371
s3: Improve error message in debug log
2017-06-11 11:22:25 +02:00
Alexander Neumann
6301620428
s3: Add more debug logs
2017-06-11 11:22:25 +02:00
Alexander Neumann
8d4417ec92
Remove unused code/variables
2017-06-11 09:29:53 +02:00
Alexander Neumann
1f0916b01b
Merge pull request #1004 from restic/add-migrate-s3
...
Add 'migrate' command, change s3 layout
2017-06-08 20:48:27 +02:00
Alexander Neumann
04ded881f6
s3: Change the default layout to "default"
2017-06-07 23:08:20 +02:00
Alexander Neumann
4f9bf5312b
Add migrate
...
This commits adds a 'migrate' command and a migration to move s3
repositories from the 's3legacy' to the 'default' layout.
2017-06-07 23:08:02 +02:00
Alexander Neumann
b8b5c8e8c9
s3: Rename struct to Backend
2017-06-07 21:59:01 +02:00
Alexander Neumann
a46baf7685
s3: Remove cache
2017-06-07 20:51:45 +02:00
Alexander Neumann
550e1feaec
Merge pull request #999 from restic/backend-use-semaphore
...
backends: Use new semaphore
2017-06-07 19:48:32 +02:00
Alexander Neumann
683ebef6c6
s3: Use semaphore
2017-06-06 00:17:39 +02:00
Alexander Neumann
16fcd07110
Add a Context to the backend
2017-06-04 14:02:44 +02:00
Alexander Neumann
aa0f874c8d
s3: Simplify IsNotExist()
2017-05-31 21:23:01 +02:00
Alexander Neumann
5c59484d2b
s3: Return only basename in List()
2017-05-31 21:22:55 +02:00
Alexander Neumann
fa41183a53
s3: Add s3.layout option and layout auto detection
2017-05-15 23:37:02 +02:00
Alexander Neumann
61cade6222
Rename layout s3 -> s3legacy
2017-05-15 21:59:28 +02:00
Alexander Neumann
250a45ab15
s3: Move interfaces to function
2017-05-14 20:36:26 +02:00
Alexander Neumann
26c16b9fd3
s3: Correctly return token in Load
2017-05-14 20:19:12 +02:00
Alexander Neumann
0bd40bae6e
s3: Prevent net/http client from closing the reader
2017-05-14 20:19:12 +02:00
Alexander Neumann
246ccf09b9
s3: add getRemainingSize
2017-05-14 20:19:12 +02:00
Alexander Neumann
9452f416bf
s3: Use low level API for saving files
...
benchmark old ns/op new ns/op delta
BenchmarkBackendMinio/Save-4 184482294 40663344 -77.96%
BenchmarkBackendS3/Save-4 35030825568 54475455819 +55.51%
benchmark old MB/s new MB/s speedup
BenchmarkBackendMinio/Save-4 90.95 412.64 4.54x
BenchmarkBackendS3/Save-4 0.48 0.31 0.65x
benchmark old allocs new allocs delta
BenchmarkBackendMinio/Save-4 631 560 -11.25%
BenchmarkBackendS3/Save-4 646 584 -9.60%
benchmark old bytes new bytes delta
BenchmarkBackendMinio/Save-4 66818060 50735 -99.92%
BenchmarkBackendS3/Save-4 66834000 73024 -99.89%
2017-05-14 20:19:12 +02:00
Alexander Neumann
1e0e6ee573
s3: Use low-level API with a Range header for Load
...
benchmark old ns/op new ns/op delta
BenchmarkBackendMinio/LoadFile-4 9213315 11001787 +19.41%
BenchmarkBackendMinio/LoadPartialFile-4 4176619 3479707 -16.69%
BenchmarkBackendMinio/LoadPartialFileOffset-4 4391521 3139214 -28.52%
BenchmarkBackendS3/LoadFile-4 2886070905 2505907501 -13.17%
BenchmarkBackendS3/LoadPartialFile-4 762702722 735694398 -3.54%
BenchmarkBackendS3/LoadPartialFileOffset-4 789724328 1108989142 +40.43%
benchmark old MB/s new MB/s speedup
BenchmarkBackendMinio/LoadFile-4 1821.21 1525.15 0.84x
BenchmarkBackendMinio/LoadPartialFile-4 1004.49 1205.67 1.20x
BenchmarkBackendMinio/LoadPartialFileOffset-4 955.34 1336.45 1.40x
BenchmarkBackendS3/LoadFile-4 5.81 6.70 1.15x
BenchmarkBackendS3/LoadPartialFile-4 5.50 5.70 1.04x
BenchmarkBackendS3/LoadPartialFileOffset-4 5.31 3.78 0.71x
benchmark old allocs new allocs delta
BenchmarkBackendMinio/LoadFile-4 406 204 -49.75%
BenchmarkBackendMinio/LoadPartialFile-4 225 206 -8.44%
BenchmarkBackendMinio/LoadPartialFileOffset-4 227 207 -8.81%
BenchmarkBackendS3/LoadFile-4 600 388 -35.33%
BenchmarkBackendS3/LoadPartialFile-4 416 302 -27.40%
BenchmarkBackendS3/LoadPartialFileOffset-4 417 303 -27.34%
benchmark old bytes new bytes delta
BenchmarkBackendMinio/LoadFile-4 29475 13904 -52.83%
BenchmarkBackendMinio/LoadPartialFile-4 4218838 13958 -99.67%
BenchmarkBackendMinio/LoadPartialFileOffset-4 4219175 14332 -99.66%
BenchmarkBackendS3/LoadFile-4 114152 97424 -14.65%
BenchmarkBackendS3/LoadPartialFile-4 4265416 56212 -98.68%
BenchmarkBackendS3/LoadPartialFileOffset-4 4266520 56308 -98.68%
2017-05-14 20:19:12 +02:00
Alexander Neumann
a963052d64
Add custom HTTP transport
2017-05-11 21:51:30 +02:00
Alexander Neumann
28968caf33
s3: Correct prefix for layout
2017-04-17 20:06:55 +02:00
Alexander Neumann
b942f61272
s3: Reduce connection limit, prevents timeouts
2017-04-17 20:06:55 +02:00
Alexander Neumann
541484d142
s3: Use Layout
2017-04-14 00:00:02 +02:00
trbs
f1ba45723c
introduce RWLock for caching obj.Stat()
2017-03-14 23:05:51 +01:00
trbs
b523eef294
Cache size of last ReadAt on S3 for performance
...
Each obj.Stat() call adds another request to the S3 endpoint
for some commands a lot of ReadAt calls are made for the same
object in S3. This patch essentially cuts the number of calls
to S3 in this case in half. Speeding up the progress and lowering
costs to S3.
2017-03-14 14:02:50 +01:00
Alexander Neumann
83538c745a
Merge pull request #792 from restic/fix-791
...
s3: Increase MaxIdleConnsPerHost
2017-02-11 11:20:20 +01:00
Alexander Neumann
f266741f40
s3/rest: raise connection limit to 40
2017-02-11 10:40:51 +01:00
Alexander Neumann
7a0303f7ae
s3: Make sure to return connection token
2017-02-10 20:49:37 +01:00
Alexander Neumann
64165ea4c8
s3: Hold connection semaphore until Close()
2017-02-10 20:17:52 +01:00
Alexander Neumann
c8fc789393
Use non-formatting functions of errors for strings
...
Commands used:
$ gofmt -w -r 'errors.Fatalf(x) -> errors.Fatal(x)' src
$ gofmt -w -r 'errors.Errorf(x) -> errors.New(x)' src
Closes #789
2017-02-10 19:39:49 +01:00
Alexander Neumann
36dee7d892
s3: Increase MaxIdleConnsPerHost
2017-02-10 19:25:42 +01:00
Alexander Neumann
b9bddeff39
Normalise the backend API
...
This makes the following changes, before:
type backend interface {
// Test a boolean value whether a File with the name and type exists.
Test(t FileType, name string) (bool, error)
// Remove removes a File with type t and name.
Remove(t FileType, name string) error
}
After:
type backend interface {
// Test a boolean value whether a File with the name and type exists.
Test(h Handle) (bool, error)
// Remove removes a File with type t and name.
Remove(h Handle) error
}
2017-01-26 22:02:22 +01:00
Alexander Neumann
03292d10cc
backend: Rename Get() -> Load()
2017-01-23 18:11:10 +01:00
Alexander Neumann
cfc9e8b2fa
backends: Remove Load()
2017-01-23 17:54:12 +01:00
Alexander Neumann
82d9163955
backend: Ensure Reader is closed on error
2017-01-23 17:54:12 +01:00
Alexander Neumann
05afedd950
Add backend.Get()
2017-01-23 17:54:11 +01:00