rdelaage
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,434 Commits 7 Branches 44 Tags 74 MiB
Commit Graph

46 Commits

Author SHA1 Message Date
Alexander Neumann aa0f874c8d s3: Simplify IsNotExist() 2017-05-31 21:23:01 +02:00
Alexander Neumann 5c59484d2b s3: Return only basename in List() 2017-05-31 21:22:55 +02:00
Alexander Neumann fa41183a53 s3: Add s3.layout option and layout auto detection 2017-05-15 23:37:02 +02:00
Alexander Neumann 61cade6222 Rename layout s3 -> s3legacy 2017-05-15 21:59:28 +02:00
Alexander Neumann 250a45ab15 s3: Move interfaces to function 2017-05-14 20:36:26 +02:00
Alexander Neumann 26c16b9fd3 s3: Correctly return token in Load 2017-05-14 20:19:12 +02:00
Alexander Neumann 0bd40bae6e s3: Prevent net/http client from closing the reader 2017-05-14 20:19:12 +02:00
Alexander Neumann 246ccf09b9 s3: add getRemainingSize 2017-05-14 20:19:12 +02:00
Alexander Neumann 9452f416bf s3: Use low level API for saving files 
benchmark                        old ns/op       new ns/op       delta
    BenchmarkBackendMinio/Save-4     184482294       40663344        -77.96%
    BenchmarkBackendS3/Save-4        35030825568     54475455819     +55.51%

    benchmark                        old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/Save-4     90.95        412.64       4.54x
    BenchmarkBackendS3/Save-4        0.48         0.31         0.65x

    benchmark                        old allocs     new allocs     delta
    BenchmarkBackendMinio/Save-4     631            560            -11.25%
    BenchmarkBackendS3/Save-4        646            584            -9.60%

    benchmark                        old bytes     new bytes     delta
    BenchmarkBackendMinio/Save-4     66818060      50735         -99.92%
    BenchmarkBackendS3/Save-4        66834000      73024         -99.89%
 2017-05-14 20:19:12 +02:00
Alexander Neumann 1e0e6ee573 s3: Use low-level API with a Range header for Load 
benchmark                                         old ns/op      new ns/op      delta
    BenchmarkBackendMinio/LoadFile-4                  9213315        11001787       +19.41%
    BenchmarkBackendMinio/LoadPartialFile-4           4176619        3479707        -16.69%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4391521        3139214        -28.52%
    BenchmarkBackendS3/LoadFile-4                     2886070905     2505907501     -13.17%
    BenchmarkBackendS3/LoadPartialFile-4              762702722      735694398      -3.54%
    BenchmarkBackendS3/LoadPartialFileOffset-4        789724328      1108989142     +40.43%

    benchmark                                         old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/LoadFile-4                  1821.21      1525.15      0.84x
    BenchmarkBackendMinio/LoadPartialFile-4           1004.49      1205.67      1.20x
    BenchmarkBackendMinio/LoadPartialFileOffset-4     955.34       1336.45      1.40x
    BenchmarkBackendS3/LoadFile-4                     5.81         6.70         1.15x
    BenchmarkBackendS3/LoadPartialFile-4              5.50         5.70         1.04x
    BenchmarkBackendS3/LoadPartialFileOffset-4        5.31         3.78         0.71x

    benchmark                                         old allocs     new allocs     delta
    BenchmarkBackendMinio/LoadFile-4                  406            204            -49.75%
    BenchmarkBackendMinio/LoadPartialFile-4           225            206            -8.44%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     227            207            -8.81%
    BenchmarkBackendS3/LoadFile-4                     600            388            -35.33%
    BenchmarkBackendS3/LoadPartialFile-4              416            302            -27.40%
    BenchmarkBackendS3/LoadPartialFileOffset-4        417            303            -27.34%

    benchmark                                         old bytes     new bytes     delta
    BenchmarkBackendMinio/LoadFile-4                  29475         13904         -52.83%
    BenchmarkBackendMinio/LoadPartialFile-4           4218838       13958         -99.67%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4219175       14332         -99.66%
    BenchmarkBackendS3/LoadFile-4                     114152        97424         -14.65%
    BenchmarkBackendS3/LoadPartialFile-4              4265416       56212         -98.68%
    BenchmarkBackendS3/LoadPartialFileOffset-4        4266520       56308         -98.68%
 2017-05-14 20:19:12 +02:00
Alexander Neumann a963052d64 Add custom HTTP transport 2017-05-11 21:51:30 +02:00
Alexander Neumann 28968caf33 s3: Correct prefix for layout 2017-04-17 20:06:55 +02:00
Alexander Neumann b942f61272 s3: Reduce connection limit, prevents timeouts 2017-04-17 20:06:55 +02:00
Alexander Neumann 541484d142 s3: Use Layout 2017-04-14 00:00:02 +02:00
trbs f1ba45723c introduce RWLock for caching obj.Stat() 2017-03-14 23:05:51 +01:00
trbs b523eef294 Cache size of last ReadAt on S3 for performance 
Each obj.Stat() call adds another request to the S3 endpoint
for some commands a lot of ReadAt calls are made for the same
object in S3. This patch essentially cuts the number of calls
to S3 in this case in half. Speeding up the progress and lowering
costs to S3.
 2017-03-14 14:02:50 +01:00
Alexander Neumann 83538c745a Merge pull request #792 from restic/fix-791 
s3: Increase MaxIdleConnsPerHost
 2017-02-11 11:20:20 +01:00
Alexander Neumann f266741f40 s3/rest: raise connection limit to 40 2017-02-11 10:40:51 +01:00
Alexander Neumann 7a0303f7ae s3: Make sure to return connection token 2017-02-10 20:49:37 +01:00
Alexander Neumann 64165ea4c8 s3: Hold connection semaphore until Close() 2017-02-10 20:17:52 +01:00
Alexander Neumann c8fc789393 Use non-formatting functions of errors for strings 
Commands used:

    $ gofmt -w -r 'errors.Fatalf(x) -> errors.Fatal(x)' src
    $ gofmt -w -r 'errors.Errorf(x) -> errors.New(x)' src

Closes #789
 2017-02-10 19:39:49 +01:00
Alexander Neumann 36dee7d892 s3: Increase MaxIdleConnsPerHost 2017-02-10 19:25:42 +01:00
Alexander Neumann b9bddeff39 Normalise the backend API 
This makes the following changes, before:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(t FileType, name string) (bool, error)

        // Remove removes a File with type t and name.
        Remove(t FileType, name string) error
    }

After:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(h Handle) (bool, error)

        // Remove removes a File with type t and name.
        Remove(h Handle) error
    }
 2017-01-26 22:02:22 +01:00
Alexander Neumann 03292d10cc backend: Rename Get() -> Load() 2017-01-23 18:11:10 +01:00
Alexander Neumann cfc9e8b2fa backends: Remove Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann 82d9163955 backend: Ensure Reader is closed on error 2017-01-23 17:54:12 +01:00
Alexander Neumann 05afedd950 Add backend.Get() 2017-01-23 17:54:11 +01:00
Alexander Neumann 9b48da5b4e Change backend Save() function signature 2017-01-23 17:54:11 +01:00
Jan Niehusmann c5897e0d62 Update debug message 
Since client.BucketExists was changed to return a separate 'found' value, instead of reporting an error when the bucket doesn't exist, the error code path does no longer imply a call to client.MakeBucket. So the second part of the debug message, "...trying to create the bucket" doesn't apply any more.
Also, changed the name of the return value from 'ok' to 'found', matching the API documentation at https://docs.minio.io/docs/golang-client-api-reference#BucketExists.
 2016-12-05 23:12:30 +01:00
Christian Kemper a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann 4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann bc42dbdf87 Create package restic/errors 2016-09-03 21:10:24 +02:00
Alexander Neumann 5e3a41dbd2 Rename struct member FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann cc6a8b6e15 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 4a0f77650b Wrap errors #2 2016-08-29 21:54:50 +02:00
Alexander Neumann b06845c545 Always use errors.Cause() for testing error values 2016-08-29 19:52:03 +02:00
Alexander Neumann 444a268ce0 Replace stdlib errors with github.com/pkg/errors 2016-08-29 19:23:50 +02:00
Alexander Neumann ebd3723a06 Properly close the minio object on Stat() 
Closes #544
 2016-08-21 16:15:41 +02:00
Alexander Neumann 06b23edb39 Fix code for newer minio-go 2016-08-21 16:14:58 +02:00
Alexander Neumann 2310773798 Compute negative offsets ourselves in the s3 backend 2016-08-16 21:30:14 +02:00
Alexander Neumann 94d157d97a Introduce interface pack.Loader 2016-08-16 21:30:14 +02:00
Ben Agricola edb1843f24 Explicitly Close() obj after ReadFull() 
Signed-off-by: Ben Agricola <bagricola@squiz.co.uk>
 2016-07-29 14:18:02 +01:00
Alexander Neumann 902f619a06 Fix call to minio.New() 
The last parameter changed semantics from `insecure` to `secure`.
 2016-06-08 21:33:18 +02:00
Alexander Neumann 3f8da47a0c Fix restic s3 backend for new minio-go version 2016-05-07 23:38:41 +02:00
Christian Kemper c2716755f1 Added missing handle to the s3.Stat log message output 2016-02-21 09:59:27 -08:00
Alexander Neumann c0bd660a9e Rename package 
* github.com/restic/restic -> restic
 2016-02-20 17:31:21 +01:00