rdelaage
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,490 Commits 7 Branches 44 Tags 74 MiB
Commit Graph

59 Commits

Author SHA1 Message Date
Alexander Neumann 73b296918b s3: Reorder debug messages 
This way the semaphore token acquisition can be observed in the debug
log.
 2017-06-11 20:49:53 +02:00
Alexander Neumann 08e1d9ffad s3: Switch back to high-level API, limit connections 2017-06-11 20:48:42 +02:00
Alexander Neumann a4e8dc3371 s3: Improve error message in debug log 2017-06-11 11:22:25 +02:00
Alexander Neumann 6301620428 s3: Add more debug logs 2017-06-11 11:22:25 +02:00
Alexander Neumann 8d4417ec92 Remove unused code/variables 2017-06-11 09:29:53 +02:00
Alexander Neumann 1f0916b01b Merge pull request #1004 from restic/add-migrate-s3 
Add 'migrate' command, change s3 layout
 2017-06-08 20:48:27 +02:00
Alexander Neumann 04ded881f6 s3: Change the default layout to "default" 2017-06-07 23:08:20 +02:00
Alexander Neumann 4f9bf5312b Add migrate 
This commits adds a 'migrate' command and a migration to move s3
repositories from the 's3legacy' to the 'default' layout.
 2017-06-07 23:08:02 +02:00
Alexander Neumann b8b5c8e8c9 s3: Rename struct to Backend 2017-06-07 21:59:01 +02:00
Alexander Neumann a46baf7685 s3: Remove cache 2017-06-07 20:51:45 +02:00
Alexander Neumann 550e1feaec Merge pull request #999 from restic/backend-use-semaphore 
backends: Use new semaphore
 2017-06-07 19:48:32 +02:00
Alexander Neumann 683ebef6c6 s3: Use semaphore 2017-06-06 00:17:39 +02:00
Alexander Neumann 16fcd07110 Add a Context to the backend 2017-06-04 14:02:44 +02:00
Alexander Neumann aa0f874c8d s3: Simplify IsNotExist() 2017-05-31 21:23:01 +02:00
Alexander Neumann 5c59484d2b s3: Return only basename in List() 2017-05-31 21:22:55 +02:00
Alexander Neumann fa41183a53 s3: Add s3.layout option and layout auto detection 2017-05-15 23:37:02 +02:00
Alexander Neumann 61cade6222 Rename layout s3 -> s3legacy 2017-05-15 21:59:28 +02:00
Alexander Neumann 250a45ab15 s3: Move interfaces to function 2017-05-14 20:36:26 +02:00
Alexander Neumann 26c16b9fd3 s3: Correctly return token in Load 2017-05-14 20:19:12 +02:00
Alexander Neumann 0bd40bae6e s3: Prevent net/http client from closing the reader 2017-05-14 20:19:12 +02:00
Alexander Neumann 246ccf09b9 s3: add getRemainingSize 2017-05-14 20:19:12 +02:00
Alexander Neumann 9452f416bf s3: Use low level API for saving files 
benchmark                        old ns/op       new ns/op       delta
    BenchmarkBackendMinio/Save-4     184482294       40663344        -77.96%
    BenchmarkBackendS3/Save-4        35030825568     54475455819     +55.51%

    benchmark                        old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/Save-4     90.95        412.64       4.54x
    BenchmarkBackendS3/Save-4        0.48         0.31         0.65x

    benchmark                        old allocs     new allocs     delta
    BenchmarkBackendMinio/Save-4     631            560            -11.25%
    BenchmarkBackendS3/Save-4        646            584            -9.60%

    benchmark                        old bytes     new bytes     delta
    BenchmarkBackendMinio/Save-4     66818060      50735         -99.92%
    BenchmarkBackendS3/Save-4        66834000      73024         -99.89%
 2017-05-14 20:19:12 +02:00
Alexander Neumann 1e0e6ee573 s3: Use low-level API with a Range header for Load 
benchmark                                         old ns/op      new ns/op      delta
    BenchmarkBackendMinio/LoadFile-4                  9213315        11001787       +19.41%
    BenchmarkBackendMinio/LoadPartialFile-4           4176619        3479707        -16.69%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4391521        3139214        -28.52%
    BenchmarkBackendS3/LoadFile-4                     2886070905     2505907501     -13.17%
    BenchmarkBackendS3/LoadPartialFile-4              762702722      735694398      -3.54%
    BenchmarkBackendS3/LoadPartialFileOffset-4        789724328      1108989142     +40.43%

    benchmark                                         old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/LoadFile-4                  1821.21      1525.15      0.84x
    BenchmarkBackendMinio/LoadPartialFile-4           1004.49      1205.67      1.20x
    BenchmarkBackendMinio/LoadPartialFileOffset-4     955.34       1336.45      1.40x
    BenchmarkBackendS3/LoadFile-4                     5.81         6.70         1.15x
    BenchmarkBackendS3/LoadPartialFile-4              5.50         5.70         1.04x
    BenchmarkBackendS3/LoadPartialFileOffset-4        5.31         3.78         0.71x

    benchmark                                         old allocs     new allocs     delta
    BenchmarkBackendMinio/LoadFile-4                  406            204            -49.75%
    BenchmarkBackendMinio/LoadPartialFile-4           225            206            -8.44%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     227            207            -8.81%
    BenchmarkBackendS3/LoadFile-4                     600            388            -35.33%
    BenchmarkBackendS3/LoadPartialFile-4              416            302            -27.40%
    BenchmarkBackendS3/LoadPartialFileOffset-4        417            303            -27.34%

    benchmark                                         old bytes     new bytes     delta
    BenchmarkBackendMinio/LoadFile-4                  29475         13904         -52.83%
    BenchmarkBackendMinio/LoadPartialFile-4           4218838       13958         -99.67%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4219175       14332         -99.66%
    BenchmarkBackendS3/LoadFile-4                     114152        97424         -14.65%
    BenchmarkBackendS3/LoadPartialFile-4              4265416       56212         -98.68%
    BenchmarkBackendS3/LoadPartialFileOffset-4        4266520       56308         -98.68%
 2017-05-14 20:19:12 +02:00
Alexander Neumann a963052d64 Add custom HTTP transport 2017-05-11 21:51:30 +02:00
Alexander Neumann 28968caf33 s3: Correct prefix for layout 2017-04-17 20:06:55 +02:00
Alexander Neumann b942f61272 s3: Reduce connection limit, prevents timeouts 2017-04-17 20:06:55 +02:00
Alexander Neumann 541484d142 s3: Use Layout 2017-04-14 00:00:02 +02:00
trbs f1ba45723c introduce RWLock for caching obj.Stat() 2017-03-14 23:05:51 +01:00
trbs b523eef294 Cache size of last ReadAt on S3 for performance 
Each obj.Stat() call adds another request to the S3 endpoint
for some commands a lot of ReadAt calls are made for the same
object in S3. This patch essentially cuts the number of calls
to S3 in this case in half. Speeding up the progress and lowering
costs to S3.
 2017-03-14 14:02:50 +01:00
Alexander Neumann 83538c745a Merge pull request #792 from restic/fix-791 
s3: Increase MaxIdleConnsPerHost
 2017-02-11 11:20:20 +01:00
Alexander Neumann f266741f40 s3/rest: raise connection limit to 40 2017-02-11 10:40:51 +01:00
Alexander Neumann 7a0303f7ae s3: Make sure to return connection token 2017-02-10 20:49:37 +01:00
Alexander Neumann 64165ea4c8 s3: Hold connection semaphore until Close() 2017-02-10 20:17:52 +01:00
Alexander Neumann c8fc789393 Use non-formatting functions of errors for strings 
Commands used:

    $ gofmt -w -r 'errors.Fatalf(x) -> errors.Fatal(x)' src
    $ gofmt -w -r 'errors.Errorf(x) -> errors.New(x)' src

Closes #789
 2017-02-10 19:39:49 +01:00
Alexander Neumann 36dee7d892 s3: Increase MaxIdleConnsPerHost 2017-02-10 19:25:42 +01:00
Alexander Neumann b9bddeff39 Normalise the backend API 
This makes the following changes, before:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(t FileType, name string) (bool, error)

        // Remove removes a File with type t and name.
        Remove(t FileType, name string) error
    }

After:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(h Handle) (bool, error)

        // Remove removes a File with type t and name.
        Remove(h Handle) error
    }
 2017-01-26 22:02:22 +01:00
Alexander Neumann 03292d10cc backend: Rename Get() -> Load() 2017-01-23 18:11:10 +01:00
Alexander Neumann cfc9e8b2fa backends: Remove Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann 82d9163955 backend: Ensure Reader is closed on error 2017-01-23 17:54:12 +01:00
Alexander Neumann 05afedd950 Add backend.Get() 2017-01-23 17:54:11 +01:00
Alexander Neumann 9b48da5b4e Change backend Save() function signature 2017-01-23 17:54:11 +01:00
Jan Niehusmann c5897e0d62 Update debug message 
Since client.BucketExists was changed to return a separate 'found' value, instead of reporting an error when the bucket doesn't exist, the error code path does no longer imply a call to client.MakeBucket. So the second part of the debug message, "...trying to create the bucket" doesn't apply any more.
Also, changed the name of the return value from 'ok' to 'found', matching the API documentation at https://docs.minio.io/docs/golang-client-api-reference#BucketExists.
 2016-12-05 23:12:30 +01:00
Christian Kemper a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann 4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann bc42dbdf87 Create package restic/errors 2016-09-03 21:10:24 +02:00
Alexander Neumann 5e3a41dbd2 Rename struct member FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann cc6a8b6e15 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 4a0f77650b Wrap errors #2 2016-08-29 21:54:50 +02:00
Alexander Neumann b06845c545 Always use errors.Cause() for testing error values 2016-08-29 19:52:03 +02:00
Alexander Neumann 444a268ce0 Replace stdlib errors with github.com/pkg/errors 2016-08-29 19:23:50 +02:00