Commit Graph

300 Commits

Author SHA1 Message Date
Alexander Neumann 2369da158f Merge pull request #1592 from ncw/helpful-tests
Make backend tests more helpful
2018-01-28 10:09:35 +01:00
Nick Craig-Wood fb62da1748 Make backend tests more helpful
* In TestList check that backend is empty first
  * Improve error message in TestBackend
2018-01-27 21:36:35 +00:00
Alexander Neumann 5dc8d3588d GS: Use generic http transport
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
2018-01-27 20:12:34 +01:00
Alexander Neumann c34db983d8 Read TLS client cert and key from the same file 2018-01-27 14:02:01 +01:00
Bryce Chidester e805b968b1 Support for TLS client certificate authentication
This adds --tls-client-cert and --tls-client-key parameters and enables use
of that certificate/key pair when connecting to https servers.
2018-01-27 13:18:22 +01:00
Alexander Neumann f99c95c766 archiver: Fix intermediate index upload
A user discovered[1] that when the backup finishes during the upload of
an intermediate index, the upload is cancelled and the index never fully
saved, but the snapshot is saved and the backup finalizes without an
error. This lead to a situation where a snapshot references data that is
contained in the repo, but not referenced in any index, leading to
strange error messages.

This commit uses a dedicated context to signal the intermediate index
uploading routine to terminate after the last index has been uploaded.
This way, an upload running when the backup finishes is completed before
the routine terminates and the snapshot is saved.

[1] https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
2018-01-26 22:01:07 +01:00
Alexander Neumann cccb2fc7e7 Merge pull request #1583 from restic/close-open-backend-files
Close backend files in case of errors
2018-01-26 21:57:28 +01:00
Alexander Neumann ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log
Optimize debug logs
2018-01-26 21:57:18 +01:00
Alexander Neumann e11a183578 Merge pull request #1588 from restic/fix-sftp-without-tty
sftp: Allow running ssh without a tty
2018-01-26 21:56:41 +01:00
Alexander Neumann 6eb2d76435 index: Lower parallel load to 4 2018-01-26 21:10:38 +01:00
Alexander Neumann f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann 7719cf88d9 b2: Check timeout 2018-01-26 21:07:05 +01:00
Alexander Neumann 00e905ebe6 sftp: Allow running ssh without a tty 2018-01-26 19:21:14 +01:00
Alexander Neumann 909d9273cc Close backend files in case of errors 2018-01-25 21:05:57 +01:00
Alexander Neumann 663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann ed99f53786 debug: Auto-shorten IDs passed as parameters 2018-01-25 20:49:26 +01:00
Igor Fedorenko abc4027083 Use errors.Cause in backend TestListCancel
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-25 08:53:50 -05:00
Alexander Neumann 9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids
More optimizations to avoid calling Index.Lookup()
2018-01-24 20:53:30 +01:00
Alexander Neumann be205563be Merge pull request #1575 from ifedorenko/1567_optimize-repository-ListPack
Optimize Repository.ListPack()
2018-01-24 20:01:15 +01:00
Alexander Neumann fc6c341b26 Merge pull request #1574 from ifedorenko/1567_optimize-pack-readHeader
Optimize pack readHeader() implementation
2018-01-24 19:54:14 +01:00
Alexander Neumann f86c141b83 Merge pull request #1577 from EdwardBetts/patch-1
Correct a spelling mistake.
2018-01-24 19:44:49 +01:00
Edward Betts b5c23695c9
Correct a spelling mistake. 2018-01-24 12:09:41 +00:00
Igor Fedorenko 0084e42cb6 Optimize Repository.ListPack()
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:39:51 -05:00
Matthew Dawson fe33c05a20
debug/log: Add benchmarks for calling the logging function
Add some benchmarks for calling Log, both with a static string
along with calling the ID.Str and ID.String functions.
2018-01-23 22:30:00 -05:00
Matthew Dawson 3789e55e20
repostiory/index: Remove logging from Lookup function.
The logging in these functions double the time they take to execute.
However, it is only really useful on failures, which are better
reported by the calling functions.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              897           395           -55.96%
BenchmarkMasterIndexLookupMultipleIndex-6            2001          1090          -45.53%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       492           215           -56.30%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     1649          912           -44.69%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              1              -88.89%
BenchmarkMasterIndexLookupMultipleIndex-6            19             1              -94.74%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       6              0              -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     16             0              -100.00%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           96            -40.00%
BenchmarkMasterIndexLookupMultipleIndex-6            240           96            -60.00%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       48            0             -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     128           0             -100.00%
2018-01-23 22:28:38 -05:00
Matthew Dawson 3a16148447
archiver/archiver: Use Index.Has() instead of Index.Lookup() in isKnownBlob
Index.Has() is a faster then Index.Lookup() for checking if a blob exists
in the index.  As the returned data is never used, this avoids a ton
of allocations.
2018-01-23 22:26:10 -05:00
Matthew Dawson df2c03a6a4
repository/master_index: Optimize Index.Lookup()
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
2018-01-23 22:25:56 -05:00
Matthew Dawson ebce4b2581
repository/index: Speed up benchmarks and tests
When setting up the index used for benchmarking, use math/rand instead of
crypto/rand since the generated ids don't need to be evenly distributed,
and not be secure against guessing.  As such, use a different random id
function (only available during tests) that uses math/rand instead.
2018-01-23 22:21:05 -05:00
Igor Fedorenko 953f3d55ee Optimize pack readHeader() implementation
Load pack header length and 15 header entries with single backend
request. This eliminates separate header Load() request for most pack
files and significantly improves index.New() performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:02:25 -05:00
Alexander Neumann 7e6bfdae79 backend/rest: Implement REST API v2 2018-01-23 23:15:26 +01:00
Alexander Neumann 59782e347c repository: Fix repack test
This reduces the chance of duplicate blobs, otherwise the tests fail
(make the contents of a blob depend on a pseudo-random number instead of
the size, sizes may be duplicate).
2018-01-23 23:14:05 +01:00
Alexander Neumann e835abeceb backend/test: Reliably trigger timeout error 2018-01-23 23:14:05 +01:00
Alexander Neumann 8e812b7ac0 errors: Make Cause() unwrap *url.Error 2018-01-23 23:14:05 +01:00
Alexander Neumann 685ce719ad Add comments for List() 2018-01-23 23:14:05 +01:00
Alexander Neumann b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann c4e9d5d11e backend: Add tests for new List() function 2018-01-21 18:35:37 +01:00
Alexander Neumann 52230b8f07 backend: Rework List()
For a discussion see #1567
2018-01-21 18:35:37 +01:00
Alexander Neumann 2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann 67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann 1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann 200415e0a1 Merge pull request #1564 from restic/sftp-password-prompt
sftp: Prompt for password, don't terminate on SIGINT
2018-01-20 09:48:17 +01:00
Alexander Neumann bcb6881ffb Merge pull request #1548 from ifedorenko/checker-backend-Test
checker: Optimize checker.Packs()
2018-01-19 13:05:35 +01:00
Igor Fedorenko 231076fa4a checker: Optimize checker.Packs()
Use result of single repository.List() to find both missing and
orphaned data packs. For 500GB repository this eliminates ~100K
repository.Test() calls and improves check time by >30M in my
environment (~45min before this change and ~7min after).

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-18 20:50:39 -05:00
Alexander Neumann 0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann 05958caf6e sftp: Prompt for password, don't terminate on SIGINT
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
2018-01-17 23:02:47 +01:00
Alexander Neumann 97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann f49f5c5903 fuse: Don't return an error for EOF 2018-01-13 10:14:10 +01:00
Alexander Neumann d89f2e5226 fuse: Remove debug log message 2018-01-13 10:10:07 +01:00
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has()
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
2018-01-08 21:46:17 +01:00
Alexander Neumann d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Alexander Neumann b8620429e2 restorer: Set directory mode as last step
Closes #1512
2018-01-07 15:13:24 +01:00
Alexander Neumann aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann 5f43e17918 Merge pull request #1535 from ifedorenko/check-retry
fixed restic-check does not retry backend.Test failures
2018-01-07 09:43:49 +01:00
Igor Fedorenko 8c550ca011 fixed restic-check does not retry backend.Test failures
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-06 23:22:35 -05:00
Alexander Neumann 032621289b errors: Add "Fatal" prefix for Fatalf() 2018-01-06 23:27:47 +01:00
Alexander Neumann b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann aabc0ccaa7 fs: Make HasPathPrefix work with relative paths 2018-01-05 17:38:52 +01:00
Alexander Neumann 18ecd9df30 Improve limiting HTTP based backends 2017-12-29 12:43:49 +01:00
Alexander Neumann 3f7d85360a fuse: Only reload list of snapshots once per minute 2017-12-28 13:18:27 +01:00
Alexander Neumann 6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann bc04ce8e6b Merge pull request #1482 from restic/fix-1478
backend: Only return top-level files for most dirs
2017-12-20 18:06:47 +01:00
Alexander Neumann 7d8765a937 backend: Only return top-level files for most dirs
Fixes #1478
2017-12-14 19:14:16 +01:00
Alexander Neumann 81a04656c5 fuse: Cleanup node name 2017-12-13 20:21:18 +01:00
Harshavardhana 27ccea6371 Since upgrade to minio-go 4.0 remove workaround
We added previously a code to fix the issue of chaining
credentials, we do not need this anymore since the
upstream minio-go already has this relevant change.
2017-12-09 02:01:42 -08:00
Alexander Neumann 8b3b7bc5ef s3: Use context 2017-12-08 22:04:55 +01:00
Alexander Neumann 934ae1b559 Update to minio-go 4 2017-12-08 21:52:50 +01:00
Alexander Neumann 3a5e040b7e Merge pull request #1439 from armhold/propagate-context
replace ad-hoc context.TODO() with gopts.ctx
2017-12-08 20:27:36 +01:00
George Armhold 1695c8ed55 use global context for check, debug, dump, find, forget, init, key,
list, mount, tag, unlock commands

gh-1434
2017-12-06 07:02:55 -05:00
George Armhold 0dc31c03e1 remove check for context.Canceled
gh-1434
2017-12-06 05:38:29 -05:00
Alexander Neumann 82ded35706 node: Correctly compare times 2017-12-03 18:33:35 +01:00
Alexander Neumann 88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
Alexander Neumann 7092af6329 fuse: Handle sockets/fifos/devs correctly
Closes #1463
2017-12-03 17:25:00 +01:00
George Armhold 20d78ab0d9 add context to repo.Flush() call in test code
gh-1434
2017-12-03 07:32:50 -05:00
George Armhold be24237063 make retry code context-aware.
detect cancellation in backend, so that retry code does not keep trying
once user has hit ^c

gh-1434
2017-12-03 07:22:14 -05:00
George Armhold d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
can properly trickle down from cmd_*.

gh-1434
2017-12-03 07:22:14 -05:00
Alexander Neumann 0b44c629f2 retry: Remove file after failed save 2017-11-30 22:05:14 +01:00
Alexander Neumann 410efe0694 Merge pull request #1454 from restic/fix-cache-dir
Fix cache dir detection
2017-11-29 18:43:07 +01:00
Alexander Neumann eddb8549ef backup: By default, do not save the access time
This can be re-enabled with `--with-atime`.
2017-11-28 21:31:35 +01:00
Alexander Neumann dc38265b54 Merge pull request #1436 from restic/remove-old-cache
Remove old cache directories
2017-11-27 21:37:05 +01:00
Alexander Neumann 0f5e38609f restore: Fix restore to relative path 2017-11-26 18:36:48 +01:00
Alexander Neumann 27d29b9853 restorer: Make sure node names are clean 2017-11-26 15:28:45 +01:00
Alexander Neumann 8a171731ba restorer: Add tests for invalid node names 2017-11-26 15:28:37 +01:00
Alexander Neumann 134abbd82b rest: Use client for creating the repository
Before, creating a new repo via REST would use the defaut HTTP client,
which is not a problem unless the server uses HTTPS and a TLS
certificate which isn't signed by a CA in the system's CA store. In this
case, all commands work except the 'init' command, which fails with a
message like "invalid certificate".
2017-11-25 20:56:40 +01:00
Alexander Neumann 1ebf0e8de8 Merge pull request #1437 from restic/fix-1292
s3: Document and remove default prefix
2017-11-25 11:34:26 +01:00
Alexander Neumann 47b326b7b5 Merge pull request #1423 from harshavardhana/creds
Fix chaining of credentials for minio-go
2017-11-24 21:57:52 +01:00
Alexander Neumann f79698dcdd Merge pull request #1410 from armhold/deadlock2
unify behavior for max http connections across backends
2017-11-24 21:32:56 +01:00
Alexander Neumann f4bab789b8 cache: Simplify cache dir creation 2017-11-24 20:53:26 +01:00
Alexander Neumann fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
Alexander Neumann 262b0cd9d4 s3: Remove default prefix "/restic" 2017-11-21 21:33:09 +01:00
Alexander Neumann e83ec17e95 s3: Correct comment 2017-11-20 22:21:39 +01:00
Alexander Neumann ea593fca1b cache: Correctly return dir for Windows/darwin 2017-11-20 06:11:18 +01:00
Alexander Neumann fe1f151ae1 cache: Return error during default dir detection 2017-11-20 06:10:42 +01:00
Harshavardhana 41c8c946ba Fix chaining of credentials for minio-go
chaining failed because chaining provider
was only looking for subsequent credentials
provider after an error. Writer a new
chaining provider which proceeds to fetch
new credentials also under situations where
providers do not return but instead return
no keys at all.

Fixes https://github.com/restic/restic/issues/1422
2017-11-18 02:51:12 -08:00
George Armhold 0268d0e7d6 swift backend: limit http concurrency in Save(), Stat(), Test(), Remove(),
List().

move comment regarding problematic List() backend api (it's s3's ListObjects
that has a problem, NOT swift's ObjectsWalk).

As per discussion in PR #1399.
2017-11-02 18:29:32 -04:00
George Armhold 8515d093e0 swift backend: fix premature release of semaphore in Load() & document
concurrency issue in List().

refactor wrapReader from b2 -> semaphore so it can be used elsewhere.

As per discussion in PR #1399.
2017-11-02 12:38:17 -04:00
George Armhold 99ac0da4bc s3 backend: limit http concurrency in Save(), Stat(), Test(), Remove()
NB: List() is NOT currently limited, as it would cause deadlock due to
be.client.ListObjects() implementation.

as per discussion in PR #1399
2017-11-01 09:40:54 -04:00
Alexander Neumann 6d46824fb0 Pass in a nil buffer to Open() 2017-11-01 10:30:40 +01:00