Commit Graph

3272 Commits

Author SHA1 Message Date
Matthew Dawson df2c03a6a4
repository/master_index: Optimize Index.Lookup()
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
2018-01-23 22:25:56 -05:00
Matthew Dawson ebce4b2581
repository/index: Speed up benchmarks and tests
When setting up the index used for benchmarking, use math/rand instead of
crypto/rand since the generated ids don't need to be evenly distributed,
and not be secure against guessing.  As such, use a different random id
function (only available during tests) that uses math/rand instead.
2018-01-23 22:21:05 -05:00
Alexander Neumann b63de7c798 script/release: Tag versioned docker image 2018-01-21 21:41:14 +01:00
Alexander Neumann a3d43a92b3 Merge pull request #1569 from restic/ext-rest-backend-test
REST: refactor backend tests
2018-01-21 18:02:09 +01:00
Alexander Neumann 2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann 67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann 1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann 663e8284b2 init: Correct text (backend vs. repository)
Closes #1565
2018-01-20 09:51:49 +01:00
Alexander Neumann 5a43ca1ccb Add entry to CHANGELOG 2018-01-20 09:50:31 +01:00
Alexander Neumann 200415e0a1 Merge pull request #1564 from restic/sftp-password-prompt
sftp: Prompt for password, don't terminate on SIGINT
2018-01-20 09:48:17 +01:00
Alexander Neumann ea50687e8b Merge pull request #1568 from martinseener/patch-1
Added hint where to place restic.exe on windows
2018-01-20 09:47:50 +01:00
Alexander Neumann 7335a2492f Add PR to changelog 2018-01-19 13:05:55 +01:00
Alexander Neumann bcb6881ffb Merge pull request #1548 from ifedorenko/checker-backend-Test
checker: Optimize checker.Packs()
2018-01-19 13:05:35 +01:00
Alexander Neumann 37df829b3d CONTRIBUTING: clarify wording 2018-01-19 13:00:49 +01:00
Martin 39f1ba1897
Added hint where to place restic.exe on windows
I've added a hint for pre-compiled binaries on windows where to place them for easier use in the shell or in scripts without the hassle for absolute paths.
2018-01-19 12:24:09 +01:00
Igor Fedorenko 231076fa4a checker: Optimize checker.Packs()
Use result of single repository.List() to find both missing and
orphaned data packs. For 500GB repository this eliminates ~100K
repository.Test() calls and improves check time by >30M in my
environment (~45min before this change and ~7min after).

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-18 20:50:39 -05:00
Alexander Neumann 0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann 05958caf6e sftp: Prompt for password, don't terminate on SIGINT
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
2018-01-17 23:02:47 +01:00
Alexander Neumann 87e31799f2 Merge pull request #1483 from dstosberg/master
document how to create a full backup without running restic as root
2018-01-16 17:21:55 +01:00
Alexander Neumann 35528506a6 Merge pull request #1554 from restic/fix-988
fuse: Correct behavior for reading after EOF, add snapshot template string
2018-01-16 17:17:55 +01:00
Alexander Neumann 2544978aaf Add entry to changelog 2018-01-15 14:47:17 +01:00
Alexander Neumann 97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann f49f5c5903 fuse: Don't return an error for EOF 2018-01-13 10:14:10 +01:00
Alexander Neumann d89f2e5226 fuse: Remove debug log message 2018-01-13 10:10:07 +01:00
Alexander Neumann 02f4f5dc66 Update build.go, correct version detection 2018-01-12 22:06:42 +01:00
Alexander Neumann 5723636b35 Add entry for changelog 2018-01-09 21:57:07 +01:00
Alexander Neumann 5632ca4f44 Merge pull request #1538 from MJDSys/make_lookup_internal
repository/index: Avoid creating an error when checking if an id is in the index.
2018-01-09 21:55:37 +01:00
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has()
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
2018-01-08 21:46:17 +01:00
Alexander Neumann d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Dennis Stosberg 60a7a9d019 manual: document how to create a full backup without running restic as root 2018-01-08 19:59:15 +01:00
Alexander Neumann 99f0fce673 Merge pull request #1536 from restic/fix-restorer-permissions
restore: Fix directory permissions
2018-01-07 17:18:29 +01:00
Alexander Neumann b708cfee3f Add entry to changelog 2018-01-07 15:16:16 +01:00
Alexander Neumann b8620429e2 restorer: Set directory mode as last step
Closes #1512
2018-01-07 15:13:24 +01:00
Alexander Neumann aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann e120b17abd Update PR template 2018-01-07 11:53:30 +01:00
Alexander Neumann 5f43e17918 Merge pull request #1535 from ifedorenko/check-retry
fixed restic-check does not retry backend.Test failures
2018-01-07 09:43:49 +01:00
Alexander Neumann 489eef5a6f Merge pull request #1534 from restic/improve-error-restore
restore: Improve error message for more than one ID
2018-01-07 09:40:06 +01:00
Igor Fedorenko 8c550ca011 fixed restic-check does not retry backend.Test failures
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-06 23:22:35 -05:00
Alexander Neumann 032621289b errors: Add "Fatal" prefix for Fatalf() 2018-01-06 23:27:47 +01:00
Alexander Neumann 92316a9853 restore: Improve error message for more than one ID 2018-01-06 22:44:18 +01:00
Alexander Neumann 180741609e Merge pull request #1530 from juliangehring/patch-1
Fix code block formatting in FAQs
2018-01-06 18:46:14 +01:00
Julian Gehring 70250762f3
Fix code block formatting in FAQs
Fixes the formatting of a code block in the FAQs and makes it render properly.
2018-01-05 23:02:03 +00:00
Alexander Neumann 39ba17a2d6 Add entry to CHANGELOG 2018-01-05 19:29:02 +01:00
Alexander Neumann cfe8c8c9cd Merge pull request #1529 from restic/fix-1528
Fix creating missing dirs in local/sftp repo
2018-01-05 19:28:17 +01:00
Alexander Neumann b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann aabc0ccaa7 fs: Make HasPathPrefix work with relative paths 2018-01-05 17:38:52 +01:00
Alexander Neumann 2bc4d200d4 Merge pull request #1518 from restic/fix-1317-again
forget: Also run prune when only IDs are forgotten
2018-01-02 20:39:12 +01:00
Alexander Neumann 0247fe01c0 Update homepage URL in README 2018-01-01 21:50:44 +01:00
Alexander Neumann c912b38bf0 forget: Also run prune when only IDs are forgotten 2018-01-01 21:27:40 +01:00