Commit Graph

899 Commits

Author SHA1 Message Date
Michael Eischer 27456f6545 debug: use proper context
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
Michael Eischer c458e114d4 pass context to Find / FindSnapshot
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer 45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer 307a6ba3a3 Upgrade minio sdk to v7
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer 50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann 30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
Michael Eischer bcc3bddcf4 filter: only check whether a child path could match when necessary
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer 375c2a56de filter: Parse filter patterns only once
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
MichaelEischer 0ae02f3030
Merge pull request #2844 from aawsome/prune-integration-tests
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
Lorenz Bausch af4100e07d
Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Weiss 6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
Michael Eischer 1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Leo R. Lundgren e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz 5d8cfff3f2
Merge pull request #2988 from greatroar/snapshots-json
Regression test for #2979
2020-10-05 21:02:52 +02:00
MichaelEischer 1ab4c710e1
Merge pull request #2987 from greatroar/no-password-for-help
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar feedf0ebce Don't read password for generate, help or self-update
Fixes #2951.
2020-10-05 19:54:07 +02:00
MichaelEischer 1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
greatroar 6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
Leo R. Lundgren 137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
rawtaz bbeb439f41
Merge pull request #2937 from andreaso/self-update-output-path-fix
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson ce14df303b
Don't require `self-update --output` placeholder file
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer 3c6671b18b
Merge pull request #2973 from restic/rawtaz-repo2
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
rawtaz 0cce6dc31c
Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz 6253ff0187
Clarify max verbose level in option description
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
jtagcat a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
Michael Eischer 61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter 97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
2020-10-01 00:50:26 +02:00
rawtaz 34ea960559
Merge pull request #2966 from MichaelEischer/recover-help-typo
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
Michael Eischer 2a79c1a44d recover: Fix typo in the command help
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
Michael Eischer aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
plumbeo 009bd907f2
Don’t print too many messages when deleting files
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
MichaelEischer 14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred 206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer 4875f7b659
Merge pull request #2614 from greatroar/simplify-fs
Simplify internal/fs
2020-09-21 22:05:10 +02:00
yoshiera 3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Michael Eischer 9333f707fa init: use Create method for rclone backend
This properly issues the initial repository creation command

Fixes #1896
2020-09-19 19:11:43 +02:00
Alexander Neumann 429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann 40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Michael Eischer f003410402 init: Add `--copy-chunker-params` option
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer 655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
Leo R. Lundgren 311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Leo R. Lundgren ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
yoshiera ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
greatroar 9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Michael Eischer f5c448aa65 diff: Optimize diff calculation for shared subtrees
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
2020-09-13 16:28:46 +02:00
Michael Eischer c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann 0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
rawtaz d3a286928a
Merge pull request #2905 from MichaelEischer/fix-ls-help
ls: Explicitly enforce that the user specifies a snapshot ID
2020-09-05 10:44:01 +02:00
Michael Eischer 1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer 068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer 460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer 49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer 2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer 37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer 337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer 2ddb7ffb7e
Merge pull request #2255 from Kidswiss/tar
Fix dumping issues with / and the first sub level
2020-09-01 21:52:17 +02:00
Michael Eischer 81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer 55071ee367
Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path
Add filter by tag and path to stats command
2020-08-31 22:11:01 +02:00
Michael Eischer bcf44a9c3f dump: Properly handle file paths without / prefix
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
2020-08-30 18:25:42 +02:00
Oliver Buschjost a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
MichaelEischer ea81a0e282
Merge pull request #2606 from middelink/fix-323
Add copy functionality.
2020-08-30 10:18:24 +02:00
Simon Beck d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck b98598e55f dump: Fix dumping issues with / and the first sub level
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
2020-08-29 21:14:29 +02:00
Michael Eischer d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
2020-08-29 14:28:53 +02:00
Michael Eischer 412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer 9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer 15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer 88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer 591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer ec9a53b7e8 copy: Mark and skip previously copied snapshots
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
2020-08-29 10:48:44 +02:00
Alexander Weiss 9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
Jason Cox 2447f3f110 Update usage strings to put flags before args
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
2020-08-28 02:16:22 +00:00
Michael Eischer b0a8c4ad6c copy: Only process each tree once
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
2020-08-26 22:17:52 +02:00
Michael Eischer 908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar 4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink 7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
Michael Eischer 17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
Alexander Neumann b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
Bruce Dillahunty fe69b83074
Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer 08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer 1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer 575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer 8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer 744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer 3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome 0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
MichaelEischer 08dee8a52b
Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
MichaelEischer 84b6f1ec53
Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer 06fb4ea3f0
Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
MichaelEischer 38a2f9c07b
Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz 5af2815627
Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00