Commit Graph

768 Commits

Author SHA1 Message Date
Michael Eischer
908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar
4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink
7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
Alexander Neumann
b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
Bruce Dillahunty
fe69b83074
Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer
08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer
1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer
575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer
8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer
744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer
3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
MichaelEischer
08dee8a52b
Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
MichaelEischer
84b6f1ec53
Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer
06fb4ea3f0
Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer
e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar
d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
MichaelEischer
38a2f9c07b
Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz
5af2815627
Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00
Sébastien Gross
9be4fe3e84 cmd/mount: honour --no-lock flag
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-08-03 19:45:39 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer
04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar
8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer
66d089e239
Merge pull request #2841 from aawsome/optimize-getUsedBlobs
Extract get used blobs in prune as separate function
2020-08-01 22:40:33 +02:00
Alexander Weiss
9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss
d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss
1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss
2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann
b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann
2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer
3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
Alexander Weiss
9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
David Sommer
5729d967f5 backup: Return exit status code 3 when failing to read source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
2020-07-20 22:19:01 +02:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
Jakub Wilk
4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
Andreas Olsson
0fcef2ec23
Use https:// for linking CACHEDIR.TAG specs
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
2020-06-21 19:36:26 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs'
=> reduces memory consumption a lot!
2020-06-14 17:25:34 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
2020-06-14 11:08:11 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command
+ modifications of changelog
2020-06-12 09:24:38 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
nunoperalta
2c3360db98
Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00