rdelaage
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'old-origin/master' into add-smb-backend

This commit is contained in:
Aneesh Nireshwalia 2023-05-18 14:25:16 -06:00
parent db9d46c5e5 2f518b7241
commit 4b02684f69
164 changed files with 8422 additions and 1257 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.github/dependabot.yml vendored
View File

@ -4,10 +4,10 @@ updates:
- package-ecosystem: "gomod"
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
interval: "monthly"
# Dependencies listed in .github/workflows/*.yml
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
interval: "monthly"

6
.github/workflows/tests.yml vendored
View File

@ -64,7 +64,7 @@ jobs:
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v3
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go }}
@ -340,7 +340,7 @@ jobs:
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v3
uses: actions/setup-go@v4
with:
go-version: ${{ env.latest_go }}
@ -365,7 +365,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v3
uses: actions/setup-go@v4
with:
go-version: ${{ env.latest_go }}

1
.gitignore vendored
View File

@ -1,3 +1,4 @@
/restic
/restic.exe
/.vagrant
/.vscode

89
CHANGELOG.md
View File

@ -1,3 +1,92 @@
Changelog for restic 0.15.2 (2023-04-24)
=======================================
The following sections list the changes in restic 0.15.2 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Sec #4275: Update golang.org/x/net to address CVE-2022-41723
* Fix #2260: Sanitize filenames printed by `backup` during processing
* Fix #4211: Make `dump` interpret `--host` and `--path` correctly
* Fix #4239: Correct number of blocks reported in mount point
* Fix #4253: Minimize risk of spurious filesystem loops with `mount`
* Enh #4180: Add release binaries for riscv64 architecture on Linux
* Enh #4219: Upgrade Minio to version 7.0.49
Details
-------
* Security #4275: Update golang.org/x/net to address CVE-2022-41723
https://github.com/restic/restic/issues/4275
https://github.com/restic/restic/pull/4213
* Bugfix #2260: Sanitize filenames printed by `backup` during processing
The `backup` command would previously not sanitize the filenames it printed during
processing, potentially causing newlines or terminal control characters to mangle the
status output or even change the state of a terminal.
Filenames are now checked and quoted if they contain non-printable or non-Unicode
characters.
https://github.com/restic/restic/issues/2260
https://github.com/restic/restic/issues/4191
https://github.com/restic/restic/pull/4192
* Bugfix #4211: Make `dump` interpret `--host` and `--path` correctly
A regression in restic 0.15.0 caused `dump` to confuse its `--host=<host>` and
`--path=<path>` options: it looked for snapshots with paths called `<host>` from hosts
called `<path>`. It now treats the options as intended.
https://github.com/restic/restic/issues/4211
https://github.com/restic/restic/pull/4212
* Bugfix #4239: Correct number of blocks reported in mount point
Restic mount points reported an incorrect number of 512-byte (POSIX standard) blocks for
files and links due to a rounding bug. In particular, empty files were reported as taking one
block instead of zero.
The rounding is now fixed: the number of blocks reported is the file size (or link target size)
divided by 512 and rounded up to a whole number.
https://github.com/restic/restic/issues/4239
https://github.com/restic/restic/pull/4240
* Bugfix #4253: Minimize risk of spurious filesystem loops with `mount`
When a backup contains a directory that has the same name as its parent, say `a/b/b`, and the GNU
`find` command was run on this backup in a restic mount, `find` would refuse to traverse the
lowest `b` directory, instead printing `File system loop detected`. This was due to the way the
restic mount command generates inode numbers for directories in the mount point.
The rule for generating these inode numbers was changed in 0.15.0. It has now been changed again
to avoid this issue. A perfect rule does not exist, but the probability of this behavior
occurring is now extremely small.
When it does occur, the mount point is not broken, and scripts that traverse the mount point
should work as long as they don't rely on inode numbers for detecting filesystem loops.
https://github.com/restic/restic/issues/4253
https://github.com/restic/restic/pull/4255
* Enhancement #4180: Add release binaries for riscv64 architecture on Linux
Builds for the `riscv64` architecture on Linux are now included in the release binaries.
https://github.com/restic/restic/pull/4180
* Enhancement #4219: Upgrade Minio to version 7.0.49
The upgraded version now allows use of the `ap-southeast-4` region (Melbourne).
https://github.com/restic/restic/pull/4219
Changelog for restic 0.15.1 (2023-01-30)
=======================================

10
CONTRIBUTING.md
View File

@ -58,6 +58,16 @@ Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
If restic gets stuck, please also include a stacktrace in the description.
On non-Windows systems, you can send a SIGQUIT signal to restic or press
`Ctrl-\` to achieve the same result. This causes restic to print a stacktrace
and then exit immediatelly. This will not damage your repository, however,
it might be necessary to manually clean up stale lock files using
`restic unlock`.
On Windows, please set the environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT`
to `true` and press `Ctrl-C` to create a stacktrace.
Development Environment
=======================

2
VERSION
View File

@ -1 +1 @@
0.15.1
0.15.2

6
build.go
View File

@ -380,6 +380,12 @@ func main() {
}
}
solarisMinVersion := GoVersion{Major: 1, Minor: 20, Patch: 0}
if env["GOARCH"] == "solaris" && !goVersion.AtLeast(solarisMinVersion) {
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s for Solaris\n", goVersion, solarisMinVersion)
os.Exit(1)
}
verbosePrintf("detected Go version %v\n", goVersion)
preserveSymbols := false

12
changelog/0.15.2_2023-04-24/issue-2260 Normal file
View File

@ -0,0 +1,12 @@
Bugfix: Sanitize filenames printed by `backup` during processing
The `backup` command would previously not sanitize the filenames it printed
during processing, potentially causing newlines or terminal control characters
to mangle the status output or even change the state of a terminal.
Filenames are now checked and quoted if they contain non-printable or
non-Unicode characters.
https://github.com/restic/restic/issues/2260
https://github.com/restic/restic/issues/4191
https://github.com/restic/restic/pull/4192

8
changelog/0.15.2_2023-04-24/issue-4211 Normal file
View File

@ -0,0 +1,8 @@
Bugfix: Make `dump` interpret `--host` and `--path` correctly
A regression in restic 0.15.0 caused `dump` to confuse its `--host=<host>` and
`--path=<path>` options: it looked for snapshots with paths called `<host>`
from hosts called `<path>`. It now treats the options as intended.
https://github.com/restic/restic/issues/4211
https://github.com/restic/restic/pull/4212

11
changelog/0.15.2_2023-04-24/issue-4239 Normal file
View File

@ -0,0 +1,11 @@
Bugfix: Correct number of blocks reported in mount point
Restic mount points reported an incorrect number of 512-byte (POSIX standard)
blocks for files and links due to a rounding bug. In particular, empty files
were reported as taking one block instead of zero.
The rounding is now fixed: the number of blocks reported is the file size
(or link target size) divided by 512 and rounded up to a whole number.
https://github.com/restic/restic/issues/4239
https://github.com/restic/restic/pull/4240

18
changelog/0.15.2_2023-04-24/issue-4253 Normal file
View File

@ -0,0 +1,18 @@
Bugfix: Minimize risk of spurious filesystem loops with `mount`
When a backup contains a directory that has the same name as its parent, say
`a/b/b`, and the GNU `find` command was run on this backup in a restic mount,
`find` would refuse to traverse the lowest `b` directory, instead printing
`File system loop detected`. This was due to the way the restic mount command
generates inode numbers for directories in the mount point.
The rule for generating these inode numbers was changed in 0.15.0. It has
now been changed again to avoid this issue. A perfect rule does not exist,
but the probability of this behavior occurring is now extremely small.
When it does occur, the mount point is not broken, and scripts that traverse
the mount point should work as long as they don't rely on inode numbers for
detecting filesystem loops.
https://github.com/restic/restic/issues/4253
https://github.com/restic/restic/pull/4255

4
changelog/0.15.2_2023-04-24/issue-4275 Normal file
View File

@ -0,0 +1,4 @@
Security: Update golang.org/x/net to address CVE-2022-41723
https://github.com/restic/restic/issues/4275
https://github.com/restic/restic/pull/4213

3
changelog/unreleased/pull-4180 → changelog/0.15.2_2023-04-24/pull-4180
View File

@ -1,5 +1,6 @@
Enhancement: Add release binaries for riscv64 architecture on Linux
We've added release binaries for riscv64 architecture on Linux.
Builds for the `riscv64` architecture on Linux are now included in the
release binaries.
https://github.com/restic/restic/pull/4180

5
changelog/0.15.2_2023-04-24/pull-4219 Normal file
View File

@ -0,0 +1,5 @@
Enhancement: Upgrade Minio to version 7.0.49
The upgraded version now allows use of the `ap-southeast-4` region (Melbourne).
https://github.com/restic/restic/pull/4219

20
changelog/unreleased/issue-1759 Normal file
View File

@ -0,0 +1,20 @@
Enhancement: Add `repair index` and `repair snapshots` commands
The `rebuild-index` command has been renamed to `repair index`. The old name
will still work, but is deprecated.
When a snapshot was damaged, the only option up to now was to completely forget
the snapshot, even if only some unimportant file was damaged.
We've added a `repair snapshots` command, which can repair snapshots by removing
damaged directories and missing files contents. Note that using this command
can lead to data loss! Please see the "Troubleshooting" section in the documentation
for more details.
https://github.com/restic/restic/issues/1759
https://github.com/restic/restic/issues/1714
https://github.com/restic/restic/issues/1798
https://github.com/restic/restic/issues/2334
https://github.com/restic/restic/pull/2876
https://forum.restic.net/t/corrupted-repo-how-to-repair/799
https://forum.restic.net/t/recovery-options-for-damaged-repositories/1571

8
changelog/unreleased/issue-2565 Normal file
View File

@ -0,0 +1,8 @@
Bugfix: Restic forget --keep-* options now interpret "-1" as "forever"
Restic would forget snapshots that should have been kept when "-1" was
used as a value for --keep-* options. It now interprets "-1" as forever,
e.g. an option like --keep-monthly -1 will keep all monthly snapshots.
https://github.com/restic/restic/issues/2565
https://github.com/restic/restic/pull/4234

9
changelog/unreleased/issue-3627 Normal file
View File

@ -0,0 +1,9 @@
Enhancement: Show progress bar during restore
The `restore` command now shows a progress report while restoring files.
Example: [0:42] 5.76% 23 files 12.98 MiB, total 3456 files 23.54 GiB
https://github.com/restic/restic/issues/3627
https://github.com/restic/restic/pull/3991
https://forum.restic.net/t/progress-bar-for-restore/5210

8
changelog/unreleased/issue-4211
View File

@ -1,8 +0,0 @@
Bugfix: Restic dump now interprets --host and --path correctly
Restic dump previously confused its --host=<host> and --path=<path>
options: it looked for snapshots with paths called <host> from hosts
called <path>. It now treats the options as intended.
https://github.com/restic/restic/issues/4211
https://github.com/restic/restic/pull/4212

8
changelog/unreleased/issue-719 Normal file
View File

@ -0,0 +1,8 @@
Enhancement: Add --retry-lock option
This option allows to specify a duration for which restic will wait if there
already exists a conflicting lock within the repository.
https://github.com/restic/restic/issues/719
https://github.com/restic/restic/pull/2214
https://github.com/restic/restic/pull/4107

7
changelog/unreleased/pull-4166 Normal file
View File

@ -0,0 +1,7 @@
Enhancement: Cancel current command if cache becomes unusable
If the cache directory was removed or ran out of space while restic was
running, this caused further caching attempts to fail and drastically slow down
the command execution. Now, the currently running command is canceled instead.
https://github.com/restic/restic/pull/4166

9
changelog/unreleased/pull-4201 Normal file
View File

@ -0,0 +1,9 @@
Change: Require Go 1.20 for Solaris builds
Building restic on Solaris now requires Go 1.20, as the library used to access
Azure uses the mmap syscall, which is only available on Solaris starting from
Go 1.20.
All other platforms continue to build with Go 1.18.
https://github.com/restic/restic/pull/4201

5
changelog/unreleased/pull-4219
View File

@ -1,5 +0,0 @@
Enhancement: Upgrade Minio to 7.0.49
Upgraded to allow use of the ap-southeast-4 region (Melbourne)
https://github.com/restic/restic/pull/4219

5
changelog/unreleased/pull-4220 Normal file
View File

@ -0,0 +1,5 @@
Enhancement: Add jq to container image
The Docker container image now contains jq which can be useful when restic outputs json data.
https://github.com/restic/restic/pull/4220

5
changelog/unreleased/pull-4304 Normal file
View File

@ -0,0 +1,5 @@
Bugfix: Avoid lock refresh issues with slow network connections
On network connections with a low upload speed, restic could often fail backups and other operations with `Fatal: failed to refresh lock in time`. We've reworked the lock refresh to avoid this error.
https://github.com/restic/restic/pull/4304

8
changelog/unreleased/pull-4318 Normal file
View File

@ -0,0 +1,8 @@
Bugfix: Correctly clean up status bar output of the `backup` command
Due to a regression in restic 0.15.2, the status bar of the `backup` command
could leave some output behind. This happened if filenames were printed that
are wider than the current terminal width. This has been fixed.
https://github.com/restic/restic/issues/4319
https://github.com/restic/restic/pull/4318

7
cmd/restic/cleanup.go
View File

@ -62,6 +62,12 @@ func CleanupHandler(c <-chan os.Signal) {
debug.Log("signal %v received, cleaning up", s)
Warnf("%ssignal %v received, cleaning up\n", clearLine(0), s)
if val, _ := os.LookupEnv("RESTIC_DEBUG_STACKTRACE_SIGINT"); val != "" {
_, _ = os.Stderr.WriteString("\n--- STACKTRACE START ---\n\n")
_, _ = os.Stderr.WriteString(debug.DumpStacktrace())
_, _ = os.Stderr.WriteString("\n--- STACKTRACE END ---\n")
}
code := 0
if s == syscall.SIGINT {
@ -78,5 +84,6 @@ func CleanupHandler(c <-chan os.Signal) {
// given exit code.
func Exit(code int) {
code = RunCleanupHandlers(code)
debug.Log("exiting with status code %d", code)
os.Exit(code)
}

2
cmd/restic/cmd_backup.go
View File

@ -506,7 +506,7 @@ func runBackup(ctx context.Context, opts BackupOptions, gopts GlobalOptions, ter
if !gopts.JSON {
progressPrinter.V("lock repository")
}
lock, ctx, err := lockRepo(ctx, repo)
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_cat.go
View File

@ -45,7 +45,7 @@ func runCat(ctx context.Context, gopts GlobalOptions, args []string) error {
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

4
cmd/restic/cmd_check.go
View File

@ -211,7 +211,7 @@ func runCheck(ctx context.Context, opts CheckOptions, gopts GlobalOptions, args
if !gopts.NoLock {
Verbosef("create exclusive lock for repository\n")
var lock *restic.Lock
lock, ctx, err = lockRepoExclusive(ctx, repo)
lock, ctx, err = lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -245,7 +245,7 @@ func runCheck(ctx context.Context, opts CheckOptions, gopts GlobalOptions, args
}
if suggestIndexRebuild {
Printf("Duplicate packs/old indexes are non-critical, you can run `restic rebuild-index' to correct this.\n")
Printf("Duplicate packs/old indexes are non-critical, you can run `restic repair index' to correct this.\n")
}
if mixedFound {
Printf("Mixed packs with tree and data blobs are non-critical, you can run `restic prune` to correct this.\n")

4
cmd/restic/cmd_copy.go
View File

@ -74,14 +74,14 @@ func runCopy(ctx context.Context, opts CopyOptions, gopts GlobalOptions, args []
if !gopts.NoLock {
var srcLock *restic.Lock
srcLock, ctx, err = lockRepo(ctx, srcRepo)
srcLock, ctx, err = lockRepo(ctx, srcRepo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(srcLock)
if err != nil {
return err
}
}
dstLock, ctx, err := lockRepo(ctx, dstRepo)
dstLock, ctx, err := lockRepo(ctx, dstRepo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(dstLock)
if err != nil {
return err

4
cmd/restic/cmd_debug.go
View File

@ -156,7 +156,7 @@ func runDebugDump(ctx context.Context, gopts GlobalOptions, args []string) error
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -462,7 +462,7 @@ func runDebugExamine(ctx context.Context, gopts GlobalOptions, args []string) er
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_diff.go
View File

@ -334,7 +334,7 @@ func runDiff(ctx context.Context, opts DiffOptions, gopts GlobalOptions, args []
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_dump.go
View File

@ -132,7 +132,7 @@ func runDump(ctx context.Context, opts DumpOptions, gopts GlobalOptions, args []
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_find.go
View File

@ -575,7 +575,7 @@ func runFind(ctx context.Context, opts FindOptions, gopts GlobalOptions, args []
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

37
cmd/restic/cmd_forget.go
View File

@ -67,12 +67,12 @@ func init() {
cmdRoot.AddCommand(cmdForget)
f := cmdForget.Flags()
f.IntVarP(&forgetOptions.Last, "keep-last", "l", 0, "keep the last `n` snapshots")
f.IntVarP(&forgetOptions.Hourly, "keep-hourly", "H", 0, "keep the last `n` hourly snapshots")
f.IntVarP(&forgetOptions.Daily, "keep-daily", "d", 0, "keep the last `n` daily snapshots")
f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
f.IntVarP(&forgetOptions.Last, "keep-last", "l", 0, "keep the last `n` snapshots (use '-1' to keep all snapshots)")
f.IntVarP(&forgetOptions.Hourly, "keep-hourly", "H", 0, "keep the last `n` hourly snapshots (use '-1' to keep all hourly snapshots)")
f.IntVarP(&forgetOptions.Daily, "keep-daily", "d", 0, "keep the last `n` daily snapshots (use '-1' to keep all daily snapshots)")
f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots (use '-1' to keep all weekly snapshots)")
f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots (use '-1' to keep all monthly snapshots)")
f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots (use '-1' to keep all yearly snapshots)")
f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that are newer than `duration` (eg. 1y5m7d2h) relative to the latest snapshot")
f.VarP(&forgetOptions.WithinHourly, "keep-within-hourly", "", "keep hourly snapshots that are newer than `duration` (eg. 1y5m7d2h) relative to the latest snapshot")
f.VarP(&forgetOptions.WithinDaily, "keep-within-daily", "", "keep daily snapshots that are newer than `duration` (eg. 1y5m7d2h) relative to the latest snapshot")
@ -99,8 +99,29 @@ func init() {
addPruneOptions(cmdForget)
}
func verifyForgetOptions(opts *ForgetOptions) error {
if opts.Last < -1 || opts.Hourly < -1 || opts.Daily < -1 || opts.Weekly < -1 ||
opts.Monthly < -1 || opts.Yearly < -1 {
return errors.Fatal("negative values other than -1 are not allowed for --keep-*")
}
for _, d := range []restic.Duration{opts.Within, opts.WithinHourly, opts.WithinDaily,
opts.WithinMonthly, opts.WithinWeekly, opts.WithinYearly} {
if d.Hours < 0 || d.Days < 0 || d.Months < 0 || d.Years < 0 {
return errors.Fatal("durations containing negative values are not allowed for --keep-within*")
}
}
return nil
}
func runForget(ctx context.Context, opts ForgetOptions, gopts GlobalOptions, args []string) error {
err := verifyPruneOptions(&pruneOptions)
err := verifyForgetOptions(&opts)
if err != nil {
return err
}
err = verifyPruneOptions(&pruneOptions)
if err != nil {
return err
}
@ -116,7 +137,7 @@ func runForget(ctx context.Context, opts ForgetOptions, gopts GlobalOptions, arg
if !opts.DryRun || !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepoExclusive(ctx, repo)
lock, ctx, err = lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

65
cmd/restic/cmd_forget_test.go Normal file
View File

@ -0,0 +1,65 @@
package main
import (
"testing"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
)
func TestForgetOptionValues(t *testing.T) {
const negValErrorMsg = "Fatal: negative values other than -1 are not allowed for --keep-*"
const negDurationValErrorMsg = "Fatal: durations containing negative values are not allowed for --keep-within*"
testCases := []struct {
input ForgetOptions
expectsError bool
errorMsg string
}{
{ForgetOptions{Last: 1}, false, ""},
{ForgetOptions{Hourly: 1}, false, ""},
{ForgetOptions{Daily: 1}, false, ""},
{ForgetOptions{Weekly: 1}, false, ""},
{ForgetOptions{Monthly: 1}, false, ""},
{ForgetOptions{Yearly: 1}, false, ""},
{ForgetOptions{Last: 0}, false, ""},
{ForgetOptions{Hourly: 0}, false, ""},
{ForgetOptions{Daily: 0}, false, ""},
{ForgetOptions{Weekly: 0}, false, ""},
{ForgetOptions{Monthly: 0}, false, ""},
{ForgetOptions{Yearly: 0}, false, ""},
{ForgetOptions{Last: -1}, false, ""},
{ForgetOptions{Hourly: -1}, false, ""},
{ForgetOptions{Daily: -1}, false, ""},
{ForgetOptions{Weekly: -1}, false, ""},
{ForgetOptions{Monthly: -1}, false, ""},
{ForgetOptions{Yearly: -1}, false, ""},
{ForgetOptions{Last: -2}, true, negValErrorMsg},
{ForgetOptions{Hourly: -2}, true, negValErrorMsg},
{ForgetOptions{Daily: -2}, true, negValErrorMsg},
{ForgetOptions{Weekly: -2}, true, negValErrorMsg},
{ForgetOptions{Monthly: -2}, true, negValErrorMsg},
{ForgetOptions{Yearly: -2}, true, negValErrorMsg},
{ForgetOptions{Within: restic.ParseDurationOrPanic("1y2m3d3h")}, false, ""},
{ForgetOptions{WithinHourly: restic.ParseDurationOrPanic("1y2m3d3h")}, false, ""},
{ForgetOptions{WithinDaily: restic.ParseDurationOrPanic("1y2m3d3h")}, false, ""},
{ForgetOptions{WithinWeekly: restic.ParseDurationOrPanic("1y2m3d3h")}, false, ""},
{ForgetOptions{WithinMonthly: restic.ParseDurationOrPanic("2y4m6d8h")}, false, ""},
{ForgetOptions{WithinYearly: restic.ParseDurationOrPanic("2y4m6d8h")}, false, ""},
{ForgetOptions{Within: restic.ParseDurationOrPanic("-1y2m3d3h")}, true, negDurationValErrorMsg},
{ForgetOptions{WithinHourly: restic.ParseDurationOrPanic("1y-2m3d3h")}, true, negDurationValErrorMsg},
{ForgetOptions{WithinDaily: restic.ParseDurationOrPanic("1y2m-3d3h")}, true, negDurationValErrorMsg},
{ForgetOptions{WithinWeekly: restic.ParseDurationOrPanic("1y2m3d-3h")}, true, negDurationValErrorMsg},
{ForgetOptions{WithinMonthly: restic.ParseDurationOrPanic("-2y4m6d8h")}, true, negDurationValErrorMsg},
{ForgetOptions{WithinYearly: restic.ParseDurationOrPanic("2y-4m6d8h")}, true, negDurationValErrorMsg},
}
for _, testCase := range testCases {
err := verifyForgetOptions(&testCase.input)
if testCase.expectsError {
rtest.Assert(t, err != nil, "should have returned error for input %+v", testCase.input)
rtest.Equals(t, testCase.errorMsg, err.Error())
} else {
rtest.Assert(t, err == nil, "expected no error for input %+v", testCase.input)
}
}
}

16
cmd/restic/cmd_generate.go
View File

@ -63,22 +63,30 @@ func writeManpages(dir string) error {
}
func writeBashCompletion(file string) error {
Verbosef("writing bash completion file to %v\n", file)
if stdoutIsTerminal() {
Verbosef("writing bash completion file to %v\n", file)
}
return cmdRoot.GenBashCompletionFile(file)
}
func writeFishCompletion(file string) error {
Verbosef("writing fish completion file to %v\n", file)
if stdoutIsTerminal() {
Verbosef("writing fish completion file to %v\n", file)
}
return cmdRoot.GenFishCompletionFile(file, true)
}
func writeZSHCompletion(file string) error {
Verbosef("writing zsh completion file to %v\n", file)
if stdoutIsTerminal() {
Verbosef("writing zsh completion file to %v\n", file)
}
return cmdRoot.GenZshCompletionFile(file)
}
func writePowerShellCompletion(file string) error {
Verbosef("writing powershell completion file to %v\n", file)
if stdoutIsTerminal() {
Verbosef("writing powershell completion file to %v\n", file)
}
return cmdRoot.GenPowerShellCompletionFile(file)
}

7
cmd/restic/cmd_init.go
View File

@ -102,7 +102,12 @@ func runInit(ctx context.Context, opts InitOptions, gopts GlobalOptions, args []
}
if !gopts.JSON {
Verbosef("created restic repository %v at %s\n", s.Config().ID[:10], location.StripPassword(gopts.Repo))
Verbosef("created restic repository %v at %s", s.Config().ID[:10], location.StripPassword(gopts.Repo))
if opts.CopyChunkerParameters && chunkerPolynomial != nil {
Verbosef(" with chunker parameters copied from secondary repository\n")
} else {
Verbosef("\n")
}
Verbosef("\n")
Verbosef("Please note that knowledge of your password is required to access\n")
Verbosef("the repository. Losing your password means that your data is\n")

8
cmd/restic/cmd_key.go
View File

@ -212,7 +212,7 @@ func runKey(ctx context.Context, gopts GlobalOptions, args []string) error {
switch args[0] {
case "list":
lock, ctx, err := lockRepo(ctx, repo)
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -220,7 +220,7 @@ func runKey(ctx context.Context, gopts GlobalOptions, args []string) error {
return listKeys(ctx, repo, gopts)
case "add":
lock, ctx, err := lockRepo(ctx, repo)
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -228,7 +228,7 @@ func runKey(ctx context.Context, gopts GlobalOptions, args []string) error {
return addKey(ctx, repo, gopts)
case "remove":
lock, ctx, err := lockRepoExclusive(ctx, repo)
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -241,7 +241,7 @@ func runKey(ctx context.Context, gopts GlobalOptions, args []string) error {
return deleteKey(ctx, repo, id)
case "passwd":
lock, ctx, err := lockRepoExclusive(ctx, repo)
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

8
cmd/restic/cmd_list.go
View File

@ -31,19 +31,19 @@ func init() {
cmdRoot.AddCommand(cmdList)
}
func runList(ctx context.Context, cmd *cobra.Command, opts GlobalOptions, args []string) error {
func runList(ctx context.Context, cmd *cobra.Command, gopts GlobalOptions, args []string) error {
if len(args) != 1 {
return errors.Fatal("type not specified, usage: " + cmd.Use)
}
repo, err := OpenRepository(ctx, opts)
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
if !opts.NoLock && args[0] != "locks" {
if !gopts.NoLock && args[0] != "locks" {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_migrate.go
View File

@ -122,7 +122,7 @@ func runMigrate(ctx context.Context, opts MigrateOptions, gopts GlobalOptions, a
return err
}
lock, ctx, err := lockRepoExclusive(ctx, repo)
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_mount.go
View File

@ -123,7 +123,7 @@ func runMount(ctx context.Context, opts MountOptions, gopts GlobalOptions, args
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

4
cmd/restic/cmd_prune.go
View File

@ -167,7 +167,7 @@ func runPrune(ctx context.Context, opts PruneOptions, gopts GlobalOptions) error
opts.unsafeRecovery = true
}
lock, ctx, err := lockRepoExclusive(ctx, repo)
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -488,7 +488,7 @@ func decidePackAction(ctx context.Context, opts PruneOptions, repo restic.Reposi
// Pack size does not fit and pack is needed => error
// If the pack is not needed, this is no error, the pack can
// and will be simply removed, see below.
Warnf("pack %s: calculated size %d does not match real size %d\nRun 'restic rebuild-index'.\n",
Warnf("pack %s: calculated size %d does not match real size %d\nRun 'restic repair index'.\n",
id.Str(), p.unusedSize+p.usedSize, packSize)
return errorSizeNotMatching
}

2
cmd/restic/cmd_recover.go
View File

@ -46,7 +46,7 @@ func runRecover(ctx context.Context, gopts GlobalOptions) error {
return err
}
lock, ctx, err := lockRepo(ctx, repo)
lock, ctx, err := lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

14
cmd/restic/cmd_repair.go Normal file
View File

@ -0,0 +1,14 @@
package main
import (
"github.com/spf13/cobra"
)
var cmdRepair = &cobra.Command{
Use: "repair",
Short: "Repair the repository",
}
func init() {
cmdRoot.AddCommand(cmdRepair)
}

38
cmd/restic/cmd_rebuild_index.go → cmd/restic/cmd_repair_index.go
View File

@ -7,15 +7,15 @@ import (
"github.com/restic/restic/internal/pack"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
)
var cmdRebuildIndex = &cobra.Command{
Use: "rebuild-index [flags]",
var cmdRepairIndex = &cobra.Command{
Use: "index [flags]",
Short: "Build a new index",
Long: `
The "rebuild-index" command creates a new index based on the pack files in the
The "repair index" command creates a new index based on the pack files in the
repository.
EXIT STATUS
@ -25,31 +25,43 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runRebuildIndex(cmd.Context(), rebuildIndexOptions, globalOptions)
return runRebuildIndex(cmd.Context(), repairIndexOptions, globalOptions)
},
}
// RebuildIndexOptions collects all options for the rebuild-index command.
type RebuildIndexOptions struct {
var cmdRebuildIndex = &cobra.Command{
Use: "rebuild-index [flags]",
Short: cmdRepairIndex.Short,
Long: cmdRepairIndex.Long,
Deprecated: `Use "repair index" instead`,
DisableAutoGenTag: true,
RunE: cmdRepairIndex.RunE,
}
// RepairIndexOptions collects all options for the repair index command.
type RepairIndexOptions struct {
ReadAllPacks bool
}
var rebuildIndexOptions RebuildIndexOptions
var repairIndexOptions RepairIndexOptions
func init() {
cmdRepair.AddCommand(cmdRepairIndex)
// add alias for old name
cmdRoot.AddCommand(cmdRebuildIndex)
f := cmdRebuildIndex.Flags()
f.BoolVar(&rebuildIndexOptions.ReadAllPacks, "read-all-packs", false, "read all pack files to generate new index from scratch")
for _, f := range []*pflag.FlagSet{cmdRepairIndex.Flags(), cmdRebuildIndex.Flags()} {
f.BoolVar(&repairIndexOptions.ReadAllPacks, "read-all-packs", false, "read all pack files to generate new index from scratch")
}
}
func runRebuildIndex(ctx context.Context, opts RebuildIndexOptions, gopts GlobalOptions) error {
func runRebuildIndex(ctx context.Context, opts RepairIndexOptions, gopts GlobalOptions) error {
repo, err := OpenRepository(ctx, gopts)
if err != nil {
return err
}
lock, ctx, err := lockRepoExclusive(ctx, repo)
lock, ctx, err := lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -58,7 +70,7 @@ func runRebuildIndex(ctx context.Context, opts RebuildIndexOptions, gopts Global
return rebuildIndex(ctx, opts, gopts, repo, restic.NewIDSet())
}
func rebuildIndex(ctx context.Context, opts RebuildIndexOptions, gopts GlobalOptions, repo *repository.Repository, ignorePacks restic.IDSet) error {
func rebuildIndex(ctx context.Context, opts RepairIndexOptions, gopts GlobalOptions, repo *repository.Repository, ignorePacks restic.IDSet) error {
var obsoleteIndexes restic.IDs
packSizeFromList := make(map[restic.ID]int64)
packSizeFromIndex := make(map[restic.ID]int64)

176
cmd/restic/cmd_repair_snapshots.go Normal file
View File

@ -0,0 +1,176 @@
package main
import (
"context"
"github.com/restic/restic/internal/backend"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/walker"
"github.com/spf13/cobra"
)
var cmdRepairSnapshots = &cobra.Command{
Use: "snapshots [flags] [snapshot ID] [...]",
Short: "Repair snapshots",
Long: `
The "repair snapshots" command repairs broken snapshots. It scans the given
snapshots and generates new ones with damaged directories and file contents
removed. If the broken snapshots are deleted, a prune run will be able to
clean up the repository.
The command depends on a correct index, thus make sure to run "repair index"
first!
WARNING
=======
Repairing and deleting broken snapshots causes data loss! It will remove broken
directories and modify broken files in the modified snapshots.
If the contents of directories and files are still available, the better option
is to run "backup" which in that case is able to heal existing snapshots. Only
use the "repair snapshots" command if you need to recover an old and broken
snapshot!
EXIT STATUS
===========
Exit status is 0 if the command was successful, and non-zero if there was any error.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runRepairSnapshots(cmd.Context(), globalOptions, repairSnapshotOptions, args)
},
}
// RepairOptions collects all options for the repair command.
type RepairOptions struct {
DryRun bool
Forget bool
restic.SnapshotFilter
}
var repairSnapshotOptions RepairOptions
func init() {
cmdRepair.AddCommand(cmdRepairSnapshots)
flags := cmdRepairSnapshots.Flags()
flags.BoolVarP(&repairSnapshotOptions.DryRun, "dry-run", "n", false, "do not do anything, just print what would be done")
flags.BoolVarP(&repairSnapshotOptions.Forget, "forget", "", false, "remove original snapshots after creating new ones")
initMultiSnapshotFilter(flags, &repairSnapshotOptions.SnapshotFilter, true)
}
func runRepairSnapshots(ctx context.Context, gopts GlobalOptions, opts RepairOptions, args []string) error {
repo, err := OpenRepository(ctx, globalOptions)
if err != nil {
return err
}
if !opts.DryRun {
var lock *restic.Lock
var err error
lock, ctx, err = lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
}
} else {
repo.SetDryRun()
}
snapshotLister, err := backend.MemorizeList(ctx, repo.Backend(), restic.SnapshotFile)
if err != nil {
return err
}
if err := repo.LoadIndex(ctx); err != nil {
return err
}
// Three error cases are checked:
// - tree is a nil tree (-> will be replaced by an empty tree)
// - trees which cannot be loaded (-> the tree contents will be removed)
// - files whose contents are not fully available (-> file will be modified)
rewriter := walker.NewTreeRewriter(walker.RewriteOpts{
RewriteNode: func(node *restic.Node, path string) *restic.Node {
if node.Type != "file" {
return node
}
ok := true
var newContent restic.IDs = restic.IDs{}
var newSize uint64
// check all contents and remove if not available
for _, id := range node.Content {
if size, found := repo.LookupBlobSize(id, restic.DataBlob); !found {
ok = false
} else {
newContent = append(newContent, id)
newSize += uint64(size)
}
}
if !ok {
Verbosef(" file %q: removed missing content\n", path)
} else if newSize != node.Size {
Verbosef(" file %q: fixed incorrect size\n", path)
}
// no-ops if already correct
node.Content = newContent
node.Size = newSize
return node
},
RewriteFailedTree: func(nodeID restic.ID, path string, _ error) (restic.ID, error) {
if path == "/" {
Verbosef(" dir %q: not readable\n", path)
// remove snapshots with invalid root node
return restic.ID{}, nil
}
// If a subtree fails to load, remove it
Verbosef(" dir %q: replaced with empty directory\n", path)
emptyID, err := restic.SaveTree(ctx, repo, &restic.Tree{})
if err != nil {
return restic.ID{}, err
}
return emptyID, nil
},
AllowUnstableSerialization: true,
})
changedCount := 0
for sn := range FindFilteredSnapshots(ctx, snapshotLister, repo, &opts.SnapshotFilter, args) {
Verbosef("\nsnapshot %s of %v at %s)\n", sn.ID().Str(), sn.Paths, sn.Time)
changed, err := filterAndReplaceSnapshot(ctx, repo, sn,
func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error) {
return rewriter.RewriteTree(ctx, repo, "/", *sn.Tree)
}, opts.DryRun, opts.Forget, "repaired")
if err != nil {
return errors.Fatalf("unable to rewrite snapshot ID %q: %v", sn.ID().Str(), err)
}
if changed {
changedCount++
}
}
Verbosef("\n")
if changedCount == 0 {
if !opts.DryRun {
Verbosef("no snapshots were modified\n")
} else {
Verbosef("no snapshots would be modified\n")
}
} else {
if !opts.DryRun {
Verbosef("modified %v snapshots\n", changedCount)
} else {
Verbosef("would modify %v snapshots\n", changedCount)
}
}
return nil
}

47
cmd/restic/cmd_restore.go
View File

@ -3,6 +3,7 @@ package main
import (
"context"
"strings"
"sync"
"time"
"github.com/restic/restic/internal/debug"
@ -10,6 +11,9 @@ import (
"github.com/restic/restic/internal/filter"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/restorer"
"github.com/restic/restic/internal/ui"
restoreui "github.com/restic/restic/internal/ui/restore"
"github.com/restic/restic/internal/ui/termstatus"
"github.com/spf13/cobra"
)
@ -31,7 +35,31 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runRestore(cmd.Context(), restoreOptions, globalOptions, args)
ctx := cmd.Context()
var wg sync.WaitGroup
cancelCtx, cancel := context.WithCancel(ctx)
defer func() {
// shutdown termstatus
cancel()
wg.Wait()
}()
term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
wg.Add(1)
go func() {
defer wg.Done()
term.Run(cancelCtx)
}()
// allow usage of warnf / verbosef
prevStdout, prevStderr := globalOptions.stdout, globalOptions.stderr
defer func() {
globalOptions.stdout, globalOptions.stderr = prevStdout, prevStderr
}()
stdioWrapper := ui.NewStdioWrapper(term)
globalOptions.stdout, globalOptions.stderr = stdioWrapper.Stdout(), stdioWrapper.Stderr()
return runRestore(ctx, restoreOptions, globalOptions, term, args)
},
}
@ -64,7 +92,9 @@ func init() {
flags.BoolVar(&restoreOptions.Verify, "verify", false, "verify restored files content")
}
func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions, args []string) error {
func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions,
term *termstatus.Terminal, args []string) error {
hasExcludes := len(opts.Exclude) > 0 || len(opts.InsensitiveExclude) > 0
hasIncludes := len(opts.Include) > 0 || len(opts.InsensitiveInclude) > 0
@ -124,7 +154,7 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions, a
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err
@ -145,7 +175,12 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions, a
return err
}
res := restorer.NewRestorer(ctx, repo, sn, opts.Sparse)
var progress *restoreui.Progress
if !globalOptions.Quiet && !globalOptions.JSON {
progress = restoreui.NewProgress(restoreui.NewProgressPrinter(term), calculateProgressInterval(!gopts.Quiet, gopts.JSON))
}
res := restorer.NewRestorer(ctx, repo, sn, opts.Sparse, progress)
totalErrors := 0
res.Error = func(location string, err error) error {
@ -209,6 +244,10 @@ func runRestore(ctx context.Context, opts RestoreOptions, gopts GlobalOptions, a
return err
}
if progress != nil {
progress.Finish()
}
if totalErrors > 0 {
return errors.Fatalf("There were %d errors\n", totalErrors)
}

59
cmd/restic/cmd_rewrite.go
View File

@ -87,36 +87,67 @@ func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *resti
return true
}
rewriter := walker.NewTreeRewriter(walker.RewriteOpts{
RewriteNode: func(node *restic.Node, path string) *restic.Node {
if selectByName(path) {
return node
}
Verbosef(fmt.Sprintf("excluding %s\n", path))
return nil
},
DisableNodeCache: true,
})
return filterAndReplaceSnapshot(ctx, repo, sn,
func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error) {
return rewriter.RewriteTree(ctx, repo, "/", *sn.Tree)
}, opts.DryRun, opts.Forget, "rewrite")
}
func filterAndReplaceSnapshot(ctx context.Context, repo restic.Repository, sn *restic.Snapshot, filter func(ctx context.Context, sn *restic.Snapshot) (restic.ID, error), dryRun bool, forget bool, addTag string) (bool, error) {
wg, wgCtx := errgroup.WithContext(ctx)
repo.StartPackUploader(wgCtx, wg)
var filteredTree restic.ID
wg.Go(func() error {
filteredTree, err = walker.FilterTree(wgCtx, repo, "/", *sn.Tree, &walker.TreeFilterVisitor{
SelectByName: selectByName,
PrintExclude: func(path string) { Verbosef(fmt.Sprintf("excluding %s\n", path)) },
})
var err error
filteredTree, err = filter(ctx, sn)
if err != nil {
return err
}
return repo.Flush(wgCtx)
})
err = wg.Wait()
err := wg.Wait()
if err != nil {
return false, err
}
if filteredTree.IsNull() {
if dryRun {
Verbosef("would delete empty snapshot\n")
} else {
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(ctx, h); err != nil {
return false, err
}
debug.Log("removed empty snapshot %v", sn.ID())
Verbosef("removed empty snapshot %v\n", sn.ID().Str())
}
return true, nil
}
if filteredTree == *sn.Tree {
debug.Log("Snapshot %v not modified", sn)
return false, nil
}
debug.Log("Snapshot %v modified", sn)
if opts.DryRun {
if dryRun {
Verbosef("would save new snapshot\n")
if opts.Forget {
if forget {
Verbosef("would remove old snapshot\n")
}
@ -125,10 +156,10 @@ func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *resti
// Always set the original snapshot id as this essentially a new snapshot.
sn.Original = sn.ID()
*sn.Tree = filteredTree
sn.Tree = &filteredTree
if !opts.Forget {
sn.AddTags([]string{"rewrite"})
if !forget {
sn.AddTags([]string{addTag})
}
// Save the new snapshot.
@ -136,8 +167,9 @@ func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *resti
if err != nil {
return false, err
}
Verbosef("saved new snapshot %v\n", id.Str())
if opts.Forget {
if forget {
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(ctx, h); err != nil {
return false, err
@ -145,7 +177,6 @@ func rewriteSnapshot(ctx context.Context, repo *repository.Repository, sn *resti
debug.Log("removed old snapshot %v", sn.ID())
Verbosef("removed old snapshot %v\n", sn.ID().Str())
}
Verbosef("saved new snapshot %v\n", id.Str())
return true, nil
}
@ -164,9 +195,9 @@ func runRewrite(ctx context.Context, opts RewriteOptions, gopts GlobalOptions, a
var err error
if opts.Forget {
Verbosef("create exclusive lock for repository\n")
lock, ctx, err = lockRepoExclusive(ctx, repo)
lock, ctx, err = lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
} else {
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
}
defer unlockRepo(lock)
if err != nil {

2
cmd/restic/cmd_snapshots.go
View File

@ -65,7 +65,7 @@ func runSnapshots(ctx context.Context, opts SnapshotOptions, gopts GlobalOptions
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_stats.go
View File

@ -83,7 +83,7 @@ func runStats(ctx context.Context, gopts GlobalOptions, args []string) error {
if !gopts.NoLock {
var lock *restic.Lock
lock, ctx, err = lockRepo(ctx, repo)
lock, ctx, err = lockRepo(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

2
cmd/restic/cmd_tag.go
View File

@ -111,7 +111,7 @@ func runTag(ctx context.Context, opts TagOptions, gopts GlobalOptions, args []st
if !gopts.NoLock {
Verbosef("create exclusive lock for repository\n")
var lock *restic.Lock
lock, ctx, err = lockRepoExclusive(ctx, repo)
lock, ctx, err = lockRepoExclusive(ctx, repo, gopts.RetryLock, gopts.JSON)
defer unlockRepo(lock)
if err != nil {
return err

43
cmd/restic/global.go
View File

@ -20,10 +20,12 @@ import (
"github.com/restic/restic/internal/backend/limiter"
"github.com/restic/restic/internal/backend/local"
"github.com/restic/restic/internal/backend/location"
"github.com/restic/restic/internal/backend/logger"
"github.com/restic/restic/internal/backend/rclone"
"github.com/restic/restic/internal/backend/rest"
"github.com/restic/restic/internal/backend/retry"
"github.com/restic/restic/internal/backend/s3"
"github.com/restic/restic/internal/backend/sema"
"github.com/restic/restic/internal/backend/sftp"
"github.com/restic/restic/internal/backend/smb"
"github.com/restic/restic/internal/backend/swift"
@ -43,7 +45,7 @@ import (
"golang.org/x/term"
)
var version = "0.15.1-dev (compiled manually)"
var version = "0.15.2-dev (compiled manually)"
// TimeFormat is the format used for all timestamps printed by restic.
const TimeFormat = "2006-01-02 15:04:05"
@ -60,6 +62,7 @@ type GlobalOptions struct {
Quiet bool
Verbose int
NoLock bool
RetryLock time.Duration
JSON bool
CacheDir string
NoCache bool
@ -116,6 +119,7 @@ func init() {
// use empty paremeter name as `-v, --verbose n` instead of the correct `--verbose=n` is confusing
f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)")
f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repository, this allows some operations on read-only repositories")
f.DurationVar(&globalOptions.RetryLock, "retry-lock", 0, "retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)")
f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache `directory`. (default: use system default cache directory)")
f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
@ -123,7 +127,7 @@ func init() {
f.StringVar(&globalOptions.TLSClientCertKeyFilename, "tls-client-cert", "", "path to a `file` containing PEM encoded TLS client certificate and private key")
f.BoolVar(&globalOptions.InsecureTLS, "insecure-tls", false, "skip TLS certificate verification when connecting to the repository (insecure)")
f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
f.Var(&globalOptions.Compression, "compression", "compression mode (only available for repository format version 2), one of (auto|off|max)")
f.Var(&globalOptions.Compression, "compression", "compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION)")
f.IntVar(&globalOptions.Limits.UploadKb, "limit-upload", 0, "limits uploads to a maximum `rate` in KiB/s. (default: unlimited)")
f.IntVar(&globalOptions.Limits.DownloadKb, "limit-download", 0, "limits downloads to a maximum `rate` in KiB/s. (default: unlimited)")
f.UintVar(&globalOptions.PackSize, "pack-size", 0, "set target pack `size` in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)")
@ -281,6 +285,7 @@ func Warnf(format string, args ...interface{}) {
if err != nil {
fmt.Fprintf(os.Stderr, "unable to write to stderr: %v\n", err)
}
debug.Log(format, args...)
}
// resolvePassword determines the password to be used for opening the repository.
@ -767,6 +772,9 @@ func open(ctx context.Context, s string, gopts GlobalOptions, opts options.Optio
return nil, errors.Fatalf("unable to open repository at %v: %v", location.StripPassword(s), err)
}
// wrap with debug logging and connection limiting
be = logger.New(sema.NewBackend(be))
// wrap backend if a test specified an inner hook
if gopts.backendInnerTestHook != nil {
be, err = gopts.backendInnerTestHook(be)
@ -811,29 +819,36 @@ func create(ctx context.Context, s string, opts options.Options) (restic.Backend
return nil, err
}
var be restic.Backend
switch loc.Scheme {
case "local":
return local.Create(ctx, cfg.(local.Config))
be, err = local.Create(ctx, cfg.(local.Config))
case "sftp":
return sftp.Create(ctx, cfg.(sftp.Config))
be, err = sftp.Create(ctx, cfg.(sftp.Config))
case "smb":
return smb.Create(ctx, cfg.(smb.Config))
be, err = smb.Create(ctx, cfg.(smb.Config))
case "s3":
return s3.Create(ctx, cfg.(s3.Config), rt)
be, err = s3.Create(ctx, cfg.(s3.Config), rt)
case "gs":
return gs.Create(cfg.(gs.Config), rt)
be, err = gs.Create(ctx, cfg.(gs.Config), rt)
case "azure":
return azure.Create(ctx, cfg.(azure.Config), rt)
be, err = azure.Create(ctx, cfg.(azure.Config), rt)
case "swift":
return swift.Open(ctx, cfg.(swift.Config), rt)
be, err = swift.Open(ctx, cfg.(swift.Config), rt)
case "b2":
return b2.Create(ctx, cfg.(b2.Config), rt)
be, err = b2.Create(ctx, cfg.(b2.Config), rt)
case "rest":
return rest.Create(ctx, cfg.(rest.Config), rt)
be, err = rest.Create(ctx, cfg.(rest.Config), rt)
case "rclone":
return rclone.Create(ctx, cfg.(rclone.Config))
be, err = rclone.Create(ctx, cfg.(rclone.Config))
default:
debug.Log("invalid repository scheme: %v", s)
return nil, errors.Fatalf("invalid scheme %q", loc.Scheme)
}
debug.Log("invalid repository scheme: %v", s)
return nil, errors.Fatalf("invalid scheme %q", loc.Scheme)
if err != nil {
return nil, err
}
return logger.New(sema.NewBackend(be)), nil
}

6
cmd/restic/integration_fuse_test.go
View File

@ -12,6 +12,7 @@ import (
"testing"
"time"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
@ -159,6 +160,11 @@ func TestMount(t *testing.T) {
t.Skip("Skipping fuse tests")
}
debugEnabled := debug.TestLogToStderr(t)
if debugEnabled {
defer debug.TestDisableLog(t)
}
env, cleanup := withTestEnvironment(t)
// must list snapshots more than once
env.gopts.backendTestHook = nil

135
cmd/restic/integration_repair_snapshots_test.go Normal file
View File

@ -0,0 +1,135 @@
package main
import (
"context"
"hash/fnv"
"io"
"math/rand"
"os"
"path/filepath"
"reflect"
"testing"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
)
func testRunRepairSnapshot(t testing.TB, gopts GlobalOptions, forget bool) {
opts := RepairOptions{
Forget: forget,
}
rtest.OK(t, runRepairSnapshots(context.TODO(), gopts, opts, nil))
}
func createRandomFile(t testing.TB, env *testEnvironment, path string, size int) {
fn := filepath.Join(env.testdata, path)
rtest.OK(t, os.MkdirAll(filepath.Dir(fn), 0o755))
h := fnv.New64()
_, err := h.Write([]byte(path))
rtest.OK(t, err)
r := rand.New(rand.NewSource(int64(h.Sum64())))
f, err := os.OpenFile(fn, os.O_CREATE|os.O_RDWR, 0o644)
rtest.OK(t, err)
_, err = io.Copy(f, io.LimitReader(r, int64(size)))
rtest.OK(t, err)
rtest.OK(t, f.Close())
}
func TestRepairSnapshotsWithLostData(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testRunInit(t, env.gopts)
createRandomFile(t, env, "foo/bar/file", 512*1024)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testListSnapshots(t, env.gopts, 1)
// damage repository
removePacksExcept(env.gopts, t, restic.NewIDSet(), false)
createRandomFile(t, env, "foo/bar/file2", 256*1024)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
snapshotIDs := testListSnapshots(t, env.gopts, 2)
testRunCheckMustFail(t, env.gopts)
// repair but keep broken snapshots
testRunRebuildIndex(t, env.gopts)
testRunRepairSnapshot(t, env.gopts, false)
testListSnapshots(t, env.gopts, 4)
testRunCheckMustFail(t, env.gopts)
// repository must be ok after removing the broken snapshots
testRunForget(t, env.gopts, snapshotIDs[0].String(), snapshotIDs[1].String())
testListSnapshots(t, env.gopts, 2)
_, err := testRunCheckOutput(env.gopts)
rtest.OK(t, err)
}
func TestRepairSnapshotsWithLostTree(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testRunInit(t, env.gopts)
createRandomFile(t, env, "foo/bar/file", 12345)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
oldSnapshot := testListSnapshots(t, env.gopts, 1)
oldPacks := testRunList(t, "packs", env.gopts)
// keep foo/bar unchanged
createRandomFile(t, env, "foo/bar2", 1024)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testListSnapshots(t, env.gopts, 2)
// remove tree for foo/bar and the now completely broken first snapshot
removePacks(env.gopts, t, restic.NewIDSet(oldPacks...))
testRunForget(t, env.gopts, oldSnapshot[0].String())
testRunCheckMustFail(t, env.gopts)
// repair
testRunRebuildIndex(t, env.gopts)
testRunRepairSnapshot(t, env.gopts, true)
testListSnapshots(t, env.gopts, 1)
_, err := testRunCheckOutput(env.gopts)
rtest.OK(t, err)
}
func TestRepairSnapshotsWithLostRootTree(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testRunInit(t, env.gopts)
createRandomFile(t, env, "foo/bar/file", 12345)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testListSnapshots(t, env.gopts, 1)
oldPacks := testRunList(t, "packs", env.gopts)
// remove all trees
removePacks(env.gopts, t, restic.NewIDSet(oldPacks...))
testRunCheckMustFail(t, env.gopts)
// repair
testRunRebuildIndex(t, env.gopts)
testRunRepairSnapshot(t, env.gopts, true)
testListSnapshots(t, env.gopts, 0)
_, err := testRunCheckOutput(env.gopts)
rtest.OK(t, err)
}
func TestRepairSnapshotsIntact(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, BackupOptions{}, env.gopts)
oldSnapshotIDs := testListSnapshots(t, env.gopts, 1)
// use an exclude that will not exclude anything
testRunRepairSnapshot(t, env.gopts, false)
snapshotIDs := testListSnapshots(t, env.gopts, 1)
rtest.Assert(t, reflect.DeepEqual(oldSnapshotIDs, snapshotIDs), "unexpected snapshot id mismatch %v vs. %v", oldSnapshotIDs, snapshotIDs)
testRunCheck(t, env.gopts)
}

95
cmd/restic/integration_test.go
View File

@ -71,6 +71,7 @@ func testRunBackupAssumeFailure(t testing.TB, dir string, target []string, opts
defer cleanup()
}
opts.GroupBy = restic.SnapshotGroupByOptions{Host: true, Path: true}
backupErr := runBackup(ctx, opts, gopts, term, target)
cancel()
@ -99,6 +100,13 @@ func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
return parseIDsFromReader(t, buf)
}
func testListSnapshots(t testing.TB, opts GlobalOptions, expected int) restic.IDs {
t.Helper()
snapshotIDs := testRunList(t, "snapshots", opts)
rtest.Assert(t, len(snapshotIDs) == expected, "expected %v snapshot, got %v", expected, snapshotIDs)
return snapshotIDs
}
func testRunRestore(t testing.TB, opts GlobalOptions, dir string, snapshotID restic.ID) {
testRunRestoreExcludes(t, opts, dir, snapshotID, nil)
}
@ -112,7 +120,7 @@ func testRunRestoreLatest(t testing.TB, gopts GlobalOptions, dir string, paths [
},
}
rtest.OK(t, runRestore(context.TODO(), opts, gopts, []string{"latest"}))
rtest.OK(t, runRestore(context.TODO(), opts, gopts, nil, []string{"latest"}))
}
func testRunRestoreExcludes(t testing.TB, gopts GlobalOptions, dir string, snapshotID restic.ID, excludes []string) {
@ -121,7 +129,7 @@ func testRunRestoreExcludes(t testing.TB, gopts GlobalOptions, dir string, snaps
Exclude: excludes,
}
rtest.OK(t, runRestore(context.TODO(), opts, gopts, []string{snapshotID.String()}))
rtest.OK(t, runRestore(context.TODO(), opts, gopts, nil, []string{snapshotID.String()}))
}
func testRunRestoreIncludes(t testing.TB, gopts GlobalOptions, dir string, snapshotID restic.ID, includes []string) {
@ -130,11 +138,11 @@ func testRunRestoreIncludes(t testing.TB, gopts GlobalOptions, dir string, snaps
Include: includes,
}
rtest.OK(t, runRestore(context.TODO(), opts, gopts, []string{snapshotID.String()}))
rtest.OK(t, runRestore(context.TODO(), opts, gopts, nil, []string{snapshotID.String()}))
}
func testRunRestoreAssumeFailure(t testing.TB, snapshotID string, opts RestoreOptions, gopts GlobalOptions) error {
err := runRestore(context.TODO(), opts, gopts, []string{snapshotID})
err := runRestore(context.TODO(), opts, gopts, nil, []string{snapshotID})
return err
}
@ -163,6 +171,11 @@ func testRunCheckOutput(gopts GlobalOptions) (string, error) {
return buf.String(), err
}
func testRunCheckMustFail(t testing.TB, gopts GlobalOptions) {
_, err := testRunCheckOutput(gopts)
rtest.Assert(t, err != nil, "expected non nil error after check of damaged repository")
}
func testRunDiffOutput(gopts GlobalOptions, firstSnapshotID string, secondSnapshotID string) (string, error) {
buf := bytes.NewBuffer(nil)
@ -187,7 +200,7 @@ func testRunRebuildIndex(t testing.TB, gopts GlobalOptions) {
globalOptions.stdout = os.Stdout
}()
rtest.OK(t, runRebuildIndex(context.TODO(), RebuildIndexOptions{}, gopts))
rtest.OK(t, runRebuildIndex(context.TODO(), RepairIndexOptions{}, gopts))
}
func testRunLs(t testing.TB, gopts GlobalOptions, snapshotID string) []string {
@ -362,6 +375,55 @@ func testBackup(t *testing.T, useFsSnapshot bool) {
testRunCheck(t, env.gopts)
}
func TestBackupWithRelativePath(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{}
// first backup
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1, "expected one snapshot, got %v", snapshotIDs)
firstSnapshotID := snapshotIDs[0]
// second backup, implicit incremental
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
// that the correct parent snapshot was used
latestSn, _ := testRunSnapshots(t, env.gopts)
rtest.Assert(t, latestSn != nil, "missing latest snapshot")
rtest.Assert(t, latestSn.Parent != nil && latestSn.Parent.Equal(firstSnapshotID), "second snapshot selected unexpected parent %v instead of %v", latestSn.Parent, firstSnapshotID)
}
func TestBackupParentSelection(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
testSetupBackupData(t, env)
opts := BackupOptions{}
// first backup
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata/0/0"}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1, "expected one snapshot, got %v", snapshotIDs)
firstSnapshotID := snapshotIDs[0]
// second backup, sibling path
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata/0/tests"}, opts, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 2, "expected two snapshots, got %v", snapshotIDs)
// third backup, incremental for the first backup
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata/0/0"}, opts, env.gopts)
// test that the correct parent snapshot was used
latestSn, _ := testRunSnapshots(t, env.gopts)
rtest.Assert(t, latestSn != nil, "missing latest snapshot")
rtest.Assert(t, latestSn.Parent != nil && latestSn.Parent.Equal(firstSnapshotID), "third snapshot selected unexpected parent %v instead of %v", latestSn.Parent, firstSnapshotID)
}
func TestDryRunBackup(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
@ -436,7 +498,16 @@ func TestBackupNonExistingFile(t *testing.T) {
testRunBackup(t, "", dirs, opts, env.gopts)
}
func removePacksExcept(gopts GlobalOptions, t *testing.T, keep restic.IDSet, removeTreePacks bool) {
func removePacks(gopts GlobalOptions, t testing.TB, remove restic.IDSet) {
r, err := OpenRepository(context.TODO(), gopts)
rtest.OK(t, err)
for id := range remove {
rtest.OK(t, r.Backend().Remove(context.TODO(), restic.Handle{Type: restic.PackFile, Name: id.String()}))
}
}
func removePacksExcept(gopts GlobalOptions, t testing.TB, keep restic.IDSet, removeTreePacks bool) {
r, err := OpenRepository(context.TODO(), gopts)
rtest.OK(t, err)
@ -1454,8 +1525,8 @@ func testRebuildIndex(t *testing.T, backendTestHook backendWrapper) {
t.Fatalf("expected no error from checker for test repository, got %v", err)
}
if !strings.Contains(out, "restic rebuild-index") {
t.Fatalf("did not find hint for rebuild-index command")
if !strings.Contains(out, "restic repair index") {
t.Fatalf("did not find hint for repair index command")
}
env.gopts.backendTestHook = backendTestHook
@ -1468,7 +1539,7 @@ func testRebuildIndex(t *testing.T, backendTestHook backendWrapper) {
}
if err != nil {
t.Fatalf("expected no error from checker after rebuild-index, got: %v", err)
t.Fatalf("expected no error from checker after repair index, got: %v", err)
}
}
@ -1549,7 +1620,7 @@ func TestRebuildIndexFailsOnAppendOnly(t *testing.T) {
env.gopts.backendTestHook = func(r restic.Backend) (restic.Backend, error) {
return &appendOnlyBackend{r}, nil
}
err := runRebuildIndex(context.TODO(), RebuildIndexOptions{}, env.gopts)
err := runRebuildIndex(context.TODO(), RepairIndexOptions{}, env.gopts)
if err == nil {
t.Error("expected rebuildIndex to fail")
}
@ -1837,8 +1908,8 @@ func TestListOnce(t *testing.T) {
testRunPrune(t, env.gopts, pruneOpts)
rtest.OK(t, runCheck(context.TODO(), checkOpts, env.gopts, nil))
rtest.OK(t, runRebuildIndex(context.TODO(), RebuildIndexOptions{}, env.gopts))
rtest.OK(t, runRebuildIndex(context.TODO(), RebuildIndexOptions{ReadAllPacks: true}, env.gopts))
rtest.OK(t, runRebuildIndex(context.TODO(), RepairIndexOptions{}, env.gopts))
rtest.OK(t, runRebuildIndex(context.TODO(), RepairIndexOptions{ReadAllPacks: true}, env.gopts))
}
func TestHardLink(t *testing.T) {

61
cmd/restic/lock.go
View File

@ -21,17 +21,29 @@ var globalLocks struct {
sync.Once
}
func lockRepo(ctx context.Context, repo restic.Repository) (*restic.Lock, context.Context, error) {
return lockRepository(ctx, repo, false)
func lockRepo(ctx context.Context, repo restic.Repository, retryLock time.Duration, json bool) (*restic.Lock, context.Context, error) {
return lockRepository(ctx, repo, false, retryLock, json)
}
func lockRepoExclusive(ctx context.Context, repo restic.Repository) (*restic.Lock, context.Context, error) {
return lockRepository(ctx, repo, true)
func lockRepoExclusive(ctx context.Context, repo restic.Repository, retryLock time.Duration, json bool) (*restic.Lock, context.Context, error) {
return lockRepository(ctx, repo, true, retryLock, json)
}
var (
retrySleepStart = 5 * time.Second
retrySleepMax = 60 * time.Second
)
func minDuration(a, b time.Duration) time.Duration {
if a <= b {
return a
}
return b
}
// lockRepository wraps the ctx such that it is cancelled when the repository is unlocked
// cancelling the original context also stops the lock refresh
func lockRepository(ctx context.Context, repo restic.Repository, exclusive bool) (*restic.Lock, context.Context, error) {
func lockRepository(ctx context.Context, repo restic.Repository, exclusive bool, retryLock time.Duration, json bool) (*restic.Lock, context.Context, error) {
// make sure that a repository is unlocked properly and after cancel() was
// called by the cleanup handler in global.go
globalLocks.Do(func() {
@ -43,7 +55,44 @@ func lockRepository(ctx context.Context, repo restic.Repository, exclusive bool)
lockFn = restic.NewExclusiveLock
}
lock, err := lockFn(ctx, repo)
var lock *restic.Lock
var err error
retrySleep := minDuration(retrySleepStart, retryLock)
retryMessagePrinted := false
retryTimeout := time.After(retryLock)
retryLoop:
for {
lock, err = lockFn(ctx, repo)
if err != nil && restic.IsAlreadyLocked(err) {
if !retryMessagePrinted {
if !json {
Verbosef("repo already locked, waiting up to %s for the lock\n", retryLock)
}
retryMessagePrinted = true
}
debug.Log("repo already locked, retrying in %v", retrySleep)
retrySleepCh := time.After(retrySleep)
select {
case <-ctx.Done():
return nil, ctx, ctx.Err()
case <-retryTimeout:
debug.Log("repo already locked, timeout expired")
// Last lock attempt
lock, err = lockFn(ctx, repo)
break retryLoop
case <-retrySleepCh:
retrySleep = minDuration(retrySleep*2, retrySleepMax)
}
} else {
// anything else, either a successful lock or another error
break retryLoop
}
}
if restic.IsInvalidLock(err) {
return nil, ctx, errors.Fatalf("%v\n\nthe `unlock --remove-all` command can be used to remove invalid locks. Make sure that no other restic process is accessing the repository when running the command", err)
}

120
cmd/restic/lock_test.go
View File

@ -3,11 +3,14 @@ package main
import (
"context"
"fmt"
"runtime"
"strings"
"testing"
"time"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/test"
rtest "github.com/restic/restic/internal/test"
)
@ -23,8 +26,8 @@ func openTestRepo(t *testing.T, wrapper backendWrapper) (*repository.Repository,
return repo, cleanup, env
}
func checkedLockRepo(ctx context.Context, t *testing.T, repo restic.Repository) (*restic.Lock, context.Context) {
lock, wrappedCtx, err := lockRepo(ctx, repo)
func checkedLockRepo(ctx context.Context, t *testing.T, repo restic.Repository, env *testEnvironment) (*restic.Lock, context.Context) {
lock, wrappedCtx, err := lockRepo(ctx, repo, env.gopts.RetryLock, env.gopts.JSON)
rtest.OK(t, err)
rtest.OK(t, wrappedCtx.Err())
if lock.Stale() {
@ -34,10 +37,10 @@ func checkedLockRepo(ctx context.Context, t *testing.T, repo restic.Repository)
}
func TestLock(t *testing.T) {
repo, cleanup, _ := openTestRepo(t, nil)
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo)
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo, env)
unlockRepo(lock)
if wrappedCtx.Err() == nil {
t.Fatal("unlock did not cancel context")
@ -45,12 +48,12 @@ func TestLock(t *testing.T) {
}
func TestLockCancel(t *testing.T) {
repo, cleanup, _ := openTestRepo(t, nil)
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
lock, wrappedCtx := checkedLockRepo(ctx, t, repo)
lock, wrappedCtx := checkedLockRepo(ctx, t, repo, env)
cancel()
if wrappedCtx.Err() == nil {
t.Fatal("canceled parent context did not cancel context")
@ -61,10 +64,10 @@ func TestLockCancel(t *testing.T) {
}
func TestLockUnlockAll(t *testing.T) {
repo, cleanup, _ := openTestRepo(t, nil)
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo)
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo, env)
_, err := unlockAll(0)
rtest.OK(t, err)
if wrappedCtx.Err() == nil {
@ -81,10 +84,10 @@ func TestLockConflict(t *testing.T) {
repo2, err := OpenRepository(context.TODO(), env.gopts)
rtest.OK(t, err)
lock, _, err := lockRepoExclusive(context.Background(), repo)
lock, _, err := lockRepoExclusive(context.Background(), repo, env.gopts.RetryLock, env.gopts.JSON)
rtest.OK(t, err)
defer unlockRepo(lock)
_, _, err = lockRepo(context.Background(), repo2)
_, _, err = lockRepo(context.Background(), repo2, env.gopts.RetryLock, env.gopts.JSON)
if err == nil {
t.Fatal("second lock should have failed")
}
@ -104,7 +107,7 @@ func (b *writeOnceBackend) Save(ctx context.Context, h restic.Handle, rd restic.
}
func TestLockFailedRefresh(t *testing.T) {
repo, cleanup, _ := openTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
repo, cleanup, env := openTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
return &writeOnceBackend{Backend: r}, nil
})
defer cleanup()
@ -117,7 +120,7 @@ func TestLockFailedRefresh(t *testing.T) {
refreshInterval, refreshabilityTimeout = ri, rt
}()
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo)
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo, env)
select {
case <-wrappedCtx.Done():
@ -136,11 +139,13 @@ type loggingBackend struct {
func (b *loggingBackend) Save(ctx context.Context, h restic.Handle, rd restic.RewindReader) error {
b.t.Logf("save %v @ %v", h, time.Now())
return b.Backend.Save(ctx, h, rd)
err := b.Backend.Save(ctx, h, rd)
b.t.Logf("save finished %v @ %v", h, time.Now())
return err
}
func TestLockSuccessfulRefresh(t *testing.T) {
repo, cleanup, _ := openTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
repo, cleanup, env := openTestRepo(t, func(r restic.Backend) (restic.Backend, error) {
return &loggingBackend{
Backend: r,
t: t,
@ -151,20 +156,99 @@ func TestLockSuccessfulRefresh(t *testing.T) {
t.Logf("test for successful lock refresh %v", time.Now())
// reduce locking intervals to be suitable for testing
ri, rt := refreshInterval, refreshabilityTimeout
refreshInterval = 40 * time.Millisecond
refreshabilityTimeout = 200 * time.Millisecond
refreshInterval = 60 * time.Millisecond
refreshabilityTimeout = 500 * time.Millisecond
defer func() {
refreshInterval, refreshabilityTimeout = ri, rt
}()
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo)
lock, wrappedCtx := checkedLockRepo(context.Background(), t, repo, env)
select {
case <-wrappedCtx.Done():
t.Fatal("lock refresh failed")
// don't call t.Fatal to allow the lock to be properly cleaned up
t.Error("lock refresh failed", time.Now())
// Dump full stacktrace
buf := make([]byte, 1024*1024)
n := runtime.Stack(buf, true)
buf = buf[:n]
t.Log(string(buf))
case <-time.After(2 * refreshabilityTimeout):
// expected lock refresh to work
}
// unlockRepo should not crash
unlockRepo(lock)
}
func TestLockWaitTimeout(t *testing.T) {
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
elock, _, err := lockRepoExclusive(context.TODO(), repo, env.gopts.RetryLock, env.gopts.JSON)
test.OK(t, err)
retryLock := 200 * time.Millisecond
start := time.Now()
lock, _, err := lockRepo(context.TODO(), repo, retryLock, env.gopts.JSON)
duration := time.Since(start)
test.Assert(t, err != nil,
"create normal lock with exclusively locked repo didn't return an error")
test.Assert(t, strings.Contains(err.Error(), "repository is already locked exclusively"),
"create normal lock with exclusively locked repo didn't return the correct error")
test.Assert(t, retryLock <= duration && duration < retryLock*3/2,
"create normal lock with exclusively locked repo didn't wait for the specified timeout")
test.OK(t, lock.Unlock())
test.OK(t, elock.Unlock())
}
func TestLockWaitCancel(t *testing.T) {
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
elock, _, err := lockRepoExclusive(context.TODO(), repo, env.gopts.RetryLock, env.gopts.JSON)
test.OK(t, err)
retryLock := 200 * time.Millisecond
cancelAfter := 40 * time.Millisecond
ctx, cancel := context.WithCancel(context.TODO())
time.AfterFunc(cancelAfter, cancel)
start := time.Now()
lock, _, err := lockRepo(ctx, repo, retryLock, env.gopts.JSON)
duration := time.Since(start)
test.Assert(t, err != nil,
"create normal lock with exclusively locked repo didn't return an error")
test.Assert(t, strings.Contains(err.Error(), "context canceled"),
"create normal lock with exclusively locked repo didn't return the correct error")
test.Assert(t, cancelAfter <= duration && duration < retryLock-10*time.Millisecond,
"create normal lock with exclusively locked repo didn't return in time")
test.OK(t, lock.Unlock())
test.OK(t, elock.Unlock())
}
func TestLockWaitSuccess(t *testing.T) {
repo, cleanup, env := openTestRepo(t, nil)
defer cleanup()
elock, _, err := lockRepoExclusive(context.TODO(), repo, env.gopts.RetryLock, env.gopts.JSON)
test.OK(t, err)
retryLock := 200 * time.Millisecond
unlockAfter := 40 * time.Millisecond
time.AfterFunc(unlockAfter, func() {
test.OK(t, elock.Unlock())
})
lock, _, err := lockRepo(context.TODO(), repo, retryLock, env.gopts.JSON)
test.OK(t, err)
test.OK(t, lock.Unlock())
}

9
doc/020_installation.rst
View File

@ -40,7 +40,7 @@ package from the official community repos, e.g. using ``apk``:
Arch Linux
==========
On `Arch Linux <https://www.archlinux.org/>`__, there is a package called ``restic``
On `Arch Linux <https://archlinux.org/>`__, there is a package called ``restic``
installed from the official community repos, e.g. with ``pacman -S``:
.. code-block:: console
@ -93,7 +93,7 @@ You may also install it using `MacPorts <https://www.macports.org/>`__:
Nix & NixOS
===========
If you are using `Nix <https://nixos.org/nix/>`__ or `NixOS <https://nixos.org/>`__
If you are using `Nix / NixOS <https://nixos.org>`__
there is a package available named ``restic``.
It can be installed using ``nix-env``:
@ -269,9 +269,10 @@ From Source
***********
restic is written in the Go programming language and you need at least
Go version 1.18. Building restic may also work with older versions of Go,
Go version 1.18. Building for Solaris requires at least Go version 1.20.
Building restic may also work with older versions of Go,
but that's not supported. See the `Getting
started <https://golang.org/doc/install>`__ guide of the Go project for
started <https://go.dev/doc/install>`__ guide of the Go project for
instructions how to install Go.
In order to build restic from source, execute the following steps:

23
doc/030_preparing_a_new_repo.rst
View File

@ -273,7 +273,7 @@ For an S3-compatible server that is not Amazon (like Minio, see below),
or is only available via HTTP, you can specify the URL to the server
like this: ``s3:http://server:port/bucket_name``.
.. note:: restic expects `path-style URLs <https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro>`__
.. note:: restic expects `path-style URLs <https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html>`__
like for example ``s3.us-west-2.amazonaws.com/bucket_name``.
Virtual-hosted–style URLs like ``bucket_name.s3.us-west-2.amazonaws.com``,
where the bucket name is part of the hostname are not supported. These must
@ -290,12 +290,11 @@ like this: ``s3:http://server:port/bucket_name``.
Minio Server
************
`Minio <https://www.minio.io>`__ is an Open Source Object Storage,
`Minio <https://min.io/>`__ is an Open Source Object Storage,
written in Go and compatible with Amazon S3 API.
- Download and Install `Minio
Server <https://minio.io/downloads/#minio-server>`__.
- You can also refer to https://docs.minio.io for step by step guidance
- Download and Install `Minio Download <https://min.io/download#/linux>`__.
- You can also refer to `Minio Docs <https://min.io/docs/minio/linux/>`__ for step by step guidance
on installation and getting started on Minio Client and Minio Server.
You must first setup the following environment variables with the
@ -350,7 +349,7 @@ this command.
Alibaba Cloud (Aliyun) Object Storage System (OSS)
**************************************************
`Alibaba OSS <https://www.alibabacloud.com/product/oss/>`__ is an
`Alibaba OSS <https://www.alibabacloud.com/product/object-storage-service>`__ is an
encrypted, secure, cost-effective, and easy-to-use object storage
service that enables you to store, back up, and archive large amounts
of data in the cloud.
@ -358,7 +357,7 @@ of data in the cloud.
Alibaba OSS is S3 compatible so it can be used as a storage provider
for a restic repository with a couple of extra parameters.
- Determine the correct `Alibaba OSS region endpoint <https://www.alibabacloud.com/help/doc-detail/31837.htm>`__ - this will be something like ``oss-eu-west-1.aliyuncs.com``
- Determine the correct `Alibaba OSS region endpoint <https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints>`__ - this will be something like ``oss-eu-west-1.aliyuncs.com``
- You'll need the region name too - this will be something like ``oss-eu-west-1``
You must first setup the following environment variables with the
@ -441,7 +440,7 @@ the naming convention of those variables follows the official Python Swift clien
Restic should be compatible with an `OpenStack RC file
<https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__
<https://docs.openstack.org/ocata/admin-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__
in most cases.
Once environment variables are set up, a new repository can be created. The
@ -614,9 +613,9 @@ The number of concurrent connections to the GCS service can be set with the
``-o gs.connections=10`` switch. By default, at most five parallel connections are
established.
.. _service account: https://cloud.google.com/iam/docs/service-accounts
.. _create a service account key: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console
.. _default authentication material: https://cloud.google.com/docs/authentication/production
.. _service account: https://cloud.google.com/iam/docs/service-account-overview
.. _create a service account key: https://cloud.google.com/iam/docs/keys-create-delete
.. _default authentication material: https://cloud.google.com/docs/authentication#service-accounts
.. _other-services:
@ -776,7 +775,7 @@ Password prompt on Windows
At the moment, restic only supports the default Windows console
interaction. If you use emulation environments like
`MSYS2 <https://msys2.github.io/>`__ or
`MSYS2 <https://www.msys2.org/>`__ or
`Cygwin <https://www.cygwin.com/>`__, which use terminals like
``Mintty`` or ``rxvt``, you may get a password error.

19
doc/040_backup.rst
View File

@ -225,7 +225,7 @@ the exclude options are:
- ``--exclude`` Specified one or more times to exclude one or more items
- ``--iexclude`` Same as ``--exclude`` but ignores the case of paths
- ``--exclude-caches`` Specified once to exclude folders containing `this special file <https://bford.info/cachedir/>`__
- ``--exclude-caches`` Specified once to exclude a folder's content if it contains `the special CACHEDIR.TAG file <https://bford.info/cachedir/>`__, but keep ``CACHEDIR.TAG``.
- ``--exclude-file`` Specified one or more times to exclude items listed in a given file
- ``--iexclude-file`` Same as ``exclude-file`` but ignores cases like in ``--iexclude``
- ``--exclude-if-present foo`` Specified one or more times to exclude a folder's content if it contains a file called ``foo`` (optionally having a given header, no wildcards for the file name supported)
@ -254,14 +254,14 @@ This instructs restic to exclude files matching the following criteria:
* All files matching ``*.go`` (second line in ``excludes.txt``)
* All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally,
see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for
syntax. Patterns are tested against the full path of a file/dir to be saved,
Patterns use the syntax of the Go function
`filepath.Match <https://pkg.go.dev/path/filepath#Match>`__
and are tested against the full path of a file/dir to be saved,
even if restic is passed a relative path to save. Empty lines and lines
starting with a ``#`` are ignored.
Environment variables in exclude files are expanded with `os.ExpandEnv
<https://golang.org/pkg/os/#ExpandEnv>`__, so ``/home/$USER/foo`` will be
<https://pkg.go.dev/os#ExpandEnv>`__, so ``/home/$USER/foo`` will be
expanded to ``/home/bob/foo`` for the user ``bob``. To get a literal dollar
sign, write ``$$`` to the file - this has to be done even when there's no
matching environment variable for the word following a single ``$``. Note
@ -381,7 +381,7 @@ contains one *pattern* per line. The file must be encoded as UTF-8, or UTF-16
with a byte-order mark. Leading and trailing whitespace is removed from the
patterns. Empty lines and lines starting with a ``#`` are ignored and each
pattern is expanded when read, such that special characters in it are expanded
using the Go function `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__
using the Go function `filepath.Glob <https://pkg.go.dev/path/filepath#Glob>`__
- please see its documentation for the syntax you can use in the patterns.
The argument passed to ``--files-from-verbatim`` must be the name of a text file
@ -533,8 +533,11 @@ Restic does not have a built-in way of scheduling backups, as it's a tool
that runs when executed rather than a daemon. There are plenty of different
ways to schedule backup runs on various different platforms, e.g. systemd
and cron on Linux/BSD and Task Scheduler in Windows, depending on one's
needs and requirements. When scheduling restic to run recurringly, please
make sure to detect already running instances before starting the backup.
needs and requirements. If you don't want to implement your own scheduling,
you can use `resticprofile <https://github.com/creativeprojects/resticprofile/#resticprofile>`__.
When scheduling restic to run recurringly, please make sure to detect already
running instances before starting the backup.
Space requirements
******************

3
doc/060_forget.rst
View File

@ -205,6 +205,7 @@ The ``forget`` command accepts the following policy options:
natural time boundaries and *not* relative to when you run ``forget``. Weeks
are Monday 00:00 to Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc.
They also only count hours/days/weeks/etc which have one or more snapshots.
A value of ``-1`` will be interpreted as "forever", i.e. "keep all".
.. note:: All duration related options (``--keep-{within,-*}``) ignore snapshots
with a timestamp in the future (relative to when the ``forget`` command is
@ -471,7 +472,7 @@ space. However, a **failed** ``prune`` run can cause the repository to become
**temporarily unusable**. Therefore, make sure that you have a stable connection to the
repository storage, before running this command. In case the command fails, it may become
necessary to manually remove all files from the `index/` folder of the repository and
run `rebuild-index` afterwards.
run `repair index` afterwards.
To prevent accidental usages of the ``--unsafe-recover-no-free-space`` option it is
necessary to first run ``prune --unsafe-recover-no-free-space SOME-ID`` and then replace

2
doc/070_encryption.rst
View File

@ -19,7 +19,7 @@ Encryption
the implementation looks sane and I guess the deduplication trade-off is worth
it. So… I’m going to use restic for my personal backups.*" `Filippo Valsorda`_
.. _Filippo Valsorda: https://blog.filippo.io/restic-cryptography/
.. _Filippo Valsorda: https://words.filippo.io/restic-cryptography/
**********************
Manage repository keys

12
doc/075_scripting.rst
View File

@ -22,18 +22,18 @@ Check if a repository is already initialized
You may find a need to check if a repository is already initialized,
perhaps to prevent your script from initializing a repository multiple
times. The command ``snapshots`` may be used for this purpose:
times. The command ``cat config`` may be used for this purpose:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
Fatal: unable to open config file: Stat: stat /srv/restic-repo/config: no such file or directory
$ restic -r /srv/restic-repo cat config
Fatal: unable to open config file: stat /srv/restic-repo/config: no such file or directory
Is there a repository at the following location?
/srv/restic-repo
If a repository does not exist, restic will return a non-zero exit code
and print an error message. Note that restic will also return a non-zero
exit code if a different error is encountered (e.g.: incorrect password
to ``snapshots``) and it may print a different error message. If there
are no errors, restic will return a zero exit code and print all the
snapshots.
to ``cat config``) and it may print a different error message. If there
are no errors, restic will return a zero exit code and print the repository
metadata.

194
doc/077_troubleshooting.rst Normal file
View File

@ -0,0 +1,194 @@
..
Normally, there are no heading levels assigned to certain characters as the structure is
determined from the succession of headings. However, this convention is used in Python’s
Style Guide for documenting which you may follow:
# with overline, for parts
* for chapters
= for sections
- for subsections
^ for subsubsections
" for paragraphs
#########################
Troubleshooting
#########################
The repository format used by restic is designed to be error resistant. In
particular, commands like, for example, ``backup`` or ``prune`` can be interrupted
at *any* point in time without damaging the repository. You might have to run
``unlock`` manually though, but that's it.
However, a repository might be damaged if some of its files are damaged or lost.
This can occur due to hardware failures, accidentally removing files from the
repository or bugs in the implementation of restic.
The following steps will help you recover a repository. This guide does not cover
all possible types of repository damages. Thus, if the steps do not work for you
or you are unsure how to proceed, then ask for help. Please always include the
check output discussed in the next section and what steps you've taken to repair
the repository so far.
* `Forum <https://forum.restic.net/>`_
* Our IRC channel ``#restic`` on ``irc.libera.chat``
Make sure that you **use the latest available restic version**. It can contain
bugfixes, and improvements to simplify the repair of a repository. It might also
contain a fix for your repository problems!
1. Find out what is damaged
***************************
The first step is always to check the repository.
.. code-block:: console
$ restic check --read-data
using temporary cache in /tmp/restic-check-cache-1418935501
repository 12345678 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-1418935501
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree 7ef8ebab:
id 7ef8ebabc59aadda1a237d23ca7abac487b627a9b86508aa0194690446ff71f6 not found in repository
[0:02] 100.00% 7 / 7 snapshots
read all data
[0:05] 100.00% 25 / 25 packs
Fatal: repository contains errors
.. note::
This will download the whole repository. If retrieving data from the backend is
expensive, then omit the ``--read-data`` option. Keep a copy of the check output
as it might be necessary later on!
If the output contains warnings that the ``ciphertext verification failed`` for
some blobs in the repository, then please ask for help in the forum or our IRC
channel. These errors are often caused by hardware problems which **must** be
investigated and fixed. Otherwise, the backup will be damaged again and again.
Similarly, if a repository is repeatedly damaged, please open an `issue on Github
<https://github.com/restic/restic/issues/new/choose>`_ as this could indicate a bug
somewhere. Please include the check output and additional information that might
help locate the problem.
2. Backup the repository
************************
Create a full copy of the repository if possible. Or at the very least make a
copy of the ``index`` and ``snapshots`` folders. This will allow you to roll back
the repository if the repair procedure fails. If your repository resides in a
cloud storage, then you can for example use `rclone <https://rclone.org/>`_ to
make such a copy.
Please disable all regular operations on the repository to prevent unexpected
changes. Especially, ``forget`` or ``prune`` must be disabled as they could
remove data unexpectedly.
.. warning::
If you suspect hardware problems, then you *must* investigate those first.
Otherwise, the repository will soon be damaged again.
Please take the time to understand what the commands described in the following
do. If you are unsure, then ask for help in the forum or our IRC channel. Search
whether your issue is already known and solved. Please take a look at the
`forum`_ and `Github issues <https://github.com/restic/restic/issues>`_.
3. Repair the index
*******************
Restic relies on its index to contain correct information about what data is
stored in the repository. Thus, the first step to repair a repository is to
repair the index:
.. code-block:: console
$ restic repair index
repository a14e5863 opened (version 2, compression level auto)
loading indexes...
getting pack files to read...
removing not found pack file 83ad44f59b05f6bce13376b022ac3194f24ca19e7a74926000b6e316ec6ea5a4
rebuilding index
[0:00] 100.00% 27 / 27 packs processed
deleting obsolete index files
[0:00] 100.00% 3 / 3 files deleted
done
This ensures that no longer existing files are removed from the index. All later
steps to repair the repository rely on a correct index. That is, you must always
repair the index first!
Please note that it is not recommended to repair the index unless the repository
is actually damaged.
4. Run all backups (optional)
*****************************
With a correct index, the ``backup`` command guarantees that newly created
snapshots can be restored successfully. It can also heal older snapshots,
if the missing data is also contained in the new snapshot.
Therefore, it is recommended to run all your ``backup`` tasks again. In some
cases, this is enough to fully repair the repository.
5. Remove missing data from snapshots
*************************************
If your repository is still missing data, then you can use the ``repair snapshots``
command to remove all inaccessible data from the snapshots. That is, this will
result in a limited amount of data loss. Using the ``--forget`` option, the
command will automatically remove the original, damaged snapshots.
.. code-block:: console
$ restic repair snapshots --forget
snapshot 6979421e of [/home/user/restic/restic] at 2022-11-02 20:59:18.617503315 +0100 CET)
file "/restic/internal/fuse/snapshots_dir.go": removed missing content
file "/restic/internal/restorer/restorer_unix_test.go": removed missing content
file "/restic/internal/walker/walker.go": removed missing content
saved new snapshot 7b094cea
removed old snapshot 6979421e
modified 1 snapshots
If you did not add the ``--forget`` option, then you have to manually delete all
modified snapshots using the ``forget`` command. In the example above, you'd have
to run ``restic forget 6979421e``.
6. Check the repository again
*****************************
Phew, we're almost done now. To make sure that the repository has been successfully
repaired please run ``check`` again.
.. code-block:: console
$ restic check --read-data
using temporary cache in /tmp/restic-check-cache-2569290785
repository a14e5863 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-2569290785
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00% 7 / 7 snapshots
read all data
[0:00] 100.00% 25 / 25 packs
no errors were found
If the ``check`` command did not complete with ``no errors were found``, then
the repository is still damaged. At this point, please ask for help at the
`forum`_ or our IRC channel ``#restic`` on ``irc.libera.chat``.

4
doc/090_participating.rst
View File

@ -33,8 +33,8 @@ The debug log will always contain all log messages restic generates. You
can also instruct restic to print some or all debug messages to stderr.
These can also be limited to e.g. a list of source files or a list of
patterns for function names. The patterns are globbing patterns (see the
documentation for `path.Glob <https://golang.org/pkg/path/#Glob>`__), multiple
patterns are separated by commas. Patterns are case sensitive.
documentation for `filepath.Match <https://pkg.go.dev/path/filepath#Match>`__).
Multiple patterns are separated by commas. Patterns are case sensitive.
Printing all log messages to the console can be achieved by setting the
file filter to ``*``:

8
doc/110_talks.rst
View File

@ -17,6 +17,8 @@ Talks
The following talks will be or have been given about restic:
- 2021-04-02: `The Changelog: Restic has your backup
(Podcast) <https://changelog.com/podcast/434>`__
- 2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
Brussels, Belgium
- 2016-01-29: `restic - Backups mal
@ -24,11 +26,11 @@ The following talks will be or have been given about restic:
Public lecture in German at `CCC Cologne
e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
- 2015-08-23: `A Solution to the Backup
Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
Inconvenience <https://programm.froscon.org/2015/events/1515.html>`__:
Lecture at `FROSCON 2015 <https://www.froscon.org/>`__ in Bonn, Germany
- 2015-02-01: `Lightning Talk at FOSDEM
2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
short introduction (with slightly outdated command line)
- 2015-01-27: `Talk about restic at CCC
Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
Aachen <https://video.fsmpi.rwth-aachen.de/cccac/4442>`__
(in German)

5
doc/design.rst
View File

@ -603,7 +603,10 @@ that the process is dead and considers the lock to be stale.
When a new lock is to be created and no other conflicting locks are
detected, restic creates a new lock, waits, and checks if other locks
appeared in the repository. Depending on the type of the other locks and
the lock to be created, restic either continues or fails.
the lock to be created, restic either continues or fails. If the
``--retry-lock`` option is specified, restic will retry
creating the lock periodically until it succeeds or the specified
timeout expires.
Read and Write Ordering
=======================

2
doc/developer_information.rst
View File

@ -10,7 +10,7 @@ refer to the documentation for the respective version. The binary produced
depends on the following things:
* The source code for the release
* The exact version of the official `Go compiler <https://golang.org>`__ used to produce the binaries (running ``restic version`` will print this)
* The exact version of the official `Go compiler <https://go.dev>`__ used to produce the binaries (running ``restic version`` will print this)
* The architecture and operating system the Go compiler runs on (Linux, ``amd64``)
* The build tags (for official binaries, it's the tag ``selfupdate``)
* The path where the source code is extracted to (``/restic``)

1
doc/index.rst
View File

@ -14,6 +14,7 @@ Restic Documentation
060_forget
070_encryption
075_scripting
077_troubleshooting
080_examples
090_participating
100_references

2
doc/man/restic-backup.1
View File

@ -205,7 +205,7 @@ Exit status is 3 if some source data could not be read (incomplete snapshot crea
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-cache.1
View File

@ -118,7 +118,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-cat.1
View File

@ -106,7 +106,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-check.1
View File

@ -123,7 +123,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-copy.1
View File

@ -147,7 +147,7 @@ new destination repository using the "init" command.
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-diff.1
View File

@ -126,7 +126,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-dump.1
View File

@ -129,7 +129,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-find.1
View File

@ -151,7 +151,7 @@ It can also be used to search for restic blobs or trees for troubleshooting.
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH EXAMPLE

2
doc/man/restic-forget.1
View File

@ -217,7 +217,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-generate.1
View File

@ -127,7 +127,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-init.1
View File

@ -134,7 +134,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-key.1
View File

@ -118,7 +118,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-list.1
View File

@ -106,7 +106,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-ls.1
View File

@ -141,7 +141,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-migrate.1
View File

@ -112,7 +112,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-mount.1
View File

@ -190,7 +190,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-prune.1
View File

@ -135,7 +135,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-rebuild-index.1
View File

@ -111,7 +111,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-recover.1
View File

@ -108,7 +108,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-restore.1
View File

@ -151,7 +151,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-rewrite.1
View File

@ -159,7 +159,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-self-update.1
View File

@ -113,7 +113,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-snapshots.1
View File

@ -130,7 +130,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-stats.1
View File

@ -152,7 +152,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-tag.1
View File

@ -137,7 +137,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-unlock.1
View File

@ -110,7 +110,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic-version.1
View File

@ -107,7 +107,7 @@ Exit status is 0 if the command was successful, and non-zero if there was any er
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

2
doc/man/restic.1
View File

@ -100,7 +100,7 @@ directories in an encrypted repository stored on different backends.
.PP
\fB-v\fP, \fB--verbose\fP[=0]
be verbose (specify multiple times or a level using --verbose=\fB\fCn\fR, max level/times is 2)
be verbose (specify multiple times or a level using --verbose=n``, max level/times is 2)
.SH SEE ALSO

17
doc/manual_rest.rst
View File

@ -26,7 +26,7 @@ Usage help is available:
dump Print a backed-up file to stdout
find Find a file, a directory or restic IDs
forget Remove snapshots from the repository
generate Generate manual pages and auto-completion files (bash, fish, zsh)
generate Generate manual pages and auto-completion files (bash, fish, zsh, powershell)
help Help about any command
init Initialize a new repository
key Manage keys (passwords)
@ -35,8 +35,8 @@ Usage help is available:
migrate Apply migrations
mount Mount the repository
prune Remove unneeded data from the repository
rebuild-index Build a new index
recover Recover data from the repository not referenced by snapshots
repair Repair the repository
restore Extract the data from a snapshot
rewrite Rewrite snapshots to exclude unwanted files
self-update Update the restic binary
@ -50,7 +50,7 @@ Usage help is available:
--cacert file file to load root certificates from (default: use system certificates)
--cache-dir directory set the cache directory. (default: use system default cache directory)
--cleanup-cache auto remove old cache directories
--compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default auto)
--compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION) (default auto)
-h, --help help for restic
--insecure-tls skip TLS certificate verification when connecting to the repository (insecure)
--json set output mode to JSON for commands that support it
@ -66,6 +66,7 @@ Usage help is available:
-q, --quiet do not output comprehensive progress report
-r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY)
--repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
--retry-lock duration retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
--tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key
-v, --verbose be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
@ -105,6 +106,7 @@ command:
--files-from-raw file read the files to backup from file (can be combined with file args; can be specified multiple times)
--files-from-verbatim file read the files to backup from file (can be combined with file args; can be specified multiple times)
-f, --force force re-reading the target files/directories (overrides the "parent" flag)
-g, --group-by group group snapshots by host, paths and/or tags, separated by comma (disable grouping with '') (default host,paths)
-h, --help help for backup
-H, --host hostname set the hostname for the snapshot manually. To prevent an expensive rescan use the "parent" flag
--iexclude pattern same as --exclude pattern but ignores the casing of filenames
@ -113,8 +115,8 @@ command:
--ignore-inode ignore inode number changes when checking for modified files
--no-scan do not run scanner to estimate size of backup
-x, --one-file-system exclude other file systems, don't cross filesystem boundaries and subvolumes
--parent snapshot use this parent snapshot (default: last snapshot in the repository that has the same target files/directories, and is not newer than the snapshot time)
--read-concurrency n read n file concurrently (default: $RESTIC_READ_CONCURRENCY or 2)
--parent snapshot use this parent snapshot (default: latest snapshot in the group determined by --group-by and not newer than the timestamp determined by --time)
--read-concurrency n read n files concurrently (default: $RESTIC_READ_CONCURRENCY or 2)
--stdin read backup from stdin
--stdin-filename filename filename to use when reading from stdin (default "stdin")
--tag tags add tags for the new snapshot in the format `tag[,tag,...]` (can be specified multiple times) (default [])
@ -126,7 +128,7 @@ command:
--cacert file file to load root certificates from (default: use system certificates)
--cache-dir directory set the cache directory. (default: use system default cache directory)
--cleanup-cache auto remove old cache directories
--compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default auto)
--compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION) (default auto)
--insecure-tls skip TLS certificate verification when connecting to the repository (insecure)
--json set output mode to JSON for commands that support it
--key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
@ -141,6 +143,7 @@ command:
-q, --quiet do not output comprehensive progress report
-r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY)
--repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
--retry-lock duration retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
--tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key
-v, --verbose be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
@ -224,7 +227,7 @@ locks with the following command:
d369ccc7d126594950bf74f0a348d5d98d9e99f3215082eb69bf02dc9b3e464c
The ``find`` command searches for a given
`pattern <https://golang.org/pkg/path/filepath/#Match>`__ in the
`pattern <https://pkg.go.dev/path/filepath#Match>`__ in the
repository.
.. code-block:: console

2
docker/Dockerfile
View File

@ -11,7 +11,7 @@ RUN go run build.go
FROM alpine:latest AS restic
RUN apk add --update --no-cache ca-certificates fuse openssh-client tzdata
RUN apk add --update --no-cache ca-certificates fuse openssh-client tzdata jq
COPY --from=builder /go/src/github.com/restic/restic/restic /usr/bin

Some files were not shown because too many files have changed in this diff Show More