Compare commits

...

3815 Commits

Author SHA1 Message Date
Michael Eischer 660679c2f6
Merge pull request #4835 from MichaelEischer/fix-list-cancel
backend/retry: do not log final error if context was canceled
2024-06-01 19:17:30 +02:00
Michael Eischer 456ebfb0c7
Merge pull request #4840 from restic/dependabot/go_modules/google.golang.org/api-0.182.0
build(deps): bump google.golang.org/api from 0.181.0 to 0.182.0
2024-06-01 13:06:49 +00:00
Michael Eischer 830e87059a
Merge pull request #4842 from restic/dependabot/github_actions/docker/login-action-3.2.0
build(deps): bump docker/login-action from 3.1.0 to 3.2.0
2024-06-01 13:06:46 +00:00
Michael Eischer a26beb9c1c
Merge pull request #4843 from restic/dependabot/github_actions/golangci/golangci-lint-action-6
build(deps): bump golangci/golangci-lint-action from 5 to 6
2024-06-01 13:06:43 +00:00
dependabot[bot] 46b0fac11e
build(deps): bump golangci/golangci-lint-action from 5 to 6
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 5 to 6.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:48:46 +00:00
dependabot[bot] fced29b479
build(deps): bump docker/login-action from 3.1.0 to 3.2.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](e92390c5fb...0d4c9c5ea7)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:48:43 +00:00
dependabot[bot] 05d7885a87
build(deps): bump google.golang.org/api from 0.181.0 to 0.182.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.181.0 to 0.182.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.181.0...v0.182.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 01:02:01 +00:00
Michael Eischer 38654a3bd7 backend/retry: do not log final error if context was canceled
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
2024-05-30 18:48:52 +02:00
Michael Eischer 2280fbfd2e
Merge pull request #4810 from konidev20/fix-gh-4768-add-custom-user-agent-for-http-client
Allow custom User-Agent to be specified for outgoing requests
2024-05-30 16:34:52 +00:00
Michael Eischer 7e0ee5974f
Merge pull request #4816 from MichaelEischer/skip-if-unchanged
backup: add support for `--skip-if-unchanged`
2024-05-30 15:39:08 +02:00
Srigovind Nayak de7b418bbe http: allow custom `User-Agent` for outgoing HTTP requests 2024-05-30 15:38:06 +02:00
Michael Eischer c103c0830e add changelog for skip-if-unchanged 2024-05-30 15:25:44 +02:00
Michael Eischer 513135b238 doc: slightly modernize backup examples 2024-05-30 15:25:44 +02:00
Michael Eischer cdd210185d
Merge pull request #4823 from MichaelEischer/fix-check-output
Fix mangled check output
2024-05-30 15:17:27 +02:00
Michael Eischer 5649334099 backup: document --skip-if-unchanged 2024-05-30 15:14:32 +02:00
Michael Eischer 6869bdaaa8 backup: implement --skip-if-unchanged 2024-05-30 15:14:31 +02:00
Michael Eischer 13127f3f6b check: remove output mangling workaround 2024-05-30 15:09:27 +02:00
Michael Eischer 939b537c80 check/migrate: convert to use termstatus to prevent mangled output
Errors reported by check would result in corrupted output.
2024-05-30 15:09:27 +02:00
Michael Eischer 7b4f81d964
Merge pull request #4829 from greatroar/termstatus
termstatus: Do formatting in Message, not Terminal
2024-05-30 11:57:44 +00:00
Michael Eischer a5c7a83470
Merge pull request #4819 from restic/update-dependencies
Update dependencies
2024-05-30 13:55:46 +02:00
Michael Eischer 52b3fee42b
Merge pull request #4818 from MichaelEischer/move-code
Move cache / index / hashing / pack packages
2024-05-30 13:54:31 +02:00
Michael Eischer e95cfff923
Merge pull request #4826 from MichaelEischer/fix-docs-warninigs
doc: fix warnings
2024-05-30 13:53:49 +02:00
Michael Eischer fe8fbcc5ed
Merge pull request #4822 from MichaelEischer/consistent-backup-source-name
backup: refer to backed up data as backup source not target
2024-05-30 13:53:25 +02:00
Michael Eischer 3d4a620089
Merge pull request #4824 from MichaelEischer/fix-cache-race
bloblru: Fix flaky test due to race condition
2024-05-30 13:52:34 +02:00
greatroar 46f04bc5f0 termstatus: Do formatting in Message, not Terminal 2024-05-27 19:05:39 +02:00
Michael Eischer a651af70d6 doc: fix warings 2024-05-26 14:32:09 +02:00
Michael Eischer 1c6067d93d bloblru: variable name cleanup 2024-05-26 12:42:46 +02:00
Michael Eischer 21ce03cff2 bloblru: move defer outside critical section 2024-05-26 12:38:20 +02:00
Michael Eischer 0ad0b7ca7c bloblru: fix race condition that can compute value multiple times 2024-05-26 12:37:24 +02:00
Michael Eischer 619ce8bfa2
Merge pull request #4820 from MichaelEischer/forget-extend-descritpion
forget: explain oldest snapshot special case in command help
2024-05-26 10:04:43 +00:00
Michael Eischer 107f18849c doc: update help output in manual_rest.rst 2024-05-25 23:16:22 +02:00
Michael Eischer 18990dbed8 backup: refer to backed up data as backup source not target
A backup reads from the source and writes it to the backup target
(repository). The name "target" is kept within the source code as it's
used there excessively.
2024-05-25 23:13:37 +02:00
Michael Eischer 3828313974
Merge pull request #4821 from MichaelEischer/master
update `repair packs` changelog
2024-05-25 17:02:21 +02:00
Michael Eischer 04c181dbd0 update-repair-packs-changelog 2024-05-25 16:42:55 +02:00
Michael Eischer 21528c9f02
Merge pull request #4805 from plant99/mkdirall-cachedir
prepareCheckCache function: Add MkdirAll() to restic-cache directory if it doesn't exist
2024-05-25 14:30:21 +00:00
Shivashis Padhi 0271bb97f8
check: enhance check command to create cache directory if it doesn't exist 2024-05-25 19:15:42 +05:30
Michael Eischer 2c7ebf4c7f forget: explain oldest snapshot special case in command help 2024-05-25 15:10:21 +02:00
Michael Eischer 7ea508e7b8
Merge pull request #4815 from greatroar/termstatus
Termstatus refactoring
2024-05-25 12:08:53 +00:00
Michael Eischer 496e57f956 hashing: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 5e0ea8fcfa pack: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 50ec408302 index: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 58bbb27b4c downgrade minio to keep go 1.19 support 2024-05-25 12:52:08 +02:00
Michael Eischer c5260373ca update backoff/xxhash libraries 2024-05-25 12:37:14 +02:00
Michael Eischer b2f7a878cd update backend libraries
minio-go is kept at 7.0.67 as later versions require Go 1.21.
2024-05-25 12:36:16 +02:00
Michael Eischer f20bf5eeba update golang.org/x packages 2024-05-25 12:35:28 +02:00
Michael Eischer 8e5d7d719c cache: move to backend package 2024-05-24 23:04:06 +02:00
Michael Eischer 80132e71d8
Merge pull request #4808 from MichaelEischer/insecure-no-password
Implement `--insecure-no-password` option.
2024-05-24 22:58:25 +02:00
Michael Eischer 130506250f document insecure-no-password 2024-05-24 22:38:20 +02:00
Michael Eischer 1d2277b4c3 Add --insecure-no-password option
This also includes two derived options `--from-insecure-no-password`
used for commands that require specifying a source repository. And
`--new-insecure-no-password` for the `key add` and `key passwd`
commands.

Specifying `--insecure-no-password` disabled the password prompt and
immediately uses an empty password. Passing a password via CLI option or
environment variable at the same time is an error.
2024-05-24 22:38:20 +02:00
Michael Eischer d4b0d21199 key add/passwd: deduplicate options setup and remove globals
The current pattern of using a global options variable is problematic.
2024-05-24 22:38:20 +02:00
Michael Eischer 55cb8d174a
Merge pull request #4354 from MichaelEischer/associated-data
Reduce prune memory usage
2024-05-24 22:26:22 +02:00
Michael Eischer 436afbff23 add changelog for memory efficient prune 2024-05-24 22:18:14 +02:00
Michael Eischer 3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer 462b82a060 index: reduce size of compressed indexes
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
2024-05-24 22:18:14 +02:00
Michael Eischer 77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer 2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer 93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer ff4775a15f
Merge pull request #4812 from MichaelEischer/streaming-index-rewrite
Resumable prune & memory-efficient index rewrite
2024-05-24 21:41:30 +02:00
Michael Eischer 860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer f680a2331d add changelog for streaming index rewrite 2024-05-24 21:33:17 +02:00
Michael Eischer 027cc64737 repository: fix prune heuristic to allow resuming interrupted runs
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
2024-05-24 21:33:17 +02:00
Michael Eischer e52033a8bd index: slightly reduce Rewrite concurrency
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
2024-05-24 21:33:17 +02:00
Michael Eischer 57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer 2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer 5f7b48e65f index: replace Save() method with Rewrite and SaveFallback
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
2024-05-24 21:33:17 +02:00
Michael Eischer ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer 72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer 68fa0e0305 prune: no longer disable automatic index updates
this allows prune to resume an interrupted prune run.
2024-05-24 21:33:17 +02:00
Michael Eischer 9aa0c90fb2 index: remove supersedes field
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
2024-05-24 21:33:17 +02:00
Michael Eischer 76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer 04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 550d1eeac3 repository: remove SaveIndex from interface
The method is now only indirectly accessible via Prune or RepairIndex.
2024-05-24 21:33:17 +02:00
Michael Eischer fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer 447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer 6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer 864995271e repository: unwrap BlobHandle parameters of LookupBlob
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer 1266a4932f repository: fix parameter order of LookupBlobSize
All methods should use blobType followed by ID.
2024-05-24 21:33:17 +02:00
Michael Eischer e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer 0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer 0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer 8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer 3eeb6723cd
Merge pull request #4764 from MichaelEischer/safe-keep-tag
Prevent unsafe uses of `forget --keep-tag`
2024-05-24 20:51:20 +02:00
Michael Eischer 3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer 7948912b31 document --unsafe-allow-remove-all option 2024-05-24 20:45:33 +02:00
Michael Eischer 3f46808898 add forget safety net changelogs 2024-05-24 20:45:33 +02:00
Michael Eischer 82ca0030b7 forget: test --unsafe-allow-remove-all and --keep-tags safety check 2024-05-24 20:45:33 +02:00
Michael Eischer 57f9739573 forget: Add --unsafe-allow-remove-all option
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
2024-05-24 20:45:33 +02:00
Michael Eischer 5b7952e426 forget: return error if no policy was specified 2024-05-24 20:45:33 +02:00
Michael Eischer c0e1f36830 forget: refuse deleting the last snapshot in a snapshot group
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
2024-05-24 20:45:33 +02:00
Michael Eischer d106ad6921 restic: regenerate snapshot keep policy golden test files 2024-05-24 20:45:33 +02:00
Michael Eischer 16ef4d515b
Merge pull request #4784 from MichaelEischer/rework-backend-retries
Rework backend retries
2024-05-24 20:29:54 +02:00
Michael Eischer e4a48085ae backend/retry: feature flag new retry behavior 2024-05-24 20:24:02 +02:00
Michael Eischer 723247c8e5 add changelog for longer retries 2024-05-24 20:24:02 +02:00
Michael Eischer b1266867d2 repository: wait max 1 minutes for lock removal if context is canceled
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
2024-05-24 20:24:02 +02:00
Michael Eischer 98709a4372 retry: reduce total number of retries
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions

The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.

Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
Michael Eischer 512cd6ef07 retry: ensure that there's always at least one retry
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
Michael Eischer a60ee9b764 retry: limit retries based on elapsed time not count
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
Michael Eischer a3633cad9e retry: explicitly log failed requests
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
Michael Eischer b9cbf623fa
Merge pull request #4814 from MichaelEischer/fix-tmp-docs
doc: fix tmpdir documentation for windows
2024-05-24 18:17:59 +02:00
greatroar 66d03c797e ui, termstatus: Move WrapStdio
Saves some imports. ui still needs to import ui/termstatus from
message.go.
2024-05-21 11:25:16 +02:00
greatroar 0b56214473 ui: Simplify stdio wrapper
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
2024-05-21 11:23:32 +02:00
greatroar 7f439a9c34 ui: Inline lineWriter into StdioWrapper 2024-05-21 11:00:09 +02:00
Michael Eischer 4021e67d97 doc: fix tmpdir documentation for windows 2024-05-20 20:48:29 +02:00
Michael Eischer 8898f61717
Merge pull request #4809 from MichaelEischer/update-changelog
add retries for corrupted blobs to changelog
2024-05-18 23:04:13 +02:00
Michael Eischer 5f23baabcc add retries for corrupted blobs to changelog 2024-05-18 23:03:24 +02:00
Michael Eischer 9c5bac6f25
Merge pull request #4799 from letmaik/letmaik/azure-force-cli-credential
Azure: add option to force use of CLI credential
2024-05-18 20:22:15 +00:00
Michael Eischer c56ecec9aa azure: deduplicate cli and default credentials case 2024-05-18 22:15:54 +02:00
Maik Riechert 355f520936 Azure: add option to force use of CLI credential 2024-05-18 22:15:54 +02:00
Michael Eischer 1dfe1b8732
Merge pull request #4802 from MichaelEischer/backend-cleanups
Repository: Remove Backend() method
2024-05-18 22:02:45 +02:00
Michael Eischer 223aa22cb0 replace some uses of restic.Repository with finegrained interfaces 2024-05-18 21:42:51 +02:00
Michael Eischer 291c9677de restic/repository: remove Backend() method 2024-05-18 21:42:51 +02:00
Michael Eischer 673496b091 repository: clean cache between CheckPack retries
The cache cleanup pattern is also used in ListPack etc.
2024-05-18 21:42:51 +02:00
Michael Eischer 3d2410ed50 Replace some repo.RemoveUnpacked usages
These will eventually be blocked as they do not delete Snapshots.
2024-05-18 21:42:51 +02:00
Michael Eischer d2c26e33f3 repository: remove further usages of repo.Backend() 2024-05-18 21:42:51 +02:00
Michael Eischer 8a425c2f0a remove usages of repo.Backend() from tests 2024-05-18 21:42:51 +02:00
Michael Eischer aa4647f773 repository: unexport PackBlobIterator 2024-05-18 21:42:51 +02:00
Michael Eischer 94e863885c check: move verification of individual pack file to repository 2024-05-18 21:42:50 +02:00
Michael Eischer e40943a75d restic: remove backend usage from lock test 2024-05-18 21:38:31 +02:00
Michael Eischer 67e2ba0d40 repository: Lock requires *repository.Repository
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer d8b184b3d3 repository: convert test helper to return *repository.Repository 2024-05-18 21:38:31 +02:00
Michael Eischer a1ca5e15c4 migrations: add temporary hack for s3_layout
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer 34d90aecf9 migrations: move logic of upgrade_repo_v2 to repository package
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer ab9077bc13 replace usages of backend.Remove() with repository.RemoveUnpacked()
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer 8274f5b101 prune: remove Backend.IsNotExist()
Only handling one specific error is not particularly useful.
2024-05-18 21:38:31 +02:00
Michael Eischer 9795198189 debug: remove Backend.Stat() usage 2024-05-18 21:38:31 +02:00
Michael Eischer 0c1ba6d95d backend: remove unused Location method 2024-05-18 21:38:31 +02:00
Michael Eischer eb6c653f89
Merge pull request #4800 from MichaelEischer/cleanup-load
Retry loading of corrupted data from backend / cache
2024-05-18 21:34:54 +02:00
Michael Eischer 74d90653e0 check: use ReadFull to load pack header in checkPack
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
2024-05-18 21:28:54 +02:00
Michael Eischer 8f8d872a68 fix compatibility with go 1.19 2024-05-18 21:28:54 +02:00
Michael Eischer ff0744b3af check: test checkPack retries 2024-05-18 21:28:54 +02:00
Michael Eischer 987c3b250c repository: test retries of ListPack 2024-05-18 21:28:54 +02:00
Michael Eischer bf16096771 repository: test LoadBlob retries 2024-05-18 21:28:54 +02:00
Michael Eischer 4f45668b7c repository: rework and extend LoadRaw tests 2024-05-18 21:28:54 +02:00
Michael Eischer ac805d6838 cache: cleanup debug logs 2024-05-18 21:28:54 +02:00
Michael Eischer 5214af88e2 cache: test forget behavior 2024-05-18 21:28:54 +02:00
Michael Eischer 3ff063e913 check: verify pack a second time if broken 2024-05-18 21:28:54 +02:00
Michael Eischer 385cee09dc repository: fix caching of tree packs in LoadBlobsFromPack 2024-05-18 21:28:54 +02:00
Michael Eischer e734746f75 cache: forget cached file at most once
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.

By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
2024-05-18 21:28:54 +02:00
Michael Eischer 97a307df1a cache: Always use cached file if it exists
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
2024-05-18 21:28:54 +02:00
Michael Eischer 8cce06d915 repair packs: drop experimental warning
This warning should already have been removed once the feature flag was
dropped.
2024-05-18 21:28:54 +02:00
Michael Eischer 433a6aad29 repository: remove redundant blob loading fallback from RepairPacks
LoadBlobsFromPack already implements the same fallback behavior.
2024-05-18 21:28:54 +02:00
Michael Eischer e401af07b2 check: fix error message formatting 2024-05-18 21:28:54 +02:00
Michael Eischer 7017adb7e9 repository: retry failed ListPack once 2024-05-18 21:28:54 +02:00
Michael Eischer e33ce7f408 repository: retry failed LoadBlob once 2024-05-18 21:28:54 +02:00
Michael Eischer 2ace242f36 repository: make reloading broken files explicit 2024-05-18 21:28:54 +02:00
Michael Eischer e9390352a7 cache: code cleanups 2024-05-18 21:26:00 +02:00
Michael Eischer 503c8140b1 repository: unify blob decoding code 2024-05-18 21:26:00 +02:00
Michael Eischer 6563f1d2ca repository: remove redundant debug log 2024-05-18 21:26:00 +02:00
Michael Eischer 021fb49559 repository: Implement repository.LoadUnpacked using LoadRaw
Both functions were using a similar implementation.
2024-05-18 21:26:00 +02:00
Michael Eischer 779c8d3527 debug/repair packs/upgrade repo v2: use repository.LoadRaw
This replaces calling the low-level backend.Load() method.
2024-05-18 21:26:00 +02:00
Michael Eischer 1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer 47232bf8b0 backend: move LimitReadCloser to util package
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
Michael Eischer dcd151147c
Merge pull request #4803 from restic/permanent-retry-failure
Do not retry permanent backend failures
2024-05-18 20:07:06 +02:00
Michael Eischer 53d15bcd1b retry: add circuit breaker to load method
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
Michael Eischer 394c8ca3ed rest/rclone/s3/sftp/swift: move short file detection behind feature gate
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
Michael Eischer 6328b7e1f5 replace "too small" with "too short" in error messages 2024-05-18 19:59:26 +02:00
Michael Eischer 53561474d9 update changelog with persistent backend error handling 2024-05-18 19:59:26 +02:00
Michael Eischer aeb7eb245c retry: do not retry permanent errors
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
Michael Eischer bf8cc59889 Use generic backend-error-redesign feature flag instead of http-timeouts
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
Michael Eischer 4740528a0b backend: add tests for IsPermanentError 2024-05-18 19:54:52 +02:00
Michael Eischer 6a85df7297 backend: add IsPermanentError() method to interface 2024-05-18 19:54:52 +02:00
Michael Eischer cfc420664a mem: stricter handling of out of bounds requests 2024-05-18 19:54:52 +02:00
Michael Eischer d40f23e716 azure/b2/gs/s3/swift: adapt cloud backend 2024-05-18 19:54:51 +02:00
Michael Eischer e793c002ec local: stricter handling of short files 2024-05-18 19:54:21 +02:00
Michael Eischer b4895ebd76 rest: rework error reporting and report too short files 2024-05-18 19:54:21 +02:00
Michael Eischer eaa3f81d6b sftp: check for truncated files without an extra backend request 2024-05-18 19:54:21 +02:00
Michael Eischer c6d74458ee sftp: improve handling of too short files 2024-05-18 19:54:21 +02:00
Michael Eischer 7ed560a201
Merge pull request #4796 from MichaelEischer/parallel-dump-load
dump: Parallelize loading large files
2024-05-14 22:35:44 +02:00
Michael Eischer 92221c2a6d
Merge pull request #4708 from zmanda/windows-securitydesc
Back up and restore SecurityDescriptors on Windows
2024-05-12 14:14:39 +00:00
Michael Eischer b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
Michael Eischer e4f9bce384
Merge pull request #4792 from restic/request-watchdog
backend: enforce that backend HTTP requests make progress
2024-05-09 23:55:30 +02:00
Michael Eischer 3740700ddc add http timeouts to changelog 2024-05-09 23:46:17 +02:00
Michael Eischer ebd01a4675 backend: add tests for watchdogRoundTripper 2024-05-09 23:46:17 +02:00
Michael Eischer 8778670232 backend: cancel stuck http requests
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
Michael Eischer 0987c731ec backend: configure protocol-level connection health checks
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
aneesh-n a4fd1b91e5
Fix review comments
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
2024-05-06 16:54:08 -06:00
Michael Eischer e184538ddf dump: add changelog 2024-05-05 12:12:21 +02:00
Michael Eischer 4d55a62ada bloblru: add test for GetOrCompute 2024-05-05 12:00:25 +02:00
Michael Eischer 7cce667f92 fuse: switch to use bloblru.GetOrCompute 2024-05-05 11:38:42 +02:00
Michael Eischer bd03af2feb dump: add GetOrCompute to bloblru cache 2024-05-05 11:38:42 +02:00
Michael Eischer 45509eafc8 dump: load blobs of a file from repository in parallel 2024-05-05 11:38:42 +02:00
Michael Eischer 24c1822220
Merge pull request #4794 from flow-c/master
Update 060_forget.rst
2024-05-04 08:25:06 +00:00
flow-c d4477a5a99
Update 060_forget.rst
Replace deprecated `-1` with `unlimited` in calendar-related `--keep-*` options
2024-05-04 09:32:25 +02:00
Michael Eischer ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling
Rework repository.StreamPacks & better restorer error handling
2024-05-01 16:37:41 +02:00
Michael Eischer 676f0dc60d add changelog 2024-05-01 16:28:57 +02:00
Michael Eischer 1e57057953
Merge pull request #4789 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.3.2
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.3.1 to 1.3.2
2024-05-01 10:45:47 +00:00
Michael Eischer 1ba0af6993
Merge pull request #4787 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.8
build(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8
2024-05-01 10:44:33 +00:00
Michael Eischer ffc41ae62a
Merge pull request #4786 from restic/dependabot/go_modules/golang.org/x/net-0.24.0
build(deps): bump golang.org/x/net from 0.23.0 to 0.24.0
2024-05-01 10:41:26 +00:00
Michael Eischer 4832c2fbfa
Merge pull request #4790 from restic/dependabot/github_actions/golangci/golangci-lint-action-5
build(deps): bump golangci/golangci-lint-action from 4 to 5
2024-05-01 10:37:37 +00:00
dependabot[bot] 30609ae6b2
build(deps): bump golangci/golangci-lint-action from 4 to 5
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 4 to 5.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:45:43 +00:00
dependabot[bot] 502e5867a5
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.1...sdk/storage/azblob/v1.3.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:39 +00:00
dependabot[bot] 18a6d6b408
build(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.7 to 1.17.8.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.7...v1.17.8)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:22 +00:00
dependabot[bot] 3bb88e8307
build(deps): bump golang.org/x/net from 0.23.0 to 0.24.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.23.0 to 0.24.0.
- [Commits](https://github.com/golang/net/compare/v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 01:02:11 +00:00
aneesh-n 672f6cd776
Fix review comments for privileges and security flags 2024-04-29 17:29:51 -06:00
aneesh-n 08c6945d61
Fix review comments 2024-04-29 16:21:38 -06:00
Aneesh N 3f76b902e5
Merge branch 'master' into windows-securitydesc 2024-04-29 14:40:34 -06:00
Michael Eischer ccac7c7fb3
Merge pull request #3067 from DRON-666/vss-options
Add options to fine tune VSS snapshots
2024-04-29 18:09:47 +00:00
DRON-666 ccd35565ee
s/sec./seconds 2024-04-29 01:48:22 +03:00
DRON-666 125dba23c5 Rearange code 2024-04-29 01:27:34 +03:00
DRON-666 7ee889bb0d Use S_FALSE and MaxInt 2024-04-29 01:25:25 +03:00
DRON-666 90b168eb6c isMountPointExcluded to isMountPointIncluded 2024-04-29 01:23:50 +03:00
DRON-666 24330c19a8 Use kebab case in option names 2024-04-29 01:21:33 +03:00
DRON-666 5703e5a652 Fix texts and comments 2024-04-29 01:18:46 +03:00
DRON-666 0a8f9c5d9c vss: Add tests for "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 739d3243d9 vss: Update docs and changelog 2024-04-28 22:45:21 +03:00
DRON-666 bb0f93ef3d vss: Add "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 3bac1f0135 vss: Fix issues reported by linters 2024-04-28 22:45:21 +03:00
DRON-666 88c509e3e9 vss: Change `ErrorHandler` signature
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
2024-04-28 22:44:16 +03:00
DRON-666 9d3d915e2c vss: Add some tests 2024-04-28 22:44:16 +03:00
DRON-666 9182e6bab5 vss: Update docs and changelog 2024-04-28 22:44:16 +03:00
DRON-666 c4f67c0064 vss: Add volume filtering
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
2024-04-28 22:44:15 +03:00
DRON-666 7470e5356e vss: Add "timeout" option
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
2024-04-28 22:44:15 +03:00
DRON-666 78dbc5ec58 vss: Add initial support for extended options 2024-04-28 22:44:15 +03:00
Michael Eischer a1d682ce0e add changelog for sftp performance fix 2024-04-28 11:58:08 +02:00
Michael Eischer 935327d480 sftp: slightly increase write concurrency
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
Michael Eischer 669a669603 sftp: Fix upload performance issue
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
Michael Eischer faffd15d13
Merge pull request #4734 from maouw/enhancement/envvar-for-host
Add support for specifying --host via environment variable
2024-04-24 20:00:15 +00:00
Michael Eischer 347e9d0765 complete RESITC_HOST environment handling & test 2024-04-24 21:52:39 +02:00
Altan Orhon 871ea1eaf3 Add support for specifying --host via environment variable
This commit adds support for specifying the `--host` option via the `RESTIC_HOST` environment variable. This is done by extending option processing in `cmd_backup.go` and for `restic.SnapshotFilter` in `find.go`.
2024-04-24 21:49:42 +02:00
Michael Eischer a7b5e09902
Merge pull request #4753 from MichaelEischer/remove-cleanup-handlers
Replace cleanup handlers with context based command cancelation
2024-04-24 21:34:19 +02:00
Michael Eischer 3f9d50865d
Merge pull request #4776 from MichaelEischer/cleanup-backend-open
unify backend open and create
2024-04-24 21:24:27 +02:00
Michael Eischer 5f263752d7 init: also apply limiter for non-HTTP backend 2024-04-24 20:42:30 +02:00
Michael Eischer 484dbb1cf4 get rid of a few global variables 2024-04-22 22:39:33 +02:00
Michael Eischer 940a3159b5 let index.Each() and pack.Size() return error on canceled context
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer 31624aeffd Improve command shutdown on context cancellation 2024-04-22 22:31:38 +02:00
Michael Eischer 910927670f mount: fix exit code on cancellation 2024-04-22 22:27:19 +02:00
Michael Eischer 6f2a4dea21 remove global shutdown hook 2024-04-22 22:27:19 +02:00
Michael Eischer 699ef5e9de debug: replace cleanup handler usage in profiling setup 2024-04-22 22:27:19 +02:00
Michael Eischer eb710a28e8 use standalone shutdown hook for readPasswordTerminal
move terminal restoration into readPasswordTerminal
2024-04-22 22:27:19 +02:00
Michael Eischer 86c7909f41 mount: use standalone shutdown hook via goroutine 2024-04-22 22:27:19 +02:00
Michael Eischer 93135dc705 lock: drop cleanup handler 2024-04-22 22:27:19 +02:00
Michael Eischer 21a7cb405c check: replace cleanup handler 2024-04-22 22:27:19 +02:00
Michael Eischer b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
Michael Eischer 2e6c43c695
Merge pull request #4761 from MichaelEischer/fix-cache-race
cache: ignore ErrNotExist during cleanup of old files
2024-04-22 21:46:06 +02:00
Michael Eischer f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob
repository: Better error message if blob is larger than 4GB
2024-04-22 21:45:06 +02:00
Michael Eischer 20d8eed400 repository: streamPack: separate requests for gap larger than 1MB
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
2024-04-22 21:21:23 +02:00
Michael Eischer cf700d8794 repository: streamPack: reuse zstd decoder 2024-04-22 21:21:23 +02:00
Michael Eischer 666a0b0bdb repository: streamPack: replace streaming with chunked download
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.

Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.

To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer 621012dac0 repository: Add blob loading fallback to LoadBlobsFromPack
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer 6c6dceade3 global: unify backend open and create 2024-04-19 22:26:14 +02:00
Michael Eischer 10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
Michael Eischer 228b35f074
Merge pull request #4769 from will-ca/patch-1
Tiny wording clarification in `restic-stats.1`.
2024-04-18 19:00:39 +00:00
will-ca 6aced61c72
Tiny docs wording clarification. 2024-04-18 07:29:55 +00:00
Michael Eischer 4d22412e0c
Merge pull request #4766 from coderwander/master
Fix struct names
2024-04-18 06:18:19 +00:00
coderwander a82ed71de7 Fix struct names
Signed-off-by: coderwander <770732124@qq.com>
2024-04-18 10:02:09 +08:00
Michael Eischer 2173c69280
Merge pull request #4770 from testwill/close_files
fix: close files
2024-04-17 16:50:20 +00:00
Michael Eischer 001bb71676 repair packs: Properly close backup files 2024-04-17 18:32:30 +02:00
Michael Eischer c9191ea72c forget: cleanup verbose output on snapshot deletion error 2024-04-14 14:17:40 +02:00
Michael Eischer 09587e6c08 repository: duplicate a few blobs in prune tests 2024-04-14 13:57:19 +02:00
Michael Eischer defd7ae729 prune/repair index: reset in-memory index after command
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
2024-04-14 13:46:24 +02:00
Michael Eischer 038586dc9d repository: add minimal test for prune 2024-04-14 13:45:17 +02:00
Michael Eischer d8622c86eb prune: clean up internal interface 2024-04-14 13:45:15 +02:00
Michael Eischer 8d507c1372 repository: add basic test for RepairIndex 2024-04-14 13:45:15 +02:00
Michael Eischer 310db03c0e repair index: improve log output if index cannot be deleted
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
2024-04-14 13:45:13 +02:00
Michael Eischer 7d1b9cde34 repository: use normal Init method in tests 2024-04-14 13:45:11 +02:00
Michael Eischer b25fc2c89d repository: remove redundant flushes from tests 2024-04-14 13:45:10 +02:00
Michael Eischer c65459cd8a repository: speed up tests 2024-04-14 13:45:10 +02:00
Michael Eischer eda9f7beb4 ui/progress: add helper to print messages during tests 2024-04-14 13:45:08 +02:00
Michael Eischer 35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer 7ba5e95a82 check: allow tests to only verify pack&index integrity 2024-04-14 13:45:04 +02:00
Michael Eischer 4c9a10ca37 repair packs: deduplicate index rebuild 2024-04-14 13:45:02 +02:00
Michael Eischer 85e4021619 prune: move additional option checks to repository 2024-04-14 13:44:58 +02:00
Michael Eischer 55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout
Deprecate s3legacy layout
2024-04-11 22:09:34 +02:00
Michael Eischer fc3b548625 prune: move logic into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer df9d4b455d prune: prepare for moving code to repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 866ddf5698 repair index: refactor code into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 32a234b67e prune/forget/repair index: convert output to use progress.Printer 2024-04-10 21:30:52 +02:00
Michael Eischer 739d11c2eb forget: replace usage of DeleteFilesChecked
This simplifies refactoring prune into the repository package.
2024-04-10 21:30:52 +02:00
Michael Eischer 591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Michael Eischer 8efc3a8b7d
Merge pull request #4668 from MichaelEischer/backup-xattr-parent-enoperm
backup: Ignore xattr.list permission error for parent directories
2024-04-10 21:25:28 +02:00
Michael Eischer bf054c09d2 backup: Ignore xattr.list permission error for parent directories
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
2024-04-10 20:46:15 +02:00
Michael Eischer 0747cf5319 cache: ignore ErrNotExist during cleanup of old files
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
2024-04-10 19:25:51 +02:00
Michael Eischer 6091029fd6
Merge pull request #4756 from mgeisler/patch-1
doc: fix typo in 047_tuning_backup_parameters.rst
2024-04-07 19:24:09 +00:00
Martin Geisler 09d2183351
doc: fix typo in 047_tuning_backup_parameters.rst 2024-04-07 18:05:53 +02:00
Michael Eischer a4b7ebecfc
Merge pull request #4750 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.40.0
build(deps): bump cloud.google.com/go/storage from 1.39.0 to 1.40.0
2024-04-03 20:10:41 +00:00
dependabot[bot] ba136b31b8
build(deps): bump cloud.google.com/go/storage from 1.39.0 to 1.40.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.39.0...spanner/v1.40.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-03 20:03:22 +00:00
Michael Eischer f328111a6e
Merge pull request #4751 from restic/dependabot/go_modules/golang.org/x/net-0.23.0
build(deps): bump golang.org/x/net from 0.21.0 to 0.23.0
2024-04-03 19:52:29 +00:00
Michael Eischer 9fb017e67a
Merge pull request #4745 from MichaelEischer/full-id-key-list
key list: include full key id in JSON output
2024-04-03 21:50:02 +02:00
Michael Eischer 49f98f25fc
Merge pull request #4742 from MichaelEischer/consistent-rtest-import
Use consistent alias for interal/test package
2024-04-03 21:47:32 +02:00
dependabot[bot] 96c602a6de
build(deps): bump golang.org/x/net from 0.21.0 to 0.23.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0.
- [Commits](https://github.com/golang/net/compare/v0.21.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-03 19:41:38 +00:00
Michael Eischer 1d0a20dd58
Merge pull request #4748 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.10.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.9.2 to 1.10.0
2024-04-03 19:34:47 +00:00
Michael Eischer 6cca1d5705
Merge pull request #4655 from ae-govau/unixsocket
Enhancement: option to send HTTP over unix socket
2024-04-03 19:29:21 +00:00
dependabot[bot] f8a72ac2a3
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.9.2 to 1.10.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.9.2...sdk/azcore/v1.10.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-01 02:32:19 +00:00
Michael Eischer 5145c8f9c0 key list: include full key id in JSON output 2024-03-31 12:25:20 +02:00
Michael Eischer 831fc4413d
Merge pull request #4737 from stephan0307/3117
json output forget command: added id's in snapshots within reasons object
2024-03-29 13:11:24 +00:00
Stephan Paul df07814ec2 forget json output: added id's in snapshots within reasons object
In order to evaluate the keep reasons for snapshots, there should be also the id's
to compare it with snapshots within the keep object. (See also Issue #3117)

In order to avoid output parameters also changed function addJSONSnapshots to asJSONSnapshots
2024-03-29 11:30:00 +01:00
Michael Eischer ec2b79834a use consistent alias for interal/test package 2024-03-29 00:24:03 +01:00
Michael Eischer 510f6f06b0
Merge pull request #4709 from MichaelEischer/refactor-locking
Refactor locking into repository package
2024-03-28 23:53:09 +01:00
Michael Eischer 07eb6c315b add changelog for locking refactor 2024-03-28 23:46:58 +01:00
Michael Eischer 5e98f1e2eb repository: fix test setup race conditions 2024-03-28 23:17:02 +01:00
Michael Eischer 8155dbe711 correctly lock repository in integration tests 2024-03-28 23:17:02 +01:00
Michael Eischer d18726cd70 ls: add missing read lock
As `ls` reads data from the repository, it must acquire a read lock
unless `--no-lock` was specified. The old behavior is equivalent to `ls
--no-lock`.
2024-03-28 23:17:02 +01:00
Michael Eischer dc441c57a7 repository: unify repository initialization in tests
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer 3ba1fa3cee repository: remove a few global variables 2024-03-28 23:17:02 +01:00
Michael Eischer 044e8bf821 repository: parallelize lock tests 2024-03-28 23:17:02 +01:00
Michael Eischer e8df50fa3c repository: remove global list of locks 2024-03-28 22:46:33 +01:00
Michael Eischer cbb5f89252 lock: move code to repository package 2024-03-28 22:46:33 +01:00
Michael Eischer 118a69a84b lock: replace lockRepo(Exclusive) with openWith(Read/Write/Exclusive)Lock
The new functions much better convey the intent behind the lock
request. This allows cleanly integrating noLock (for read) and dryRun
(write/exclusive) handling.

There are only minor changes to existing behavior with two exceptions:
- `tag` no longer accepts the `--no-lock` flag. As it replaces files in
  the repository, this always requires an exclusive lock.
- `debug examine` now returns an error if both `--extract-pack` and
  `--no-lock` are given.
2024-03-28 22:46:33 +01:00
Michael Eischer 7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics
Store snapshot statistics & print snapshot size
2024-03-28 22:41:45 +01:00
Michael Eischer 71b6284155
Merge pull request #4006 from MichaelEischer/deviceID-only-for-hardlinks
archiver: only store deviceID for hardlinks
2024-03-28 22:33:28 +01:00
Michael Eischer cf81f8ced6 stats: only check for hardlinks for files with more than one link 2024-03-28 21:29:27 +01:00
Michael Eischer 21cf38fe96 add changelog for deviceID only for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer d705741571 backup: test that deviceID is only stored for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer a9b3d86c4f features: remove example feature 2024-03-28 19:12:07 +01:00
Michael Eischer a26d6ffa72 archiver: move deviceID handling behind feature flag 2024-03-28 19:12:07 +01:00
Michael Eischer 2ba21fe72b archiver: only store deviceID for hardlinks
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
2024-03-28 19:12:07 +01:00
Michael Eischer 870904d3ae
Merge pull request #4731 from facutuesca/powershell-completion-doc
doc: Add instructions to configure PowerShell completions
2024-03-28 17:46:55 +00:00
Facundo Tuesca 15555c9898 doc: Use consistent case for PROFILE env variable in PowerShell 2024-03-28 18:35:18 +01:00
Michael Eischer 63a2350c9e
Merge pull request #4741 from MichaelEischer/ci-upgrade-golangci-lint
CI: Update golangci-lint to version 1.57.1
2024-03-28 18:27:05 +01:00
Michael Eischer 8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index
Deprecated legacy index format
2024-03-28 18:24:47 +01:00
Michael Eischer 4f4979f4e8
Merge pull request #4740 from MichaelEischer/ci-upgrade-docker-actions
CI: update docker actions
2024-03-28 18:16:38 +01:00
Michael Eischer 1497525e15 CI: Update golangci-lint to version 1.57.1 2024-03-28 18:15:50 +01:00
Michael Eischer a8face3a25
Merge pull request #4739 from MichaelEischer/ci-allow-annotations
CI: Allow golangci-lint to annotate PRs
2024-03-28 18:02:16 +01:00
Michael Eischer aee6d311f1 CI: update docker actions 2024-03-28 18:01:48 +01:00
Michael Eischer 6ac7519188 add changelog for rest unix socket support 2024-03-28 17:41:41 +01:00
Michael Eischer add37fcd9f CI: uses rest-server from master branch until unix sockets are released 2024-03-28 17:41:41 +01:00
Adam Eijdenberg 6e775d3787 Enhancement: option to send HTTP over unix socket
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
Michael Eischer 5c4a4b4a30 CI: Allow golangci-lint to annotate PRs 2024-03-28 17:09:59 +01:00
Michael Eischer d1d773cfcd
Merge pull request #4727 from avoidalone/master
fix some typos
2024-03-17 18:52:40 +00:00
Facundo Tuesca 521713fc94 doc: Add instructions to configure PowerShell completions 2024-03-16 18:54:27 +01:00
avoidalone ac948fccda fix some typos
Signed-off-by: avoidalone <wuguangdong@outlook.com>
2024-03-11 14:35:12 +08:00
rawtaz 9284f7413a
Merge pull request #4725 from leoheitmannruiz/master
Minor README.md cleanups
2024-03-10 12:24:07 +00:00
rawtaz 1287b977b4
Merge pull request #4726 from leoheitmannruiz/patch-1
Capitalize Homebrew and minor edit in wording
2024-03-10 12:23:48 +00:00
Leo Heitmann Ruiz 00f762373f
Capitalize Homebrew 2024-03-10 00:20:26 +01:00
Leo Heitmann Ruiz 9f3e1462c0
Minor README.md cleanups 2024-03-09 23:56:16 +01:00
Michael Eischer 69ca12d2eb check: treat legacy index format as errors 2024-03-09 18:36:33 +01:00
Michael Eischer 98a6817d01 add changelog for legacy index deprecation 2024-03-09 18:35:00 +01:00
Michael Eischer f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer 1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Michael Eischer 396a61a992
Merge pull request #4666 from MichaelEischer/feature-flags
Implement feature flags
2024-03-09 17:36:29 +01:00
Michael Eischer a9b64cd7ad features: print warning for stable/depreacted feature flags 2024-03-09 17:29:52 +01:00
Michael Eischer fe68d2cafb add feature flag documentation 2024-03-09 17:29:52 +01:00
Michael Eischer 70839155f2 features: add tests 2024-03-09 17:29:52 +01:00
Michael Eischer 1c77c51a03 features: initialize based on RESTIC_FEATURES environment variable 2024-03-09 17:29:52 +01:00
Michael Eischer 5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer 0589da60b3
Merge pull request #4717 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.39.0
build(deps): bump cloud.google.com/go/storage from 1.37.0 to 1.39.0
2024-03-07 14:10:24 +00:00
dependabot[bot] 608116817b
build(deps): bump cloud.google.com/go/storage from 1.37.0 to 1.39.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.37.0 to 1.39.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.37.0...spanner/v1.39.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-07 13:59:23 +00:00
Michael Eischer f742da8b2f
Merge pull request #4722 from konidev20/upgrade-docker-golang-base-image-to-1-22
docker: update the base image to golang:1.22-alpine
2024-03-07 13:44:48 +00:00
Michael Eischer af1684743f
Merge pull request #4716 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.7
build(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7
2024-03-07 13:41:46 +00:00
Michael Eischer 5b9de4d8b7
Merge pull request #4715 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.3.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.2.1 to 1.3.1
2024-03-07 13:40:44 +00:00
Michael Eischer fceef67abe
Merge pull request #4718 from restic/dependabot/go_modules/github.com/spf13/cobra-1.8.0
build(deps): bump github.com/spf13/cobra from 1.7.0 to 1.8.0
2024-03-07 13:37:17 +00:00
Michael Eischer 8506cae710
Merge pull request #4714 from restic/dependabot/github_actions/golangci/golangci-lint-action-4
build(deps): bump golangci/golangci-lint-action from 3 to 4
2024-03-07 13:34:21 +00:00
Michael Eischer 87d47ef189
Merge pull request #4713 from restic/dependabot/github_actions/docker/login-action-5139682d94efc37792e6b54386b5b470a68a4737
build(deps): bump docker/login-action from 3d58c274f17dffee475a5520cbe67f0a882c4dbb to 5139682d94efc37792e6b54386b5b470a68a4737
2024-03-07 13:33:49 +00:00
Michael Eischer 55abf25ea8
Merge pull request #4719 from restic/dependabot/go_modules/golang.org/x/oauth2-0.17.0
build(deps): bump golang.org/x/oauth2 from 0.16.0 to 0.17.0
2024-03-07 13:32:36 +00:00
Srigovind Nayak b48b1fa2c9
docker: update the base image to golang:1.22-alpine 2024-03-03 16:18:13 +05:30
dependabot[bot] 8e7f29ae28
build(deps): bump golang.org/x/oauth2 from 0.16.0 to 0.17.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.16.0 to 0.17.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.16.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:47 +00:00
dependabot[bot] 79e8ddac3f
build(deps): bump github.com/spf13/cobra from 1.7.0 to 1.8.0
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:41 +00:00
dependabot[bot] b5a9b5d0bc
build(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.6 to 1.17.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.6...v1.17.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:25:00 +00:00
dependabot[bot] f185c80cf0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.2.1...sdk/azcore/v1.3.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:24:46 +00:00
dependabot[bot] 70c8aaa303
build(deps): bump golangci/golangci-lint-action from 3 to 4
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3 to 4.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:17:27 +00:00
dependabot[bot] e1a588b75c
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 3d58c274f17dffee475a5520cbe67f0a882c4dbb to 5139682d94efc37792e6b54386b5b470a68a4737.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](3d58c274f1...5139682d94)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 01:17:20 +00:00
Michael Eischer e71660cd1e backup: rename data_added_in_repo statistic to data_added_packed 2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia 062d408987
Clean up SecurityDescriptor helper 2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia 5764300022
Add changelog and fix lint error 2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia c0a1b9ada5
Update docs for security descriptors 2024-02-24 13:28:18 -07:00
Aneesh Nireshwalia 90916f53de
Add test cases for security descriptors 2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia 70cf8e3788
Add support for backup/restore of security descriptors 2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia e3e59fef24
Fix CombineErrors and fillExtendedAttr error handling 2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia 09ce1b4e58
Create helper for SecurityDescriptor related functions 2024-02-24 13:16:25 -07:00
Michael Eischer 6a13e451b1 document snapshot statistics 2024-02-23 22:32:04 +01:00
Michael Eischer a8f5684f68 archiver: test statistics in snapshot 2024-02-23 22:05:15 +01:00
Michael Eischer 681395955e archiver: test backup summary calculation 2024-02-23 21:46:39 +01:00
Michael Eischer b6520038fd snapshots: Print snapshot size stored in snapshots 2024-02-23 20:27:13 +01:00
Michael Eischer 38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer 86897314d5 backup: expose data_added_in_repo in JSON output
The value describes how much data was added after compression.
Previously, it was only available in the text output.
2024-02-23 20:27:13 +01:00
Michael Eischer a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Michael Eischer 8b1a85711f archiver: unexport save/saveDir/saveTree methods 2024-02-23 20:24:21 +01:00
Michael Eischer b953dc8f58
Merge pull request #4611 from zmanda/windows-metadata-support
Back up and restore windows metadata like created ts, file attribs like hidden, readonly, encrypted with a common extensible mechanism
2024-02-23 18:16:09 +00:00
Aneesh Nireshwalia e8211cb64a
Add changelog and update docs for windows attr 2024-02-22 17:59:56 -07:00
Aneesh Nireshwalia 4bbd25a37f
Add tests for generic attribute changes 2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia d4be734c73
Handle readonly empty files in windows 2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia eeb1aa5388
Add ability to report warnings to terminal
Report warnings to terminal when unrecognized generic attributes are found in the repository.
2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia 0962917974
Support windows metadata using generic attribs
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia 62a8a599f1
Add optional messages for Equals helper 2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia 94de87d4b7
Add CombineErrors helper function 2024-02-22 16:57:00 -07:00
Michael Eischer c6311c1e32
Merge pull request #4703 from ferringb/master
Catch SIGTERM, run cleanup
2024-02-22 21:06:29 +00:00
Michael Eischer 0a65a0f94f update comment 2024-02-22 22:00:42 +01:00
Brian Harring b41107dcaf
Add changelog for SIGTERM bugfix.
Signed-off-by: Brian Harring <ferringb@gmail.com>
2024-02-19 11:31:48 +01:00
Brian Harring 30e979d252
Catch SIGTERM, run cleanup
The previous code only ran cleanup (lock release for example) on SIGINT.  For
anyone running restic in a container, the signal is going to be SIGTERM which
means containerized execution would leave locks behind.

While this could be addressed via interposing dumb-init to translate the signal,
a `kill` invocation is going to default to SIGTERM, so the same problem exists
for non container users.

Signed-off-by: Brian Harring <ferringb@gmail.com>
2024-02-19 11:12:15 +01:00
Michael Eischer cfbeb2cde5
Merge pull request #4701 from MichaelEischer/better-streampack-errors
repository: Improve StreamPack error messages
2024-02-18 16:57:56 +01:00
Michael Eischer 80754dbf0c
Merge pull request #4664 from MichaelEischer/ls-unified-json-output
ls: include standard `message_type` field in output
2024-02-18 15:47:41 +00:00
Michael Eischer 4c3218ef9f repository: include packID in StreamPack for decrypt/decompress errors 2024-02-17 19:38:01 +01:00
Michael Eischer 18b0bbbf42 repository: use fmt.Errorf in StreamPacks 2024-02-17 19:37:32 +01:00
Michael Eischer 6fbb470835
Merge pull request #4665 from MichaelEischer/check-repair-packs
check: Suggest usage of `repair packs` if pack files are damaged
2024-02-17 15:54:07 +00:00
Michael Eischer 0a36d193d8 add changelog for enhanced repair packs 2024-02-12 21:43:35 +01:00
Michael Eischer 69304cd74f check: clarify repair pack usage 2024-02-12 21:43:35 +01:00
Michael Eischer c3b0e6d004
Merge pull request #4700 from MichaelEischer/remove-duplicate-changelog-entries
remove changelogs that are already included in restic 0.16.4
2024-02-12 21:40:04 +01:00
Michael Eischer 9e3703ded5 remove changelogs that are already included in restic 0.16.4 2024-02-12 20:39:31 +01:00
Michael Eischer 527a3ff2b2 check: link to troubleshooting guide 2024-02-12 20:25:15 +01:00
Michael Eischer ed4a4f8748 check: exclude inaccessible files from the repair pack suggestion 2024-02-12 20:25:15 +01:00
Michael Eischer 4073299a7c check: fix missing error if blob is invalid 2024-02-12 20:20:13 +01:00
Michael Eischer 6397615fbb check: document that check will show repair pack instructions 2024-02-12 20:20:13 +01:00
Michael Eischer 544fe38786 check: suggest repair pack for all damaged packs 2024-02-12 20:20:13 +01:00
Michael Eischer 772e3416d1 repair pack: drop feature flag 2024-02-12 20:20:12 +01:00
Michael Eischer 22a3cea1b3 checker: wrap all pack errors in ErrPackData 2024-02-12 20:19:32 +01:00
Michael Eischer 19bf2cf52d
Merge pull request #4697 from MichaelEischer/report-blob-errors
backup: report files whose chunks failed to upload
2024-02-12 20:18:51 +01:00
Michael Eischer 5b5d506472 backup: report files whose chunks failed to upload 2024-02-11 22:43:26 +01:00
Michael Eischer dde556e8e8
Merge pull request #4696 from MichaelEischer/fix-exclude-load-error-msg
backup: improve error message if exclude file cannot be loaded
2024-02-11 21:35:31 +00:00
Michael Eischer ee1ff3c1d0 backup: improve error message if exclude file cannot be loaded 2024-02-11 22:26:13 +01:00
Michael Eischer 667a2f5369
Merge pull request #4694 from restic/update-go-versions
Update go versions
2024-02-11 17:10:12 +00:00
Michael Eischer 2ab18a92e6 CI: keep tests for Go 1.19 2024-02-10 23:42:34 +01:00
Alexander Neumann c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Alexander Neumann a8cda0119c Upgrade golangci-lint 2024-02-10 22:08:43 +01:00
Alexander Neumann 9720935c56 Update Go version for tests to 1.22 2024-02-10 21:56:01 +01:00
Michael Eischer 68cc327b15
Merge pull request #4692 from 27149chen/dump-to-existing-file
feat: dump flag --target should be allowed to write existing file
2024-02-10 17:44:54 +00:00
Michael Eischer 15d6fa1f83 dump: update docs for --target option 2024-02-10 18:39:06 +01:00
lou 80db02fc35 dump flag --target should be allowed to write existing file
Signed-off-by: lou <alex1988@outlook.com>
2024-02-10 18:39:06 +01:00
Michael Eischer 6a2b10e2a8
Merge pull request #4685 from konidev20/fix-gh-4676-sub-commands-for-key-management
Move key add, list, remove and passwd as separate sub-commands and improve key sub-command documentation
2024-02-08 19:59:16 +00:00
Michael Eischer e46b21ab80 key: fix integration test for invalid arguments 2024-02-08 20:52:30 +01:00
Michael Eischer eb389a2d25
Merge pull request #4687 from MichaelEischer/upgrade-zstd-library
Upgrade zstd library to latest version
2024-02-08 20:29:13 +01:00
Srigovind Nayak 795d33b3ee key: move add, list, remove, passwd to sub-commands
docs: improve the sub-command docs

changelog: add the unreleased changelog for the key command updates

key: update integration tests
2024-02-06 01:47:43 +05:30
Michael Eischer 0cffdb7493
Merge pull request #4682 from konidev20/feat-add-target-for-dump-command
feat: set --target for dump command
2024-02-05 19:16:42 +00:00
Michael Eischer f5ffa40652 dump: minor cleanups 2024-02-05 20:10:52 +01:00
Srigovind Nayak 175c14b5c9 dump: add --target option 2024-02-05 20:10:52 +01:00
Michael Eischer bca099ac7f Upgrade zstd library to latest version
The data corruption bug is fixed, thus remove the override.
2024-02-05 19:53:02 +01:00
Michael Eischer 0f09a8870c
Merge pull request #4684 from konidev20/fix-gh-4658-update-azure-documentation
Update azure storage account authentication documentation
2024-02-04 20:13:18 +00:00
Srigovind Nayak 5771c4ecfb docs: update environment variables for `az login` to azure backend 2024-02-05 01:32:43 +05:30
Michael Eischer b63bfd2257 Merge branch 'patch-release' 2024-02-04 20:21:42 +01:00
Alexander Neumann 0f9fa44de5 Set development version for 0.16.4 2024-02-04 19:50:56 +01:00
Alexander Neumann 3786536dc1 Add version for 0.16.4 2024-02-04 19:50:52 +01:00
Alexander Neumann 811be5984d Update manpages and auto-completion 2024-02-04 19:50:51 +01:00
Alexander Neumann b0ead75de5 Generate CHANGELOG.md for 0.16.4 2024-02-04 19:50:34 +01:00
Alexander Neumann 6cd2804bff Prepare changelog for 0.16.4 2024-02-04 19:50:34 +01:00
Michael Eischer a72c2b74f3 Apply changelog entry / documentation improvements from review 2024-02-04 19:10:06 +01:00
Michael Eischer 261b1455c7 add documentation for --no-extra-verify option 2024-02-04 19:10:06 +01:00
Michael Eischer 2a0bd2b637 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 19:10:05 +01:00
Michael Eischer 4589da7eb9 add data verification changelog entry 2024-02-04 19:09:49 +01:00
Michael Eischer 75e72d826c pack: verify integrity of pack file header 2024-02-04 19:09:49 +01:00
Michael Eischer d8916bc3d9 repository: ask users to report corrupted data while saving blobs 2024-02-04 19:09:49 +01:00
Michael Eischer dc11d012bb Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 19:09:49 +01:00
Michael Eischer 8ef5425351 repository: test verification of blobs/unpacked data 2024-02-04 19:09:46 +01:00
Michael Eischer 885431ec2b repository: Allow skipping verification for tests
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-04 19:08:30 +01:00
Michael Eischer cb85fb46dd backup: verify unpacked files before upload 2024-02-04 19:07:48 +01:00
Michael Eischer 2f30c940b2 backup: verify blobs before upload
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-04 19:07:48 +01:00
Michael Eischer 0ea62b5ac6 repository: make repo.Options configurable for test repos 2024-02-04 19:07:46 +01:00
Michael Eischer 29e1caf825 add changelog draft for data corruption on max compression 2024-02-04 19:05:51 +01:00
Michael Eischer 0164f5310d Downgrade klauspost/compress to fix data corruption at max. compression 2024-02-04 19:05:50 +01:00
Michael Eischer d5e662315a
Merge pull request #4681 from MichaelEischer/verify-integrity-on-upload
backup: verify blobs before upload
2024-02-04 18:04:27 +00:00
Michael Eischer effe76aaf5
Merge pull request #4679 from MichaelEischer/workaround-compression-bug
Downgrade klauspost/compress to fix data corruption at max. compression
2024-02-04 18:03:34 +00:00
Michael Eischer 5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer 219d8e3c18 add changelog draft for data corruption on max compression 2024-02-04 18:11:48 +01:00
Michael Eischer a737fe1e47 add documentation for --no-extra-verify option 2024-02-04 17:11:49 +01:00
Michael Eischer 86b38a0b17 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 17:01:05 +01:00
Michael Eischer 7d31180fe6 add data verification changelog entry 2024-02-04 15:48:11 +01:00
Michael Eischer c32e5e2abb pack: verify integrity of pack file header 2024-02-04 15:31:42 +01:00
Michael Eischer c97a271e89 repository: ask users to report corrupted data while saving blobs 2024-02-04 15:31:42 +01:00
Michael Eischer 66e8971659 Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 15:31:42 +01:00
Michael Eischer 193140525c repository: test verification of blobs/unpacked data 2024-02-04 15:31:42 +01:00
Michael Eischer 96518d7c4a
Merge pull request #4674 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.37.0
build(deps): bump cloud.google.com/go/storage from 1.34.0 to 1.37.0
2024-02-03 17:23:49 +00:00
Michael Eischer 2dbb18128c repository: Allow skipping verification for tests
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-03 18:22:47 +01:00
Michael Eischer 30a84e9003 backup: verify unpacked files before upload 2024-02-03 18:22:47 +01:00
Michael Eischer c01a0c6da7 backup: verify blobs before upload
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-03 18:22:47 +01:00
Michael Eischer 16e3f79e8b repository: make repo.Options configurable for test repos 2024-02-03 18:22:47 +01:00
Michael Eischer bb92b487f7 repository: fix repack test 2024-02-03 18:22:47 +01:00
dependabot[bot] cf7cad11de
build(deps): bump cloud.google.com/go/storage from 1.34.0 to 1.37.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.34.0 to 1.37.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.34.0...spanner/v1.37.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-02 20:40:57 +00:00
Michael Eischer 370d9c31f4
Merge pull request #4671 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.2.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.2.0 to 1.2.1
2024-02-02 20:31:16 +00:00
Michael Eischer 6581133e85
Merge pull request #4675 from restic/dependabot/go_modules/golang.org/x/oauth2-0.16.0
build(deps): bump golang.org/x/oauth2 from 0.15.0 to 0.16.0
2024-02-02 20:29:39 +00:00
Michael Eischer 207a4a5e8e Downgrade klauspost/compress to fix data corruption at max. compression 2024-02-02 20:10:29 +01:00
Michael Eischer cbf9cd4a7f
Merge pull request #4670 from joram-berger/patch-1
Link to Go Match syntax directly in 040_backup.rst
2024-02-02 18:40:13 +00:00
dependabot[bot] 552f01662b
build(deps): bump golang.org/x/oauth2 from 0.15.0 to 0.16.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.15.0 to 0.16.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 01:53:12 +00:00
dependabot[bot] 7f5ea511bc
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/v1.2...sdk/azidentity/v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 01:51:32 +00:00
Joram Berger b07afa9b02
Link to Go Match syntax directly in 040_backup.rst
The docs linked to filepath.Glob (which is used) but the syntax is described in the documentation of filepath.Match. So it makes sense to link that directly.
2024-01-31 23:40:21 +01:00
Michael Eischer 8b08b522c9
Merge pull request #4669 from MichaelEischer/fix-rewrite-typo
rewrite: fix typo in dry-run output
2024-01-31 22:08:12 +01:00
Michael Eischer eaf9659efc
Merge pull request #4657 from numerigraphe/fix-key-add-id
Properly report the ID of newly added keys
2024-01-31 20:59:41 +00:00
Michael Eischer ba136ff60c rewrite: fix typo in dry-run output 2024-01-31 21:48:37 +01:00
Lionel Sausin 8fbe328371 Properly report the ID of newly added keys
Other commands like key list and key remove show the key's ID.

Showing the ID here lets users easily reuse the ID as a key hint for subsequent
commands.
In particular, a key hint is needed when the repository has many keys - otherwise
opening the repository may fail with "Fatal: maximum number of keys reached" even
when a proper password is provided.

Fixes #4656
2024-01-29 10:12:49 +01:00
Michael Eischer 4273e06a43
Merge pull request #4662 from MichaelEischer/clarify-backup-ignore-inode
backup: clarify that --ignore-inode also ignores ctime
2024-01-27 18:25:11 +01:00
Michael Eischer 248c144f72
Merge pull request #4663 from MichaelEischer/key-subcommand-bugfix
key: return an error if subcommand is unknown
2024-01-27 18:24:50 +01:00
Michael Eischer 5dca8a70d5 ls: include standard `message_type` field in output 2024-01-27 15:48:24 +01:00
Michael Eischer 765729d009 key: return an error if subcommand is unknown 2024-01-27 15:33:49 +01:00
Michael Eischer a09d51d96c backup: clarify that --ignore-inode also ignores ctime 2024-01-27 13:42:29 +01:00
Michael Eischer e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu
Ls ncdu
2024-01-27 12:27:35 +00:00
Michael Eischer 10e71af759 describe ls command in docs 2024-01-27 13:22:00 +01:00
Michael Eischer c90f24a06c
Merge pull request #4641 from MichaelEischer/reduce-restic-repository-usage
Misc cleanups
2024-01-27 13:18:20 +01:00
Michael Eischer d4ed7c8858 walker: add tests for leaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer 2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer 261737abc8 ls: only allow either --json or --ncdu 2024-01-27 13:17:33 +01:00
Michael Eischer a2f2f8fb4c fix linter warning 2024-01-27 13:17:33 +01:00
Michael Eischer 4bae54d040 ls: test ncdu output format 2024-01-27 13:17:33 +01:00
Michael Eischer 509b339d54 ls: correctly handle setuid/setgit/sticky bit in ncdu output 2024-01-27 13:17:33 +01:00
Michael Eischer a2fe337610 ls: unify printer implementations 2024-01-27 13:17:33 +01:00
Michael Eischer 1b008c92d3 ls: rework ncdu output to use walker.LeaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer 9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Nils Decker b2703a4089 add changelog for ls --ncdu 2024-01-27 13:17:15 +01:00
Nils Decker a9310948cf command ls: add option for ncdu output
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories.
It has an option to save a directory tree and analyse it later.
This patch adds an output option to the ls command.

A snapshot can be seen with

`restic ls latest --ncdu | ncdu -f -`

- https://dev.yorhel.nl/ncdu
2024-01-27 13:06:52 +01:00
Michael Eischer 246559e654 check: cleanup s3 legacy detection 2024-01-27 13:02:04 +01:00
Michael Eischer 1dfd854769 lock: switch to repository.List 2024-01-27 13:02:04 +01:00
Michael Eischer bfb56b78e1 replace some usages of restic.Repository with more specific interface
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer 3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer 724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer 42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer 764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer 7c351bc53c repair pack: reenable auto index updates
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer feeab84204 repair pack: extract the repair logic into the repository package
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer d7a50fe739 properly show termstatus progress bar if visible less than one frame
If a progress bar using termstatus was only visible for less than one
frame, then its output could be lost.
2024-01-27 12:51:40 +01:00
Michael Eischer 6b65a495b1 backup/restore: fix termstatus initialization
The termstatus must only be canceled once the command has returned.
Otherwise output may be lost when the context gets canceled.
2024-01-27 12:51:08 +01:00
Michael Eischer d26d2d41f8 backup/restore: extract termstatus initialization 2024-01-27 12:51:08 +01:00
Michael Eischer cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Michael Eischer bedff1ed6d split deleteFiles into UI and logic parts 2024-01-27 12:51:08 +01:00
Michael Eischer c13bf0b607 repository: Introduce RemoveKey function
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer 25ac1549e7
Merge pull request #4661 from MichaelEischer/clarify-contributing
CONTRIBUTING.md: Clarify handling of small bugfixes
2024-01-26 23:04:58 +00:00
Michael Eischer ae9683336d CONTRIBUTING.md: Clarify handling of small bugfixes
Opening an issue for a small bugfix is usually not useful. It primarily
adds overhead.
2024-01-26 23:51:54 +01:00
Michael Eischer 446167ae80
Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask
Remove redundant poly1305 key masking
2024-01-23 19:46:40 +01:00
Michael Eischer 5b36c4eb5f
Merge pull request #4647 from MichaelEischer/reduce-globals
Remove all usages of the global command-specific options
2024-01-23 19:46:15 +01:00
Michael Eischer 1419baf67a
Merge pull request #4645 from MichaelEischer/improve-lock-checking
lock: checkForOtherLocks processes each lock at most once
2024-01-23 19:46:00 +01:00
Michael Eischer 66103aea3d Remove all usages of the global command-specific options
Now, every command uses an options struct, which is passed to the run*
function by the command.RunE method.
2024-01-23 19:21:39 +01:00
Michael Eischer 79f2939eb9
Merge pull request #4654 from adrian5/docfix
docs: fix formatting
2024-01-23 18:08:59 +00:00
Michael Eischer 0e2ee06803
Merge pull request #4650 from MichaelEischer/improve-stdin-from-command-description
backup: Improve help text for `--stdin-from-command`
2024-01-23 19:22:15 +01:00
Michael Eischer 2927982256
Merge pull request #4649 from MichaelEischer/simplify-termstatus-shutdown
ui/termstatus: simplify cleaning up on termination
2024-01-23 19:16:21 +01:00
Michael Eischer 6cc2bec5dd apply suggestion from review 2024-01-23 19:09:04 +01:00
Michael Eischer 18806944f6 doc: remove blockquotes from unordered lists 2024-01-23 19:03:54 +01:00
adrian5 609f84e095 docs: fix formatting 2024-01-22 21:12:12 +01:00
Michael Eischer 767c2539a0 backup: Improve help text for `--stdin-from-command` 2024-01-21 22:06:54 +01:00
Michael Eischer 6bdca13603 ui/termstatus: simplify cleaning up on termination
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
2024-01-21 21:27:27 +01:00
Michael Eischer f1f34eb3e5 lock: checkForOtherLocks processes each lock at most once
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
2024-01-20 22:40:12 +01:00
Michael Eischer fee83e1c09 Remove redundant poly1305 key masking
The implementation in crypto/poly1305 already performs the exact same
masking.
2024-01-20 12:36:59 +01:00
Michael Eischer 6696195f38
Merge pull request #4584 from elkemper/fix-stop-archiving-metadata
S3: Don't archive metadata files on S3 Glacier
2024-01-20 10:30:37 +00:00
Michael Eischer a763a5c67d s3: minor cleanups for archive storage class handling 2024-01-20 11:25:28 +01:00
Vladislav Belous 8ca58b487c S3: do not set storage class for metadata when using archive storage 2024-01-20 11:04:15 +01:00
Michael Eischer 62111f4379
Merge pull request #4625 from MichaelEischer/refactor-streampacks
Refactor repository.StreamPacks
2024-01-19 21:48:37 +01:00
Michael Eischer 2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer 6b7b5c89e9 repository: prepare StreamPack refactor 2024-01-19 21:40:43 +01:00
Michael Eischer 22d0c3f8dc check: Use PackBlobIterator instead of StreamPack
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
2024-01-19 21:40:36 +01:00
Michael Eischer fb422497af repository: split StreamPack implementation
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer 54c5c72e5a
Merge pull request #4616 from MichaelEischer/fix-rest-connection-close
rest: fix and cleanup closing of http response body
2024-01-19 21:31:35 +01:00
Michael Eischer 5f49eec655
Merge pull request #4615 from MichaelEischer/fix-find-empty-dirs
walker: Remove ignoreTrees functionality
2024-01-19 21:25:40 +01:00
Michael Eischer ec13105093
Merge pull request #4623 from MichaelEischer/docs-verify-release-binaries
Add documentation for the verify-release-binaries.sh script
2024-01-19 21:17:38 +01:00
Michael Eischer bd883caae1 CI: enable bodyclose linter 2024-01-19 21:17:18 +01:00
Michael Eischer b1a8fd1d03 rest: fix and cleanup closing of http response body
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.

This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
Michael Eischer fdcbb53017 walker: test skipping for root node 2024-01-19 21:16:06 +01:00
Michael Eischer 0b39940fdb walker: Remove ignoreTrees functionality
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
  unlikely to be important
- find: filtered directories would be ignored. However, this
  optimization missed that it is possible that two directories have the
  exact same content. Such directories would be incorrectly ignored too.
  Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```

Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer 147b0e54cb
Merge pull request #4639 from northben/patch-1
Update Backblaze documentation
2024-01-19 20:08:48 +00:00
Ben Northway 5413877d33 Update Backblaze documentation
clarify documentation regarding B2 bucket lifecycle settings. The default lifecycle setting is probably fine for most users now; a custom policy is not necessary.
2024-01-18 16:41:01 -06:00
Michael Eischer 03e06d0797 Merge branch 'patch-release' 2024-01-14 21:38:17 +01:00
Alexander Neumann 0ec9383ba2 Set development version for 0.16.3 2024-01-14 20:21:45 +01:00
Alexander Neumann abca112404 Add version for 0.16.3 2024-01-14 20:21:45 +01:00
Alexander Neumann b70b94507a Generate CHANGELOG.md for 0.16.3 2024-01-14 20:21:19 +01:00
Alexander Neumann d987582594 Prepare changelog for 0.16.3 2024-01-14 20:21:19 +01:00
Leo R. Lundgren ef2e473b99 doc: Polish changelogs 2024-01-10 00:19:07 +01:00
Michael Eischer 7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer e4bbde7036 rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:28:17 +01:00
Michael Eischer ec0fb46f6c add changelog for reliable restores 2024-01-09 18:27:48 +01:00
Michael Eischer 103beb96bc restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-09 18:27:48 +01:00
Michael Eischer f0f89d7f27 restore: split error reporting from downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer cf352ccafb restore: cleanup downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer b856e9489a restore: split downloadPack into smaller methods 2024-01-09 18:27:48 +01:00
Michael Eischer c31e9418ba
Merge pull request #4626 from MichaelEischer/reliable-large-restores
Improve reliability of large restores
2024-01-09 18:23:09 +01:00
Michael Eischer 2e8de9edfd rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
Michael Eischer ce7db90e08 sync CI and go dependencies with master branch 2024-01-08 21:33:05 +01:00
Michael Eischer 620518aec6 add changelog for better restore error reporting 2024-01-08 21:33:05 +01:00
Michael Eischer f2fafbffaa restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-08 21:33:05 +01:00
Michael Eischer 7a3a884874 repository: test that StreamPack only delivers blobs once 2024-01-08 21:33:05 +01:00
Michael Eischer 772a907533 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-08 21:33:05 +01:00
Michael Eischer a9446c1184 add changelog for irregular files on windows 2024-01-08 21:33:05 +01:00
Michael Eischer 1bab29c336 archiver: Add filepath to error message if it is not included yet 2024-01-08 21:33:05 +01:00
Michael Eischer e886c3f6b2 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-08 21:33:05 +01:00
Michael Eischer c95de54726 restic: cleanup node type determination
os.ModeCharDevice is already included in os.ModeType
2024-01-08 21:33:05 +01:00
Michael Eischer d4b8abd3e2 fix deduplicated files on windows 2024-01-08 21:33:05 +01:00
Joram Berger 948ab3ccaf Add a note that the oldest snapshot may be kept additionally
Documentation enhancement.
2024-01-08 21:33:05 +01:00
Markus Zoppelt bb0c923298 docs: add pkgx install option
PR in pkgx pantry: https://github.com/pkgxdev/pantry/pull/4098

restic pkg:
https://pkgx.dev/pkgs/restic.net/restic/
2024-01-08 21:33:05 +01:00
Michael Eischer ff0c975443 regenerate changelog 2024-01-08 21:33:05 +01:00
Michael Eischer 7e61e117d6 cleanup changelog whitespace 2024-01-08 21:33:05 +01:00
mmattel 220a28582e Add a table of contents (TOC) to the changelog template 2024-01-08 21:33:05 +01:00
Giuseppe D'Andrea f44fd73230 docs: fix typo in working with repos
When using the `copy` command, `--from-password-file` and `--from-password-command` flags are used to specify the password of the source repository, not of the destination repository.
2024-01-08 21:33:05 +01:00
Joda Stößer 76bd975e03 docs(scripting): correct stats output comment to be about the correct command
not about the snapshots command
2024-01-08 21:33:05 +01:00
Quang-Linh LE 64b7aed362 docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991 2024-01-08 21:33:05 +01:00
Michael Kuhn 3fa6b2de4a Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2024-01-08 21:33:05 +01:00
Michael Eischer 5cd000f4b0 CI: update golangci-lint
Necessary to properly support Go 1.21.
2024-01-08 21:33:05 +01:00
Michael Eischer 4ea3796455 add changelog for reliable restores 2024-01-08 21:03:10 +01:00
Michael Eischer e78be75d1e restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer 2267910418 restore: split error reporting from downloadPack 2024-01-08 20:57:00 +01:00
Michael Eischer 00d18b7a88 restore: cleanup downloadPack 2024-01-08 20:53:08 +01:00
Michael Eischer 9328f34d43 restore: split downloadPack into smaller methods 2024-01-08 20:52:36 +01:00
Michael Eischer 77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer 4248c6c3ca s3: update documentation 2024-01-07 19:30:11 +01:00
Michael Eischer e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer 100872308f add changelog for better restore error reporting 2024-01-07 11:06:42 +01:00
Michael Eischer dac3508170 restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer 77b1c52673 repository: test that StreamPack only delivers blobs once 2024-01-07 10:54:53 +01:00
Michael Eischer fe5c337ca2 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer 3e29f8dddf add changelog for irregular files on windows 2024-01-07 10:52:12 +01:00
Michael Eischer 76f507c775
Merge pull request #4621 from MichaelEischer/fix-windows-dedup-files
Fix backup of deduplicated files on windows
2024-01-07 10:44:50 +01:00
Michael Eischer 6ef23b401b fix deduplicated files on windows 2024-01-07 10:23:31 +01:00
Michael Eischer 62f99a3b2f Add documentation for the verify-release-binaries.sh script 2024-01-07 10:16:30 +01:00
Michael Eischer 0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer e6dfefba13 termstatus: update import path of golang.org/x/term 2024-01-06 21:59:26 +01:00
Michael Eischer 02bc73f5eb s3: minor code cleanups 2024-01-06 21:44:53 +01:00
Michael Eischer 20cf4777cb s3: check for EnvAWS credentials before Static credentials
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen 5ffb536aae feat: support AWS assume role 2024-01-06 21:19:58 +01:00
Michael Eischer 1604922360
Merge pull request #4527 from adamantike/cmd/copy/prefix-hostname-to-snapshot-paths
cmd: Add hostname to snapshot display output
2024-01-06 19:32:14 +00:00
Michael Eischer c7844530d8 update docs 2024-01-06 20:25:24 +01:00
Michael Eischer 33b7c84a7a deduplicate string formatting of snapshot metadata
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Manganiello 045aa64558 cmd/copy: Prefix hostname to snapshot display output
This change better resembles the output generated by `Snapshot.String()`,
which includes both username and hostname.

Closes #4506

Before:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
created new cache in /home/mike/.cache/restic
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot 10331fdd saved
```

After:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03 by mike@desktop)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot a67bd1ee saved
```
2024-01-06 20:20:46 +01:00
Michael Eischer b2b7669ca0
Merge pull request #4526 from dnnr/detect-bitrot-in-diff
Add bitrot detection to "diff" command
2024-01-06 19:18:34 +00:00
Michael Eischer 4f6b1bb6f6 diff: document limitations regarding metadata 2024-01-06 20:12:47 +01:00
Michael Eischer 3549635243 diff: copy nodes before modifying them for bitrot detection 2024-01-06 20:12:47 +01:00
Daniel Danner a7dc18e697 Add bitrot detection to "diff" command
This introduces a new modifier to the output of the diff command. It
appears whenever two files being compared only differ in their content
but not in their metadata. As far as we know, under normal
circumstances, this should only ever happen if some kind of bitrot has
happened in the source file. The prerequisite for this detection to work
is that the right-side snapshot of the comparison has been created with
"backup --force".
2024-01-06 20:12:47 +01:00
Michael Eischer 51419c51d3 archiver: Add filepath to error message if it is not included yet 2024-01-06 19:08:24 +01:00
Michael Eischer 6b79834cc8 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-06 19:03:11 +01:00
Michael Eischer 0018bb7854 restic: cleanup node type determination
os.ModeCharDevice is already included in os.ModeType
2024-01-06 18:43:16 +01:00
Michael Eischer 634e2a46d9
Merge pull request #4608 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.66
build(deps): bump github.com/minio/minio-go/v7 from 7.0.63 to 7.0.66
2024-01-06 11:40:02 +00:00
Michael Eischer dfcab92db2
Merge pull request #4609 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.9.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.8.0 to 1.9.1
2024-01-06 11:06:07 +00:00
dependabot[bot] 3666eef76c
build(deps): bump github.com/minio/minio-go/v7 from 7.0.63 to 7.0.66
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.63 to 7.0.66.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.63...v7.0.66)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-06 11:03:52 +00:00
Michael Eischer 3a61622dfe
Merge pull request #4610 from restic/dependabot/github_actions/actions/setup-go-5
build(deps): bump actions/setup-go from 4 to 5
2024-01-06 11:01:07 +00:00
Michael Eischer 5c4fca76df
Merge pull request #4607 from restic/dependabot/go_modules/golang.org/x/crypto-0.17.0
build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0
2024-01-06 10:56:40 +00:00
Michael Eischer 98da0bdd12
Merge pull request #4606 from restic/dependabot/go_modules/github.com/klauspost/compress-1.17.4
build(deps): bump github.com/klauspost/compress from 1.17.2 to 1.17.4
2024-01-06 10:55:00 +00:00
dependabot[bot] 2c60dd97ae
build(deps): bump actions/setup-go from 4 to 5
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:33:20 +00:00
dependabot[bot] 40905403f4
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.9.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azcore/v1.9.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:18:36 +00:00
dependabot[bot] 7e7cbe8e19
build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.16.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.16.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:18:08 +00:00
dependabot[bot] 44646c20be
build(deps): bump github.com/klauspost/compress from 1.17.2 to 1.17.4
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.2 to 1.17.4.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.2...v1.17.4)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-01 01:17:57 +00:00
Michael Eischer 8f9a35779e
Merge pull request #4600 from MichaelEischer/doc-forget-note-1
Add a note that the oldest snapshot may be kept additionally
2023-12-29 17:56:54 +01:00
Michael Eischer 23e1b4bbb1
Merge pull request #4573 from gab50000/rewrite_time
Rewrite metadata
2023-12-24 14:42:00 +00:00
Michael Eischer 01b33734ab rewrite: update command output in docs 2023-12-24 15:36:22 +01:00
Michael Eischer 649a6409ee rewrite: cleanup tests 2023-12-24 15:36:22 +01:00
Michael Eischer c31f5f986c rewrite: Minor cleanups 2023-12-24 15:36:22 +01:00
Michael Eischer 2730d05fce rewrite: Don't walk snapshot content if only metadata is modified 2023-12-24 15:36:22 +01:00
Michael Eischer 893d0d6325 rewrite: cleanup new metadata options and fix no parameters check 2023-12-24 15:36:22 +01:00
Gabriel Kabbe 7de97d7480 rewrite: Add documentation 2023-12-24 15:36:22 +01:00
Gabriel Kabbe 004520a238 rewrite: Add changelog 2023-12-24 15:36:22 +01:00
Gabriel Kabbe a02d8d75c2 rewrite: Implement rewriting metadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe 7bf38b6c50 rewrite: Add test TestRewriteMetadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe da1704b2d5 rewrite: Add tests
Pass nil instead of metadata to existing tests
2023-12-24 15:36:19 +01:00
Gabriel Kabbe 3026baea07 rewrite: Add structs for tracking metadata changes
Adds

  * snapshotMetadataArgs, which holds the new metadata as strings parsed from
    the command line

  * snapshotMetadata, which holds the new metadata converted to the
    correct types
2023-12-24 14:43:07 +01:00
Michael Eischer 1196c72819
Merge pull request #4570 from MarkusZoppelt/docs/pkgx
docs: add pkgx install option
2023-12-24 11:04:59 +00:00
Michael Eischer 433dd92959
Merge pull request #4590 from renard/optimize-mount-failure
mount: detect mountpoint does not exist before opening the repository
2023-12-24 10:59:56 +00:00
Markus Zoppelt c14740c50f docs: add pkgx install option
PR in pkgx pantry: https://github.com/pkgxdev/pantry/pull/4098

restic pkg:
https://pkgx.dev/pkgs/restic.net/restic/
2023-12-24 11:59:12 +01:00
Michael Eischer 5537460664
Merge pull request #4598 from MichaelEischer/add_table_of_contents
Add table of contents
2023-12-24 11:56:05 +01:00
Sébastien Gross f7587be28f mount: detect mountpoint does not exist before opening the repository
Bug #1681 suggests that restic should not be nice to user and should
refrain from creating a mountpoint if it does not exist. Nevertheless,
it currently opens the repository before checking for the mountpoint's
existence. In the case of large or remote repositories, this process
can be time-consuming, delaying the inevitable outcome.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    repository 33f14e42 opened (version 2, compression level max)
    [0:38] 100.00%  162 / 162 index files loaded
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m39.534s
    user	1m53.961s
    sys	0m3.044s

In this scenario, 40 seconds could have been saved if the nonexistence
of the path had been verified beforehand.

This patch relocates the mountpoint check to the beginning of the
runMount function, preceding the opening of the repository.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m0.136s
    user	0m0.018s
    sys	0m0.027s

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2023-12-24 11:54:18 +01:00
Michael Eischer 91fb703756 regenerate changelog 2023-12-24 11:47:31 +01:00
Michael Eischer d7ff862b8d cleanup changelog whitespace 2023-12-24 11:47:31 +01:00
mmattel db1d920c80 Add a table of contents (TOC) to the changelog template 2023-12-24 11:47:31 +01:00
Michael Eischer c6299f8dbd
Merge pull request #4582 from giuseppedandrea/docs/fix-typo
docs: fix typo in working with repos
2023-12-24 00:09:59 +00:00
Giuseppe D'Andrea a128976014 docs: fix typo in working with repos
When using the `copy` command, `--from-password-file` and `--from-password-command` flags are used to specify the password of the source repository, not of the destination repository.
2023-12-24 01:04:36 +01:00
Michael Eischer e2f6109a52
Merge pull request #4580 from restic/dependabot/go_modules/golang.org/x/oauth2-0.15.0
build(deps): bump golang.org/x/oauth2 from 0.13.0 to 0.15.0
2023-12-24 00:02:30 +00:00
Michael Eischer 30e6ed038c
Merge pull request #4586 from Gelma/typos
Fix typos
2023-12-23 13:21:50 +00:00
Michael Eischer e96d1ee33e
Merge pull request #4593 from SimJoSt/patch-1
docs(scripting): correct stats output comment to be about the correct command
2023-12-23 13:09:20 +00:00
dependabot[bot] 0054db394f
build(deps): bump golang.org/x/oauth2 from 0.13.0 to 0.15.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.13.0 to 0.15.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.13.0...v0.15.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-23 13:08:29 +00:00
Michael Eischer 53ebe91a50 Move changelog typo fixes to changelog files 2023-12-23 14:05:47 +01:00
Michael Eischer 9ceaea34dd
Merge pull request #4572 from linktohack/patch-1
docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991
2023-12-23 13:00:32 +00:00
Michael Eischer 356b7aac16
Merge pull request #4571 from michaelkuhn/init-repo-file
Fix repository not being printed when using repository file
2023-12-23 12:59:06 +00:00
Joda Stößer eef7c65655 docs(scripting): correct stats output comment to be about the correct command
not about the snapshots command
2023-12-23 13:56:40 +01:00
Michael Eischer 97b8629336
Merge pull request #4579 from restic/dependabot/go_modules/golang.org/x/sync-0.5.0
build(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0
2023-12-23 12:51:52 +00:00
Michael Eischer d2ecd6bef2
Merge pull request #4577 from restic/dependabot/go_modules/golang.org/x/time-0.5.0
build(deps): bump golang.org/x/time from 0.3.0 to 0.5.0
2023-12-23 12:50:59 +00:00
Michael Eischer 634750a732
Merge pull request #4576 from restic/dependabot/github_actions/docker/login-action-3d58c274f17dffee475a5520cbe67f0a882c4dbb
build(deps): bump docker/login-action from 1f401f745bf57e30b3a2800ad308a87d2ebdf14b to 3d58c274f17dffee475a5520cbe67f0a882c4dbb
2023-12-23 12:50:52 +00:00
Quang-Linh LE c554825e2d docs: Mention progress for restore command. This is available after https://github.com/restic/restic/pull/3991 2023-12-23 13:50:20 +01:00
Michael Kuhn fd2fb233aa Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2023-12-23 13:49:22 +01:00
Michael Eischer da4e3edbbc
Merge pull request #4596 from MichaelEischer/update-golangcilint
CI: update golangci-lint
2023-12-23 13:48:33 +01:00
Michael Eischer dbbd31bc3a CI: update golangci-lint
Necessary to properly support Go 1.21.
2023-12-23 13:41:30 +01:00
Joram Berger 12af20e606
Add a note that the oldest snapshot may be kept additionally
Documentation enhancement.
2023-12-18 18:24:57 +01:00
Andrea Gelmini 241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
dependabot[bot] 427b90cf82
build(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.4.0 to 0.5.0.
- [Commits](https://github.com/golang/sync/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:15:09 +00:00
dependabot[bot] eec6e014f4
build(deps): bump golang.org/x/time from 0.3.0 to 0.5.0
Bumps [golang.org/x/time](https://github.com/golang/time) from 0.3.0 to 0.5.0.
- [Commits](https://github.com/golang/time/compare/v0.3.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/time
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:14:53 +00:00
dependabot[bot] fa46a47e22
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 1f401f745bf57e30b3a2800ad308a87d2ebdf14b to 3d58c274f17dffee475a5520cbe67f0a882c4dbb.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](1f401f745b...3d58c274f1)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-01 01:02:16 +00:00
Michael Eischer b72de5a883 Merge branch 'patch-release' 2023-11-12 11:52:21 +01:00
Michael Eischer 6086ae4ca7
Merge pull request #4563 from smlx/mac-signature
fix: drop reference to signature and define MAC
2023-11-08 21:59:13 +00:00
Scott Leggett aeaf527be1
fix: drop reference to signature and define MAC
Poly1305-AES is not a signature, so don't mention that.

In addition, the term MAC was used without being defined, so add a
definition.

Signed-off-by: Scott Leggett <scott@sl.id.au>
2023-11-06 20:12:42 +08:00
Michael Eischer 19068aa82f
Merge pull request #4559 from DRON-666/html-zip
Restore generation of `HTMLZip` files on `readthedocs.com`.
2023-11-02 20:53:12 +00:00
Michael Eischer 81ca9d28f2
Merge pull request #4553 from CommanderRoot/add-version-json
Add --json option to version command
2023-11-01 21:58:38 +00:00
Tobias Speicher ce53ea32c6
Split `go_target` into `go_os` and `go_arch` 2023-11-01 22:43:38 +01:00
Tobias Speicher 10cbc169c1
Use different function to be more consistent with other code 2023-11-01 22:18:37 +01:00
DRON-666 03f8f494e9 doc: add HTMLZip format to .readthedocs.yaml 2023-11-02 00:16:47 +03:00
Tobias Speicher ab23d033b6
Add version command output to JSON format documentation 2023-11-01 22:13:57 +01:00
Tobias Speicher 6f1efcb28b
Update wording on changelog entry 2023-11-01 22:12:19 +01:00
Michael Eischer 9c399e55e3
Merge pull request #4554 from restic/dependabot/github_actions/docker/login-action-1f401f745bf57e30b3a2800ad308a87d2ebdf14b
build(deps): bump docker/login-action from b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6 to 1f401f745bf57e30b3a2800ad308a87d2ebdf14b
2023-11-01 20:42:11 +00:00
Michael Eischer e550bc0713
Merge pull request #4555 from restic/dependabot/go_modules/google.golang.org/api-0.149.0
build(deps): bump google.golang.org/api from 0.148.0 to 0.149.0
2023-11-01 20:40:45 +00:00
Michael Eischer 28aa9826af
Merge pull request #4557 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.34.0
build(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.0
2023-11-01 20:40:40 +00:00
dependabot[bot] 6dde019ac8
build(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.33.0 to 1.34.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.33.0...spanner/v1.34.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:45:02 +00:00
dependabot[bot] c2f9e21d3c
build(deps): bump google.golang.org/api from 0.148.0 to 0.149.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.148.0 to 0.149.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.148.0...v0.149.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:44:28 +00:00
dependabot[bot] 67e6b9104a
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6 to 1f401f745bf57e30b3a2800ad308a87d2ebdf14b.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](b4bedf8053...1f401f745b)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 01:25:22 +00:00
Tobias Speicher 6ca07ee004
add changelog/unreleases for issue-4547 2023-10-31 19:39:52 +01:00
Tobias Speicher d45cc52468
command version: add json output option 2023-10-31 19:31:07 +01:00
Alexander Neumann 59fe24cb2b Set development version for 0.16.2 2023-10-29 19:50:12 +01:00
Alexander Neumann 1a5efcf680 Add version for 0.16.2 2023-10-29 19:50:07 +01:00
Alexander Neumann d33fe6dd3c Generate CHANGELOG.md for 0.16.2 2023-10-29 19:50:04 +01:00
Alexander Neumann c8dd95f104 Prepare changelog for 0.16.2 2023-10-29 19:50:04 +01:00
Leo R. Lundgren 7d980b469d doc: Polish changelogs 2023-10-29 00:44:53 +02:00
Michael Eischer d863234e3e add changelog for missing documentation 2023-10-28 20:25:24 +02:00
Michael Eischer 42ab3ea2b9
Merge pull request #4410 from Enrico204/restic-stdin-command
add --stdin-from-command flag to backup command
2023-10-27 22:09:06 +00:00
Michael Eischer be28a02626 doc: tweak description for --stdin-from-command 2023-10-27 23:58:52 +02:00
Michael Eischer 5d152c7720 extend changelog for --stdin-from-command 2023-10-27 23:58:52 +02:00
Michael Eischer ee305e6041 backup: rework error reporting for subcommand 2023-10-27 23:58:52 +02:00
Michael Eischer 8bceb8e359 fs: add tests for CommandReader 2023-10-27 23:58:52 +02:00
Michael Eischer 317144c1d6 fs: merge command startup into CommandReader 2023-10-27 23:58:51 +02:00
Michael Eischer 7d879705ad fs: cleanup CommandReader implementation 2023-10-27 23:58:51 +02:00
Enrico204 37a312e505 restic-from-command: use standard behavior when no output and exit code 0 from command
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204 c0ca54dc8a restic-from-command: add tests 2023-10-27 23:58:51 +02:00
Enrico204 81f8d473df restic-from-command: abort snapshot on non-zero exit codes 2023-10-27 23:58:51 +02:00
Enrico204 6990b0122e Add issue-4251 (stdin-from-command) in the changelog 2023-10-27 23:58:51 +02:00
Enrico204 072b227544 stdin-from-command: add documentation in backup sub-command 2023-10-27 23:58:51 +02:00
Enrico204 4e5caab114 stdin-from-command: implemented suggestions in #4254
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß c133065a9f Check command result before snapshotting
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß 25350a9c55 Extend SnapshotOptions w/ command data
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß a2b76ff34f Start command from --stdin-from-command
It acts similar to --stdin but reads its data from the stdout of the given command instead of os.Stdin.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß 333fe1c3cf Align Stdin and StdinCommand in conditionals
In order to run with --stdin-from-command we need to short-circuit some functions similar to how it is handled for the --stdin flag. The only difference here is that --stdin-from-command actually expects that len(args) should be greater 0 whereas --stdin does not expect any args at all.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß a8657bde68 Add --stdin-from-command option
This new flag is added to the backup subcommand in order to allow restic to control the execution of a command and determine whether to save a snapshot if the given command succeeds.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Michael Eischer 104107886a
Merge pull request #4503 from MichaelEischer/fix-stats-with-hardlinks
stats: Fix hardlink tracking across multiple filesystems
2023-10-27 23:53:03 +02:00
Michael Eischer 731b3a4357 stats: fix hardlink tracking in a snapshot
inodes are only unique within a device. Use the HardlinkIndex from the
restorer instead of the custom (broken) hashmap to correctly account for
both inode and deviceID.
2023-10-27 23:40:42 +02:00
Michael Eischer a8fdcf79b7 restorer: Make hardlink index generic
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
2023-10-27 23:40:42 +02:00
Michael Eischer 45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Michael Eischer 4be45de1c2 Restore support for ARMv5 platforms 2023-10-27 19:52:14 +02:00
Leo R. Lundgren 8c1125fe13 doc: Correct two typos 2023-10-27 19:48:00 +02:00
Martin Michlmayr 0b6ccea461 Fix typos 2023-10-27 19:48:00 +02:00
Martin Michlmayr de6135351e Format option correctly 2023-10-27 19:48:00 +02:00
Michael Eischer d47581b25e verify-release-binaries.sh: don't show warning if binaries are correct 2023-10-27 19:48:00 +02:00
Michael Eischer 69dec02a14 Remove readthedocs special case from docs configuration
Apparently it's now required to bring your own theme.
2023-10-27 19:47:19 +02:00
Michael Eischer 826d880614 Fix doc path typo in readthedocs configuration 2023-10-27 19:47:15 +02:00
Michael Eischer dbf7ef72b9 Add read the docs config file version 2
The config file is by now necessary to build documentation:
https://blog.readthedocs.com/migrate-configuration-v2/
2023-10-27 19:47:11 +02:00
Michael Eischer 50ef01131a
Merge pull request #4542 from MichaelEischer/support-armv6
Only support ARMv6 on ARM platforms
2023-10-27 19:42:49 +02:00
rawtaz 6be3a8fe51
Merge pull request #4539 from tbm/docs
Fix typos in docs
2023-10-27 17:30:08 +00:00
Michael Eischer 5166bde386 Only support ARMv6 on ARM platforms
Go 1.21 has switched the default from GOARM=5 to GOARM=7. So far there
have been complaints from Raspberry Pi 1 users, as the first raspberry
pi version only supports ARMv6. Exclude older ARM versions as these are
likely not relevant (rest-server also only supports ARMv6/7) and enforce
the use of software floating point emulation.
2023-10-27 19:12:12 +02:00
Leo R. Lundgren aafb806a8c doc: Correct two typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr 41e6a02bcc Fix typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr b51fe2fb69 Format option correctly 2023-10-27 18:56:32 +02:00
Michael Eischer 56537fb48e
Merge pull request #4545 from restic/fix-rtd3
Try to fix documentation build
2023-10-26 22:00:20 +02:00
Michael Eischer feea567868 Remove readthedocs special case from docs configuration
Apparently it's now required to bring your own theme.
2023-10-26 21:56:36 +02:00
Michael Eischer 2968b52f84
Merge pull request #4543 from MichaelEischer/fix-rtd2
Fix doc path typo in readthedocs configuration
2023-10-26 20:40:45 +02:00
Michael Eischer 619e80d7cc Fix doc path typo in readthedocs configuration 2023-10-26 20:39:43 +02:00
Michael Eischer 3804e50d64
Merge pull request #4541 from MichaelEischer/fix-binary-check-script
verify-release-binaries.sh: don't show warning if binaries are correct
2023-10-26 20:37:56 +02:00
Michael Eischer c19e39968f verify-release-binaries.sh: don't show warning if binaries are correct 2023-10-26 19:59:27 +02:00
Michael Eischer 550be5c1e9
Merge pull request #4538 from MichaelEischer/fix-rtd
Add read the docs config file version 2
2023-10-26 19:42:36 +02:00
Michael Eischer 249605843b prune: get backend connection count via repository 2023-10-25 23:01:54 +02:00
Michael Eischer c7b770eb1f convert MemorizeList to be repository based
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer 1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer ceb0774af1 backend: make LoadAll independent of restic package 2023-10-25 22:58:39 +02:00
Michael Eischer b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer 7881309d63 backend: move backend implementation helpers to util package
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
Michael Eischer 8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer c2e3e8d6ea Add read the docs config file version 2
The config file is by now necessary to build documentation:
https://blog.readthedocs.com/migrate-configuration-v2/
2023-10-25 22:00:42 +02:00
Alexander Neumann 27ec320eae Set development version for 0.16.1 2023-10-24 20:02:58 +02:00
Alexander Neumann baca3f6303 Add version for 0.16.1 2023-10-24 20:02:55 +02:00
Alexander Neumann 524c2721b4 Update manpages and auto-completion 2023-10-24 20:02:55 +02:00
Alexander Neumann be1b978ac8 Generate CHANGELOG.md for 0.16.1 2023-10-24 20:02:29 +02:00
Alexander Neumann d4d9c1b8f1 Prepare changelog for 0.16.1 2023-10-24 20:02:29 +02:00
Michael Eischer ead8dd0173
Merge pull request #4524 from MichaelEischer/fix-autocompletion
Don't load password when autocompleting CLI commands
2023-10-23 20:06:53 +00:00
Michael Eischer cd09ef4614 don't load password when autocompleting CLI commands 2023-10-23 21:44:22 +02:00
Michael Eischer d399e32590
Merge pull request #4535 from MichaelEischer/fix-cat-command
Only show index loading progress bar if stdout is a terminal
2023-10-23 21:28:13 +02:00
Michael Eischer 54a4034ec0 Only show index progress bar if stdout is a terminal
This ensures that there is no interference when using restic for
scripting use cases.
2023-10-23 21:16:52 +02:00
rawtaz 138b7b3328
Merge pull request #4534 from MichaelEischer/polish-changelogs
Polish changelogs
2023-10-23 19:00:01 +00:00
Leo R. Lundgren 6d19e0260d doc: Polish changelogs 2023-10-23 20:45:15 +02:00
Michael Eischer 85abceb99c doc: update minimum required Go version 2023-10-23 19:02:20 +02:00
Michael Eischer 85c15e6fa3 polish changelogs 2023-10-23 19:01:45 +02:00
Michael Eischer d6917c7e00
Merge pull request #4533 from MichaelEischer/update-dependencies
update dependencies
2023-10-23 19:01:03 +02:00
Michael Eischer 8c20301172 update dependencies 2023-10-23 18:48:13 +02:00
Michael Eischer 4b4f916bdc
Merge pull request #4532 from MichaelEischer/update-zstd
Update klauspost/compress to fix data corruption at maximum compression
2023-10-23 18:45:35 +02:00
Michael Eischer 9707956375
Merge pull request #4530 from MichaelEischer/repair-packs
Add `repair packs` command to simplify recovery from damaged pack
2023-10-23 18:44:42 +02:00
Michael Eischer d1d4510974 repair packs: Add stub feature flag implementation 2023-10-23 18:36:28 +02:00
Michael Eischer a28940ea29 check: Suggest usage of `restic repair packs` for corrupted blobs
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Michael Eischer db26dc75e1 repair packs: add experimental command
This allows recovering a repository from several cases of damaged blobs.
2023-10-23 18:36:28 +02:00
Michael Eischer efef38d0e8 drop support for Go 1.18 2023-10-23 18:30:38 +02:00
Michael Eischer d00e72fed4 Update klauspost/compress to fix data corruption at maximum compression 2023-10-23 18:30:38 +02:00
Arash Farr d15ffd9c92 retry: Do not retry Load() if file does not exist 2023-10-22 13:25:32 -05:00
Michael Eischer 62af0d769a
Merge pull request #4528 from adamantike/misc/run-tests-with-go-1.21
Run tests with Go 1.21
2023-10-22 16:00:27 +00:00
Michael Eischer ae83a9002a
Merge pull request #4531 from MichaelEischer/silence-automaxprocs
Disable log output from automaxprocs library
2023-10-22 17:43:36 +02:00
Michael Eischer ceff4af1ac Disable log output from automaxprocs library 2023-10-22 17:09:46 +02:00
Michael Manganiello b15ba553a4 Run tests with Go 1.21 2023-10-22 10:47:42 -03:00
Michael Eischer 46cb1df1bc
Merge pull request #4505 from MichaelEischer/verify-release-binaries
Add script to verify release binaries
2023-10-21 20:00:25 +02:00
Michael Eischer bd3816fa14 CI: Ensure that github containers match the official binaries
The binaries accidentally included VCS information whereas binaries
built from the release tarball do not. For consistency remove the .git
directory before building the container on Github.
2023-10-21 19:44:59 +02:00
Michael Eischer b2b0856908 Add helper script to verify release binaries
The script checks that the released binaries and the container binaries
can be reproduced.
2023-10-21 19:44:59 +02:00
Michael Eischer 7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer eabc177a42
Merge pull request #4442 from Refutable4890/copy-doc
Add a description of the resumption of `copy`
2023-10-21 17:38:15 +00:00
Michael Eischer ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer fe1f61570b
Merge pull request #4518 from adamantike/backend/b2/switch-to-official-library
backend/b2: Switch to official library `Backblaze/blazer`
2023-10-21 17:29:19 +00:00
Michael Eischer baf9b54891 sftp: simplify documentation for sftp.args 2023-10-21 19:28:27 +02:00
Michael Eischer 6a4d6d5da4 sftp: test that Args and Command options cannot be set at the same time 2023-10-21 19:26:39 +02:00
Michael Manganiello 41f70f1f4f backend/sftp: Add sftp.args option
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.

Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer 6c7560e537 backend/b2: Switch to official library `Backblaze/blazer`
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.

Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace 0f97356b21
docs: add new REST server env variables 2023-10-20 13:04:58 +02:00
Agatha V. Lovelace 2089c54310
changelog: add unreleased entry 2023-10-20 13:04:58 +02:00
Agatha V. Lovelace f1877e721e
feat: support reading REST credentials from env 2023-10-20 13:04:58 +02:00
Michael Eischer 17f2301cc2
Merge pull request #4514 from grembo/key-list-no-lock
Make `key list` command honor `--no-lock`
2023-10-19 20:50:56 +00:00
Michael Eischer 79deb99605
Merge pull request #4500 from MichaelEischer/simplify-issue-template
Simplify issue template
2023-10-19 20:48:36 +00:00
Michael Eischer 643180b415 simplify issue template 2023-10-19 22:16:38 +02:00
Michael Eischer d27cfd10a9
Merge pull request #4487 from lbausch/tweak-restore-summary
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Gmelin 34f3b13b7c Make `key list` command honor `--no-lock`
Fixes #4513
2023-10-19 22:09:35 +02:00
Michael Eischer 495982232c
Merge pull request #4511 from dnnr/inode-in-json
Include inode in JSON output for find and ls
2023-10-17 21:06:34 +00:00
Michael Eischer d173d1c532
Merge pull request #4486 from MichaelEischer/separate-large-blobs
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer f955222750
Merge pull request #4502 from MichaelEischer/replace-ratelimit
backend/limiter: replace juju/ratelimit with x/time/rate
2023-10-17 23:03:17 +02:00
Michael Eischer cb9cbe55d9 repository: store oversized blobs in separate pack files
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer f750aa8dfb backend/limiter: replace juju/ratelimit with x/time/rate 2023-10-17 22:49:25 +02:00
Michael Eischer c635e30e3f
Merge pull request #4485 from MichaelEischer/uber-automaxprocs
Automatically adjust GOMAXPROCS when run in a cpu-constrained container
2023-10-17 22:47:10 +02:00
Michael Eischer f5d5e8fd0a automatically adjust GOMAXPROCS when run in a cpu-constrained container 2023-10-17 20:31:57 +02:00
Daniel Danner a858ab254b Include inode in JSON output for find and ls
This introduces the inode attribute to the JSON output emitted for nodes
in `ls` and matches in `find`. There doesn't seem to be any discernible
reason to omit the inode and it can be useful in scripting scenarios.
2023-10-11 21:10:01 +02:00
Michael Eischer 4087c3aff7
Merge pull request #4498 from PeterDaveHello/ImproveDockerfile
Remove additional `--update` for apk in Dockerfile
2023-10-09 20:53:49 +00:00
Refutable4890 aa86c76aab
Add a description of the resumption of `copy` 2023-10-08 12:07:40 +08:00
Michael Eischer 76ef94d15c
Merge pull request #4419 from arjunajesh/issue#299
Progress Bar when loading index files
2023-10-01 18:06:35 +00:00
Michael Eischer 91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer b0da0f152f deduplicate index progress bar setup 2023-10-01 19:53:26 +02:00
Michael Eischer 75f6bd89ed hide index loading progress for JSON output 2023-10-01 19:53:26 +02:00
Michael Eischer 3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh b6593ad7df changelog 2023-10-01 19:53:26 +02:00
arjunajesh ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer eac1c4a8d0
Merge pull request #4471 from jalseth/master
docs(examples): Use restic binary from system path
2023-10-01 12:31:28 +00:00
Michael Eischer f519454f33
Merge pull request #4493 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.33.0
build(deps): bump cloud.google.com/go/storage from 1.31.0 to 1.33.0
2023-10-01 12:09:43 +00:00
Michael Eischer 54ae8a0c40
Merge pull request #4492 from restic/dependabot/github_actions/docker/login-action-b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6
build(deps): bump docker/login-action from cf8514a65188af1d4f94f8c28a7a4153af1088ce to b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6
2023-10-01 12:08:10 +00:00
Michael Eischer a36b5b6391
Merge pull request #4491 from restic/dependabot/github_actions/docker/setup-buildx-action-3
build(deps): bump docker/setup-buildx-action from 2 to 3
2023-10-01 12:03:48 +00:00
Michael Eischer 5e36e4da96
Merge pull request #4489 from restic/dependabot/github_actions/docker/setup-qemu-action-3
build(deps): bump docker/setup-qemu-action from 2 to 3
2023-10-01 12:03:19 +00:00
Michael Eischer 8ee08e5d09
Merge pull request #4490 from restic/dependabot/github_actions/actions/checkout-4
build(deps): bump actions/checkout from 3 to 4
2023-10-01 12:03:09 +00:00
Michael Eischer 9f9f736ec2
Merge pull request #4495 from restic/dependabot/go_modules/golang.org/x/oauth2-0.12.0
build(deps): bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
2023-10-01 12:01:12 +00:00
Michael Eischer af98c3ccbe
Merge pull request #4496 from restic/dependabot/go_modules/github.com/hashicorp/golang-lru/v2-2.0.7
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7
2023-10-01 11:59:38 +00:00
Peter Dave Hello 6edfc73879
Remove additional `--update` for apk in Dockerfile 2023-10-01 18:00:15 +08:00
dependabot[bot] d985ed27d1
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7
Bumps [github.com/hashicorp/golang-lru/v2](https://github.com/hashicorp/golang-lru) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/hashicorp/golang-lru/releases)
- [Commits](https://github.com/hashicorp/golang-lru/compare/v2.0.6...v2.0.7)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/golang-lru/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:45:19 +00:00
dependabot[bot] 4278ec6553
build(deps): bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:45:07 +00:00
dependabot[bot] 15cb498c47
build(deps): bump cloud.google.com/go/storage from 1.31.0 to 1.33.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.31.0 to 1.33.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.31.0...pubsub/v1.33.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:44:54 +00:00
dependabot[bot] 59e217b003
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from cf8514a65188af1d4f94f8c28a7a4153af1088ce to b4bedf8053341df3b5a9f9e0f2cf4e79e27360c6.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](cf8514a651...b4bedf8053)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:15 +00:00
dependabot[bot] 3457f50c8c
build(deps): bump docker/setup-buildx-action from 2 to 3
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:10 +00:00
dependabot[bot] bbe2f1ecf2
build(deps): bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:05 +00:00
dependabot[bot] 362917afb9
build(deps): bump docker/setup-qemu-action from 2 to 3
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-01 01:24:01 +00:00
Michael Eischer b92ab458b0
Merge pull request #4459 from restic/dependabot/go_modules/github.com/go-ole/go-ole-1.3.0
build(deps): bump github.com/go-ole/go-ole from 1.2.6 to 1.3.0
2023-09-28 20:42:14 +00:00
Lorenz Bausch 2657217574
Tweak restore summary 2023-09-20 10:59:55 +02:00
James Alseth 02ab511c2f
docs(examples): Use restic binary from system path
Signed-off-by: James Alseth <james@jalseth.me>
2023-09-17 19:39:45 -07:00
Michael Eischer 6e586b64e4
Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
Michael Eischer fb4d458cce
Merge pull request #4452 from lbausch/docs/add-current-key-hint
docs: explain meaning of asterisk when managing repository keys
2023-09-06 20:28:33 +00:00
Michael Eischer c7b5ddc012
Merge pull request #4451 from greatroar/bh-string
internal/restic: Return summary from BlobSet.String
2023-09-06 20:27:26 +00:00
Michael Eischer 3eb825e47c
Merge pull request #4464 from restic/dependabot/go_modules/golang.org/x/term-0.12.0
build(deps): bump golang.org/x/term from 0.11.0 to 0.12.0
2023-09-06 20:18:34 +00:00
dependabot[bot] 4d60011030
build(deps): bump golang.org/x/term from 0.11.0 to 0.12.0
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.11.0 to 0.12.0.
- [Commits](https://github.com/golang/term/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/term
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 21:43:28 +00:00
Michael Eischer 507ed32469
Merge pull request #4457 from restic/dependabot/go_modules/google.golang.org/api-0.138.0
build(deps): bump google.golang.org/api from 0.134.0 to 0.138.0
2023-09-05 21:32:34 +00:00
dependabot[bot] 22fdfe1ffe
build(deps): bump github.com/go-ole/go-ole from 1.2.6 to 1.3.0
Bumps [github.com/go-ole/go-ole](https://github.com/go-ole/go-ole) from 1.2.6 to 1.3.0.
- [Release notes](https://github.com/go-ole/go-ole/releases)
- [Changelog](https://github.com/go-ole/go-ole/blob/master/ChangeLog.md)
- [Commits](https://github.com/go-ole/go-ole/compare/v1.2.6...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/go-ole/go-ole
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 21:17:03 +00:00
Michael Eischer e05cd9abca
Merge pull request #4462 from restic/dependabot/github_actions/docker/login-action-cf8514a65188af1d4f94f8c28a7a4153af1088ce
build(deps): bump docker/login-action from a9794064588be971151ec5e7144cb535bcb56e36 to cf8514a65188af1d4f94f8c28a7a4153af1088ce
2023-09-05 21:07:57 +00:00
Michael Eischer ea55ca5303
Merge pull request #4458 from restic/dependabot/go_modules/github.com/hashicorp/golang-lru/v2-2.0.6
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.4 to 2.0.6
2023-09-05 21:05:45 +00:00
Michael Eischer df53f4782b
Merge pull request #4453 from lbausch/docs/fix-table
docs: fix malformed table
2023-09-05 20:37:30 +00:00
Michael Eischer 35055adbc4
Merge pull request #4454 from lbausch/tweak-readme
README: add Mastodon link
2023-09-05 20:36:32 +00:00
dependabot[bot] bd9eb528c0
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from a9794064588be971151ec5e7144cb535bcb56e36 to cf8514a65188af1d4f94f8c28a7a4153af1088ce.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](a979406458...cf8514a651)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:45:03 +00:00
dependabot[bot] 02032f3109
build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.4 to 2.0.6
Bumps [github.com/hashicorp/golang-lru/v2](https://github.com/hashicorp/golang-lru) from 2.0.4 to 2.0.6.
- [Release notes](https://github.com/hashicorp/golang-lru/releases)
- [Commits](https://github.com/hashicorp/golang-lru/compare/v2.0.4...v2.0.6)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/golang-lru/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:41:13 +00:00
dependabot[bot] cfff1367c1
build(deps): bump google.golang.org/api from 0.134.0 to 0.138.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.134.0 to 0.138.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.134.0...v0.138.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-01 01:40:56 +00:00
Lorenz Bausch 353265a329
README: add Mastodon link 2023-08-27 10:52:12 +02:00
Lorenz Bausch 84a1170dee
docs: remove trailing whitespace 2023-08-27 10:49:34 +02:00
Lorenz Bausch 0b4d9c9a51
docs: add missing whitespace to avoid malformed table warning 2023-08-27 10:46:55 +02:00
Lorenz Bausch 5422a7daa5
docs: explain meaning of asterisk when managing repository keys 2023-08-27 10:40:55 +02:00
greatroar 691c01963b internal/restic: Return summary from BlobSet.String
Fixes #4449: error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer 2bec99dc6f master_index: fix inconsistent length blob length in test
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer e60c5b2d7f
Merge pull request #4436 from MichaelEischer/add-snapshot-subfolder-syntax-to-help-text
Add snapshot subfolder syntax in help output of diff/dump/restore
2023-08-13 19:36:38 +02:00
Michael Eischer a04964bb86
Merge pull request #4446 from 2000yeshu/feat-cat
Check for arguments before opening remote connection in cat command
2023-08-13 17:25:37 +00:00
Michael Eischer fe54912a46 cat: extract parameter validation and add a test 2023-08-13 19:17:46 +02:00
Michael Eischer feb6abb7bb
Merge pull request #4394 from greatroar/acl
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar aaf5254e26 dump: Rewrite Linux ACL handling
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Yakul Garg 705556f134
Check for arguments before opening remote connection in cat command
Signed-off-by: Yakul Garg <2000yeshu@gmail.com>
2023-08-12 23:54:40 +05:30
Michael Eischer c23eebc225
Merge pull request #4428 from MichaelEischer/fix-restore-help
restore: Fix help text for `--iinclude` and `--iexclude`
2023-08-12 20:21:22 +02:00
Refutable4890 e09f6f540f
Add a description of the resumption of `copy` 2023-08-08 12:51:40 +08:00
Michael Eischer 0fca028491 restore: Fix help text for `--iinclude` and `--iexclude` 2023-08-05 18:21:03 +02:00
Michael Eischer 57a08291f5
Merge pull request #4429 from restic/dependabot/github_actions/docker/login-action-a9794064588be971151ec5e7144cb535bcb56e36
build(deps): bump docker/login-action from 65b78e6e13532edd9afa3aa52ac7964289d1a9c1 to a9794064588be971151ec5e7144cb535bcb56e36
2023-08-05 13:41:08 +00:00
Michael Eischer 2117ce4cfb
Merge pull request #4434 from lbausch/docs-fix-wording
docs: fix `percent_done` description
2023-08-05 13:39:02 +00:00
Michael Eischer 3a478bc522 unify snapshotID spelling in command parameters 2023-08-05 12:35:45 +02:00
Michael Eischer 9a7704fa2b diff/dump/restore: describe snapshot:subfolder syntax in help text 2023-08-05 12:35:08 +02:00
Lorenz Bausch 9a69f44de2
docs: fix `percent_done` description 2023-08-04 16:15:18 +02:00
dependabot[bot] 7a6339180b
build(deps): bump docker/login-action
Bumps [docker/login-action](https://github.com/docker/login-action) from 65b78e6e13532edd9afa3aa52ac7964289d1a9c1 to a9794064588be971151ec5e7144cb535bcb56e36.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](65b78e6e13...a979406458)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-01 01:19:51 +00:00
Alexander Neumann 82e6e28781 Set development version for 0.16.0 2023-07-31 20:24:29 +02:00
Alexander Neumann eb33e564c9 Add version for 0.16.0 2023-07-31 20:23:24 +02:00
Alexander Neumann 62680af734 Update manpages and auto-completion 2023-07-31 20:23:24 +02:00
Alexander Neumann 68460fd3d1 Generate CHANGELOG.md for 0.16.0 2023-07-31 20:22:50 +02:00
Alexander Neumann 9459328d8d Prepare changelog for 0.16.0 2023-07-31 20:22:50 +02:00
rawtaz 9d71990c26
Merge pull request #4424 from MichaelEischer/tweak-changelog
Tweak changelog entries
2023-07-29 11:29:30 +00:00
rawtaz 9219b6a0ef
Merge pull request #4427 from MichaelEischer/changelog-template
Extend style guide in changelog template
2023-07-29 11:15:17 +00:00
Michael Eischer 9a0ec05145 Extend style guide in changelog template 2023-07-29 12:55:56 +02:00
Michael Eischer 2a5b6af2e9 extend gc tweak changelog 2023-07-29 12:51:05 +02:00
Leo R. Lundgren 51e4e5ef82 doc: Polish changelog 2023-07-29 12:51:04 +02:00
Michael Eischer e998314088 tweak changelog entries 2023-07-29 12:50:01 +02:00
Michael Eischer be03c1a457
Merge pull request #4422 from MichaelEischer/document-non-utf-symlinks
Document `linktarget_raw` field for non utf symlinks
2023-07-28 23:14:08 +02:00
Michael Eischer 95050117eb
Merge pull request #4426 from MichaelEischer/consistent-forget-policy
forget: replace `--keep-* -1` with `--keep-* unlimited`
2023-07-28 19:06:00 +00:00
Michael Eischer 21a3a41b69
Merge pull request #4423 from MichaelEischer/update-dependencies
Update dependencies
2023-07-28 19:25:02 +02:00
Michael Eischer 3f919f2371 forget: simplify test 2023-07-28 19:21:40 +02:00
Michael Eischer 50b43fbac0 forget: replace `--keep-* -1` with `--keep-* unlimited`
This ensures consistency with the `prune --max-unused unlimited` option.
2023-07-28 19:21:40 +02:00
Michael Eischer f689e2638d reenable aix builds 2023-07-27 21:27:31 +02:00
Michael Eischer f9ef2b1e44 update all dependencies 2023-07-27 21:26:44 +02:00
Michael Eischer b87a37f318 doc: Fix typos 2023-07-25 20:14:56 +02:00
Michael Eischer 647ebf352a doc: Describe linktarget_raw field 2023-07-25 20:14:47 +02:00
Michael Eischer 62d3ef4a93
Merge pull request #4417 from MichaelEischer/fix-flaky-list-cancel-test
backend: Fix unreliable TestListCancel/Timeout test
2023-07-23 18:04:11 +02:00
Michael Eischer 6c5fc32967
Merge pull request #3225 from wojas/restic-cmd-more-help
Extend help output with short overview
2023-07-23 15:43:03 +00:00
Michael Eischer ae13cf15c1 doc: move quickstart to docs 2023-07-23 17:32:37 +02:00
Michael Eischer 1fe1ec40a2
Merge pull request #4409 from arjunajesh/issue#1495
snapshots sorted by timestamp in restic find
2023-07-23 11:00:41 +00:00
arjunajesh 32f5ee6f4e snapshots sorted by timestamp 2023-07-23 12:49:23 +02:00
Konrad Wojas 5bd8a6d7eb Extend help output with short overview
The list of subcommands can be confusing for first time users. This
adds a short overview of the most important commands to help get people
started.
2023-07-23 12:01:40 +02:00
Michael Eischer 4a33370072 backend: Fix unreliable TestListCancel/Timeout test
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.

The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer 98fb56baa6
Merge pull request #3802 from MichaelEischer/support-non-utf-symlinks
Support non utf symlink targets
2023-07-23 00:15:01 +02:00
Michael Eischer f12bbd9229 restic: check that Node.LinkTarget can handle non-utf8 characters 2023-07-23 00:05:14 +02:00
Michael Eischer 6adb629608 Add support for non-utf8 symlink targets 2023-07-23 00:05:14 +02:00
Michael Eischer 25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer bdaec8fdb8
Merge pull request #4374 from MichaelEischer/try-refresh-stale-locks
Try to refresh stale locks
2023-07-22 23:50:52 +02:00
Michael Eischer 55b440b520 tweak snapshot:subfolder docs 2023-07-22 23:50:17 +02:00
Michael Eischer 3f63b53090 lock: Shrink critical section in RefreshStaleLock
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer 76253b2a20 Hopefully fix flaky TestLockWaitCancel test 2023-07-22 23:23:17 +02:00
Michael Eischer 20e82d1fcf lock: add changelog for stale lock refresh 2023-07-22 23:23:17 +02:00
Michael Eischer 96eada3d5f backend/sema: add test for freeze functionality 2023-07-22 23:23:17 +02:00
Michael Eischer b8f4267a36 lock: add more debug information to flaky test 2023-07-22 23:23:17 +02:00
Michael Eischer 1f6883a05c lock: test stale lock refresh 2023-07-22 23:23:17 +02:00
Michael Eischer 8154f6a77a Ensure consistent naming for `<snapshot>:<subfolder>` syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 184b7616ba document snapshot:subfolder syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 67f237b4f3 add changelog for snapshot:path syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer 321cc35cde restic: add test for FindTreeDirectory 2023-07-22 19:55:57 +02:00
Michael Eischer a97915642c restic: Ensure snapshots created by TestCreateSnapshot are valid
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00
Michael Eischer c64d81063e test snapshot filtering with snapshot:path syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 233b841ad9 Add cat tree snapshot:subpath command
This command prints the tree for the given path.
2023-07-22 19:55:57 +02:00
Michael Eischer 85860e6e97 Add support for snapshot:subpath syntax
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
2023-07-22 19:55:57 +02:00
Michael Eischer 2dd6769429 lock: Fix possible deadlock during refresh of stale lock
A delayed lock refresh could send a signal on the `refreshed` channel
while the `monitorLockRefresh` goroutine waits for a reply to its
`refreshLockRequest`. As the channels are unbuffered, this resulted in a
deadlock.
2023-07-22 19:55:44 +02:00
Michael Eischer 5d9b0d894e lock: add unit test for RefreshStaleLock 2023-07-22 19:55:44 +02:00
Michael Eischer d4bf7a3cb1 restic: cleanup lock test 2023-07-22 19:55:44 +02:00
Michael Eischer 24c8a33da9 restic: reduce sleeps in lock tests 2023-07-22 19:55:44 +02:00
Michael Eischer 399f8e84a1 lock: use mem backend for locking tests
This hopefully fixes the timing related test failures which appear to be
caused by very slow fsync calls.
2023-07-22 19:55:44 +02:00
Michael Eischer 6a436d731d lock: rename test helper function 2023-07-22 19:55:44 +02:00
Michael Eischer b2fcbc21cb lock: rework stale lock refresh to avoid data race 2023-07-22 19:55:44 +02:00
Michael Eischer 05e5e29a8c backup: don't pretend to lock repo for dry run
The dry run did not actually create lock files, but still ran all other
parts of the lock processing.
2023-07-22 19:55:44 +02:00
Michael Eischer f490288738 lock: freeze backend operations while refreshing stale lock
Freeze new backend operations while trying to refresh a stale lock.
2023-07-22 19:55:44 +02:00
Michael Eischer 51718ec561 restic: generic implemention of AsBackend 2023-07-22 19:55:44 +02:00
Michael Eischer 11eb88a2ea lock: cleanup 2023-07-22 19:55:44 +02:00
Michael Eischer bee3231ed4 lock: try refreshing of stale locks
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
2023-07-22 19:55:44 +02:00
Michael Eischer 60d8066568
Merge pull request #4407 from MichaelEischer/simpler-required-ci
CI: Add check that verifies the results of other checks
2023-07-22 19:55:06 +02:00
Michael Eischer 08dea911bd CI: Add check that verifies the results of other checks
To properly use merge queues, it is necessary to mark certain checks as
"required" in the branch protection rules. However, this doesn't scale
with matrix jobs and would require manual tweaks of the CI jobs change.
The new analyze job simplifies this by allowing the branch protection
rule just check for that job. All further dependencies are then
configured within the CI config.
2023-07-16 14:13:41 +02:00
Michael Eischer 47206a6579
Merge pull request #4351 from kenny-y-dev/human-readable-ls
Add --human-readable to ls and find output
2023-07-16 11:44:41 +00:00
Michael Eischer 594689db32
Merge pull request #4400 from MichaelEischer/rest-handle-missing-dirs
rest: Ignore missing directories
2023-07-14 23:29:37 +02:00
Michael Eischer 5705326bb8 add changelog 2023-07-14 22:39:38 +02:00
Michael Eischer 978ebaac49 rest: use http status code constants 2023-07-14 22:39:38 +02:00
Michael Eischer 307aeb6849 rest: Don't return error if listing non-existent directory
When transferring a repository from S3 to, for example, a local disk
then all empty folders will be missing.

When saving files, the missing intermediate folders are created
automatically. Therefore, missing directories can be ignored by the
`List()` operation.
2023-07-14 22:39:38 +02:00
Michael Eischer 9cd85d5956
Merge pull request #4029 from ekarlso/add-azure-workload-identity-support
add azure workload identity support
2023-07-09 19:06:59 +00:00
Michael Eischer c9f506925c azure: Document additional auth options 2023-07-09 20:56:21 +02:00
Michael Eischer e990d3d483 azure: Support authentication using managed / workload identity
See https://github.com/Azure/azure-sdk-for-go/tree/sdk/azidentity/v1.3.0/sdk/azidentity
2023-07-09 14:47:03 +02:00
Michael Eischer 7042190807
Merge pull request #4403 from MichaelEischer/fix-certificate-env-handling
Fix certificate env handling
2023-07-09 14:46:38 +02:00
Michael Eischer c158741e2e CI: add minimal CLI test
Just create a repository and run a minimal backup.
2023-07-09 14:29:43 +02:00
Michael Eischer 89fbd39e59 Don't print stacktrace on invalid cacert option 2023-07-09 14:23:54 +02:00
Michael Eischer 1ce599d2ae Fix handling of empty cacert environment variable
This resulted in a "empty filename for root certificate supplied" error.
2023-07-09 14:23:54 +02:00
Michael Eischer 789fec3da7
Merge pull request #4402 from MichaelEischer/merge-queue-setup
CI: Enable missing CI tests for Github merge queue
2023-07-09 13:57:10 +02:00
Michael Eischer 8a120c8800 CI: Enable missing CI tests for Github merge queue 2023-07-08 23:51:39 +02:00
Michael Eischer 41395e83c5
Merge pull request #2740 from metalsp0rk/document-json
Add JSON documentation
2023-07-08 21:34:02 +00:00
Michael Eischer 2a193195b0
Merge pull request #4364 from MichaelEischer/multiplatform-docker
Setup multiplatform docker builds
2023-07-08 23:31:58 +02:00
Michael Eischer 229c7b24a4 doc: add program_version field of snapshot JSON output 2023-07-08 23:31:20 +02:00
Michael Eischer b34ce57dd4 doc: describe JSON output of restore command 2023-07-08 23:31:20 +02:00
Michael Eischer 1ce839228e doc: Expand JSON documentation 2023-07-08 23:31:20 +02:00
Kyle Brennan fb1170c1d6 doc tweaks 2023-07-08 23:21:41 +02:00
Kyle Brennan e457fe22bc Documentation improvement 2023-07-08 23:21:41 +02:00
Kyle Brennan 39299e36ef add json documentation 2023-07-08 23:21:41 +02:00
Michael Eischer 0512f292e8
Merge pull request #4361 from MichaelEischer/stats-repo-debug-info
stats: Add `debug` mode to collect repository statistics
2023-07-08 23:20:23 +02:00
Michael Eischer 3a93e28605 CI: Remove .dockerignore to ensure reproducible builds
Since go 1.18, built binaries also include VCS information such as the
built commit. This information is also included in the official
binaries. To ensure that the Docker container recreates the same
binaries, the .git folder must also be transferred into the container.
Thus, remove the .dockerignore file.

The copied files must also be owned by the current user within the
container, as git refuses to work otherwise.
2023-07-08 23:16:14 +02:00
Michael Eischer 6b82cce1bd add changelog for multiplatform containers 2023-07-08 23:16:14 +02:00
Michael Eischer 31e07cecbb docker: update to Go 1.20 for custom container builds 2023-07-08 23:05:39 +02:00
Michael Eischer c181b51360 doc: Update docker build process 2023-07-08 23:05:38 +02:00
Michael Eischer ccd19b7e88 CI: run cloud backend tests only on restic/restic 2023-07-08 23:05:02 +02:00
Michael Eischer b0987ff570 CI: only build containers on restic/restic 2023-07-08 23:05:02 +02:00
Michael Eischer eff3124f15 CI: Setup automatic container builds for ghcr.io
Containers are built for new tags and pushes to the master branch.
2023-07-08 23:05:02 +02:00
Michael Eischer 43fa051546 Directly build restic binary in release Docker container 2023-07-08 23:05:02 +02:00
Michael Eischer 2293835242 Release multi-platform docker containers 2023-07-08 23:05:02 +02:00
Michael Eischer 0fcb6c7f94 ask for debug statistics in resource usage issues 2023-07-08 23:04:47 +02:00
Michael Eischer 325fa916b5 stats: Add debug mode to collect repository statistics 2023-07-08 23:04:47 +02:00
Michael Eischer 170e495334
Merge pull request #4401 from MichaelEischer/fix-windows-build
fuse: fix windows build
2023-07-08 23:01:22 +02:00
Michael Eischer 9d44682e3e fuse: fix windows build 2023-07-08 22:40:22 +02:00
Michael Eischer 4d43509423
Merge pull request #4379 from chenxiaolong/symlink_xattrs
Add support for extended attributes on symlinks
2023-07-08 16:56:51 +00:00
Michael Eischer ea9ad77e05 restic: refactor node test 2023-07-08 18:54:56 +02:00
Michael Eischer cc84884d2e restic: basic xattr test for files/dirs 2023-07-08 18:49:21 +02:00
Michael Eischer 4a5ae2ba84 restic: test NodeFromFileInfo for symlinks 2023-07-08 18:18:13 +02:00
Michael Eischer 1f1e50f49e fuse: add test for symlink xattr 2023-07-08 18:02:17 +02:00
Michael Eischer f3c3b0f377 fuse: deduplicate xattr code 2023-07-08 17:41:45 +02:00
Michael Eischer 7e2be9e081
Merge pull request #4384 from arjunajesh/issue#1926
certificates can be passed through env vars
2023-07-08 07:46:31 +00:00
Michael Eischer cc3c218baf small cleanups for certificate environment variables 2023-07-08 09:44:59 +02:00
arjunajesh c1578a2035 certificates can be passed through env vars 2023-07-08 09:44:59 +02:00
Michael Eischer 678b983300
Merge pull request #4360 from Refutable4890/master
Add an description of `check` temporary cache directory location
2023-07-07 21:51:52 +00:00
Michael Eischer 1b3870dc43
Merge pull request #4378 from MichaelEischer/add-version-to-snapshot
Add program version to snapshot
2023-07-07 23:50:44 +02:00
Michael Eischer ef40aee2bd
Merge pull request #4373 from MichaelEischer/performance-issue-help
document how to analyze performance / memory usage issues
2023-07-07 23:50:27 +02:00
Michael Eischer 3a32c4e59f document how to analyze performance / memory usage issues 2023-07-07 23:38:53 +02:00
Michael Eischer e703e89e9b add changelog for program version 2023-07-07 23:36:57 +02:00
Michael Eischer 389f6ee74c backup: add minimal test for program versioni 2023-07-07 23:33:23 +02:00
Michael Eischer bbac74b172 add program version to snapshot 2023-07-07 23:27:10 +02:00
Michael Eischer a280b7364e
Merge pull request #4395 from greatroar/parsebytes
cmd, ui: Move size parsing code and make it more robust
2023-07-07 21:24:47 +00:00
Michael Eischer 825651a135
Merge pull request #4387 from arjunajesh/issue#2468
azure domain can be overridden with env variable
2023-07-07 21:17:01 +00:00
Michael Eischer e36d17a6f8 azure: tweak documentation for endpoint suffix 2023-07-07 23:10:26 +02:00
arjunajesh 068b115abc added azure domain parameter 2023-07-07 23:10:26 +02:00
greatroar 41a5bf357f cmd, ui: Move size parsing code and make it more robust 2023-07-02 21:44:50 +02:00
Michael Eischer f96896a9c0
Merge pull request #4383 from LouisMT/add-docker-hostname
Add hostname flag to Docker docs
2023-07-01 18:58:43 +02:00
dependabot[bot] 2ab9a3b9c3
Merge pull request #4389 from restic/dependabot/go_modules/google.golang.org/api-0.129.0 2023-07-01 16:57:18 +00:00
Louis Matthijssen dbe2eef80c Add hostname flag to Docker docs
Fixes #4380
2023-07-01 18:46:31 +02:00
dependabot[bot] 6e7c6674ad
build(deps): bump google.golang.org/api from 0.116.0 to 0.129.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.116.0 to 0.129.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.116.0...v0.129.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 16:11:44 +00:00
Michael Eischer 7fe830ee1e
Merge pull request #4392 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.6.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.6.0 to 1.6.1
2023-07-01 18:10:47 +02:00
Michael Eischer a07c7166ba
Merge pull request #4390 from restic/dependabot/go_modules/golang.org/x/oauth2-0.9.0
build(deps): bump golang.org/x/oauth2 from 0.8.0 to 0.9.0
2023-07-01 18:10:38 +02:00
dependabot[bot] d2f8f9de23
build(deps): bump golang.org/x/oauth2 from 0.8.0 to 0.9.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.8.0 to 0.9.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 15:32:55 +00:00
dependabot[bot] d5fe5107c8
Merge pull request #4391 from restic/dependabot/go_modules/golang.org/x/sys-0.9.0 2023-07-01 15:31:42 +00:00
dependabot[bot] f08ba1a005
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 01:23:19 +00:00
dependabot[bot] 70fb554854
build(deps): bump golang.org/x/sys from 0.8.0 to 0.9.0
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.8.0 to 0.9.0.
- [Commits](https://github.com/golang/sys/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-01 01:23:05 +00:00
Andrew Gunnerson 8c02ebb029
Add support for extended attributes on symlinks
Linux allows the use of non-`user.` extended attributes on symlinks. One
of the main users of this functionality is SELinux's `security.selinux`
xattr for storing a path's label. By storing symlink xattrs, restic is
now suitable for backing up the root filesystem on Linux distributions
that use SELinux.

This commit adds support for symlink xattrs when backing up data,
restoring data, and mounting snapshots via a fuse mount. All calls to
the xattr library have been updated to the use `L` variants of the
various functions, which always operate on the path given, without
following symlinks.

Fixes: #4375

Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2023-06-19 14:37:31 -04:00
Michael Eischer 8dcb0c4a9d doc: improve description of caching behavior of the check command 2023-06-17 22:18:50 +02:00
Michael Eischer 74ca82a6f8
Merge pull request #4362 from restic/unified-backend-factory
Unified backend initialization
2023-06-17 15:52:37 +02:00
Michael Eischer 9cbc2502c6
Merge pull request #4366 from gautammenghani/master
Issue #3624: Preserve oldest snapshot when keep-* values are not sati…
2023-06-17 15:41:17 +02:00
Michael Eischer 93038ed8f4 s3: restore retries for minio tests 2023-06-17 15:25:08 +02:00
Gautam Menghani 8da5a6649b Preserve oldest snapshot when keep-within* does not collect enough 2023-06-17 15:17:20 +02:00
Michael Eischer 3888c21a27 reword changelog 2023-06-17 15:16:58 +02:00
Michael Eischer 1257c2c075 forget: Add comments to snapshot policy 2023-06-17 15:16:58 +02:00
Gautam Menghani 182b9796e4 Issue #3624: Preserve oldest snapshot when keep-* values are not satisfied 2023-06-17 15:16:58 +02:00
Michael Eischer cbf87fbdb3 init: don't include password in debug log 2023-06-17 15:15:58 +02:00
Michael Eischer b5511e8e4c Fix linter warnings 2023-06-17 15:15:58 +02:00
Michael Eischer 50e0d5e6b5 backend: Hardcode backend scheme in Factory
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
2023-06-17 15:15:58 +02:00
Michael Eischer 705ad51bcc backend: check that StripPassword can be called 2023-06-17 15:15:58 +02:00
Michael Eischer 13a8b5822f backend: Adjust tests to use the Factory to instantiate the backend
This drastically reduces the amount of duplicated test code.
2023-06-17 15:15:58 +02:00
Michael Eischer 3d3bb88745 b2: remove duplicate check for config file during repository creation
No other backend implements that check. The check that a repository is
not yet initialized is handled by the Repository later on.
2023-06-17 15:15:58 +02:00
Michael Eischer 3a3cf608f5 b2/s3: Move config validation from ApplyEnvironment to Open/Create
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
2023-06-17 15:15:58 +02:00
Michael Eischer 19ac12d95b location: make genericBackendFactory private 2023-06-17 15:15:58 +02:00
Michael Eischer 3325a7c862 location: extract backend specific part of StripPassword
The tests for the rest backend now reside there.
2023-06-17 15:15:58 +02:00
Michael Eischer 555be49a79 location: Make ParseConfig-test backend agnostic
The backend specific parts of the test are now directly handled by the
respective backend. Duplicate tests were removed.
2023-06-17 15:15:58 +02:00
Michael Eischer 9aa9e0d1ec local/sftp: move limiter setup into backend 2023-06-17 15:15:58 +02:00
Michael Eischer 7d12c29286 backend: Unify backend construction using factory and registry
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
2023-06-17 15:15:57 +02:00
Michael Eischer 56836364a4 backend: pass context into every backend constructor 2023-06-17 15:15:57 +02:00
Michael Eischer 4df77e9f26
Merge pull request #4314 from MichaelEischer/restore-json
Add JSON support for the restore progress bar
2023-06-16 23:53:17 +02:00
Michael Eischer 2545c84321
Merge pull request #4365 from MichaelEischer/ci-rework-build-tests
CI: rework build tests
2023-06-16 23:29:39 +02:00
Michael Eischer 191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc
Tune Go garbage collector
2023-06-16 23:24:39 +02:00
Michael Eischer dd1ef13c1c
Merge pull request #4352 from MichaelEischer/pointerless-index
index: optimize go GC performance
2023-06-16 23:23:14 +02:00
Michael Eischer 4d5ee987a7 add changelog about missing AIX builds 2023-06-16 23:19:36 +02:00
Michael Eischer b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer 61042a77a4 building on aix is currently not possible 2023-06-09 13:35:35 +02:00
Michael Eischer 4e9e2c3229 CI: Use build-release-binaries to run the cross-compilation tests
gox silently ignored linux/mips and aix/ppc64. This change also removes
the duplicate platform list.
2023-06-09 13:27:57 +02:00
Michael Eischer faec0ff816 build-release-binaries: support building a subset of all platforms 2023-06-09 13:27:33 +02:00
Michael Eischer 07d1f8047e restore: More descriptive field names for the JSON output 2023-06-09 12:07:55 +02:00
Michael Eischer b2b0760eb0 restore: add test for json output 2023-06-09 12:03:33 +02:00
Michael Eischer cf16239058 restore: print output via termStatus 2023-06-09 12:03:33 +02:00
Michael Eischer 1531eab746 mention restore json support in changelog 2023-06-09 12:03:33 +02:00
Michael Eischer d54176ce5d restore: add basic json progress 2023-06-09 12:03:33 +02:00
Michael Eischer a9aff885d6 restore: reorganize progress bar code
The structure is now much more similar to that of the backup command.
2023-06-09 12:03:33 +02:00
Michael Eischer bb20078641 restore: pass termStatus to restore in tests 2023-06-09 12:03:33 +02:00
Michael Eischer 237f32c651
Merge pull request #3261 from DRON-666/prealloc-packs
Prevent local backend file fragmentation by file preallocation.
2023-06-09 11:50:46 +02:00
DRON-666 74e4656850 Update changelog 2023-06-09 11:49:06 +03:00
DRON-666 c37d587f81 Use PreallocateFile in local backend 2023-06-09 11:49:06 +03:00
DRON-666 ffc6b3d887 Move PreallocateFile to fs package 2023-06-09 11:49:06 +03:00
Michael Eischer 88c63a029c
Merge pull request #3563 from matta/improve-eta
Improve the ETA displayed during backup
2023-06-08 20:49:19 +02:00
Michael Eischer 0b908bb1fb Address review comments 2023-06-08 20:24:21 +02:00
Matt Armstrong 0372c7ef04 Improve the ETA displayed during backup
The ETA restic displays was based on a rate computed across the entire
backup operation. Often restic can progress at uneven rates. In the worst
case, restic progresses over most of the backup at a very high rate and
then finds new data to back up. The displayed ETA is then unrealistic and
never adapts.

Restic now estimates the transfer rate based on a sliding window, with the
goal of adapting to observed changes in rate. To avoid wild changes in the
estimate, several heuristics are used to keep the sliding window wide
enough to be relatively stable.
2023-06-08 20:05:56 +02:00
Michael Eischer 9464c63550 Make formatNode test timezone independent
formatNode formats the timestamp according to the current time zone. Pin
the local timezone to UTC to ensure the test works everywhere.
2023-06-08 19:18:30 +02:00
Michael Eischer 6ebf2dd235 Reduce duplicate code in test for fomatNode 2023-06-08 19:16:16 +02:00
Michael Eischer 5f153109ba Refactor formatNode 2023-06-08 19:12:49 +02:00
Michael Eischer 2beaa74892 tweak changelog 2023-06-08 19:12:26 +02:00
Michael Eischer 55c21846b1 Revert "index: remove redundant storage of indexmap size"
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
2023-06-08 18:08:46 +02:00
Michael Eischer 0f80b6a137 add changelog for gc tuning 2023-06-08 18:02:46 +02:00
Michael Eischer e14ccb1142
Merge pull request #4298 from restic/backend-parseconfig-cleanup
Unified and slightly type-safer backend config parsing
2023-06-08 12:02:27 +02:00
Michael Eischer 609367195a backend: Fix test compilation with Go 1.18 and 1.19 2023-06-07 22:31:15 +02:00
Michael Eischer 18eb1d3ab0 backend: test cleanup 2023-06-07 22:31:15 +02:00
Michael Eischer 32a6b66267 backend: add standardized Config.ApplyEnvironment
This removes the backend specific special cases while parsing the
configuration in `global.go`.
2023-06-07 22:31:15 +02:00
Michael Eischer f903db492c backend: let ParseConfig return a Config pointer
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
2023-06-07 22:31:15 +02:00
Michael Eischer 25a0be7f26 backend: fix linter warnings 2023-06-07 22:31:15 +02:00
Michael Eischer a27b7f1370 backend: use generic instead of any type for test suite 2023-06-07 22:31:14 +02:00
Michael Eischer fa361dbfbd backend: use generic implementation for ParseConfig tests 2023-06-07 22:30:35 +02:00
Michael Eischer 5260d38980 backend: let ParseConfig return concrete type 2023-06-07 22:30:35 +02:00
Michael Eischer 2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
Refutable4890 aea7538936
Add an description of `check` temporary cache directory location 2023-06-07 22:35:33 +08:00
Michael Eischer 49a6a4f5bf
Merge pull request #4356 from MichaelEischer/fix-unlock-hint
lock: fix missing hint to unlock command if repository is locked
2023-06-06 21:03:06 +02:00
Michael Eischer 237d00000e lock: fix missing hint to unlock command if repository is locked 2023-06-02 23:16:49 +02:00
dependabot[bot] 0c727f6ad1
Merge pull request #4355 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.56 2023-06-02 20:51:26 +00:00
Michael Eischer 4e7d3efad4
Merge pull request #4345 from DanielGibson/patch-1
restic restore docs: Mention that restic restore is fastest
2023-06-02 22:44:38 +02:00
Michael Eischer 17446da5fd
Merge pull request #4343 from greatroar/cache
cache: Restructure New to remove redundant operations
2023-06-02 22:41:17 +02:00
dependabot[bot] a3cee840d2
build(deps): bump github.com/minio/minio-go/v7 from 7.0.52 to 7.0.56
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.52 to 7.0.56.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.52...v7.0.56)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:35:42 +00:00
dependabot[bot] c76f3a1e27
Merge pull request #4348 from restic/dependabot/go_modules/github.com/spf13/cobra-1.7.0 2023-06-02 20:34:38 +00:00
dependabot[bot] 1e9714088d
Merge pull request #4347 from restic/dependabot/go_modules/golang.org/x/sync-0.2.0 2023-06-02 20:32:27 +00:00
Michael Eischer 58e3f5955c
Merge pull request #4350 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.6.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.4.0 to 1.6.0
2023-06-02 22:26:55 +02:00
dependabot[bot] dd8d8b1ae0
build(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.6.1 to 1.7.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.6.1...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:22:39 +00:00
dependabot[bot] e1ac0f0e0c
build(deps): bump golang.org/x/sync from 0.1.0 to 0.2.0
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.1.0 to 0.2.0.
- [Commits](https://github.com/golang/sync/compare/v0.1.0...v0.2.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 20:22:36 +00:00
Michael Eischer a88d90b8e5
Merge pull request #4346 from restic/dependabot/go_modules/golang.org/x/oauth2-0.8.0
build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0
2023-06-02 22:21:22 +02:00
Michael Eischer 2fcb3947df prune: trigger GC after prune planning 2023-06-02 21:57:40 +02:00
Michael Eischer eef0ee7a85 repository: trigger GC after loading the index
Loading the index requires some scratch space, thus make sure that this
memory does not factor into the targeted gc memory usage limit.
2023-06-02 21:56:14 +02:00
Michael Eischer f1b73c9301 Reduce GOGC to 50
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
2023-06-02 21:51:50 +02:00
Michael Eischer ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
Kenny Y 098de3554c
Add pull request link 2023-06-01 21:30:51 -04:00
Kenny Y 8812dcd56a
Add --human-readable to ls and find output
Modifies format module to add options for human readable storage size formatting, using size parsing already in ui/format.
Cmd flag --human-readable added to ls and find commands.
Additional option added to formatNode to support printing size in regular or new human readable format
2023-06-01 21:18:18 -04:00
dependabot[bot] 379282299a
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.4.0 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.4.0...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-01 01:59:16 +00:00
dependabot[bot] 0c796dbd9b
build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.7.0 to 0.8.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-01 01:58:29 +00:00
Daniel Gibson f9dded83b3 doc/050_restore.rst: Mention that restic restore is fastest
Especially with lots of files (like when backing up git repos),
restic mount + cp -r can be really slow.
2023-05-31 22:44:36 +02:00
Michael Eischer 88a10a368f
Merge pull request #4342 from greatroar/errors
cmd: Don't check for errno == 0
2023-05-31 22:17:07 +02:00
Michael Eischer 9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer fc05e35a08 index: let indexmap.Each iterate in allocation order
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
2023-05-30 20:12:36 +02:00
Michael Eischer f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer 12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer fed33295c3 index: store indexEntries in hashed array tree
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
2023-05-29 00:24:15 +02:00
Michael Eischer b217f38ee7 index: Remove pointers from within indexentrys
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
2023-05-29 00:24:15 +02:00
Michael Eischer 0c1240360d index: add garbage collection benchmark
Allocates an index and repeatedly triggers the GC.
2023-05-29 00:23:04 +02:00
Michael Eischer ffca602315 repository: Fix panic in benchmarkLoadIndex 2023-05-28 23:55:47 +02:00
greatroar da419be43c cache: Restructure New to remove redundant operations
New and its helpers used to create the cache directories several times
over. They now only do so once. The added test ensures that the cache is
produced in a consistent state when parts are deleted.
2023-05-27 10:32:08 +02:00
greatroar 8f1ca8fabe cmd: Don't check for errno == 0
Maybe term.Restore's predecessor could return this, but the current
implementation never does.
2023-05-27 08:52:37 +02:00
Michael Eischer f74dad2afb
Merge pull request #4340 from restic/least-privilege-ci
CI: least privilege for workflows
2023-05-26 20:29:05 +02:00
Michael Eischer 3e287afdbf
Merge pull request #4226 from Sohalt/gcs-region
gs: support other regions than us
2023-05-26 20:28:52 +02:00
Michael Eischer 06894484a1 shorten changelog title 2023-05-26 20:05:49 +02:00
Michael Eischer 6e5b42d5c4 CI: least privilege for workflows 2023-05-26 19:58:02 +02:00
sohalt ed5b2c2c9b gs: support other regions than us 2023-05-26 19:54:42 +02:00
Michael Eischer 13c32b0fbe
Merge pull request #4328 from MichaelEischer/cleanup-integration-tests
Cleanup integration tests
2023-05-26 19:49:16 +02:00
Michael Eischer 9747cef338 fix linter warnings 2023-05-26 19:38:44 +02:00
Michael Eischer 8e913e6d3a repair index: always read Quiet flags from GlobalOptions passed as parameter 2023-05-26 19:38:44 +02:00
Michael Eischer b93459cbb0 repair snapshots: use local copy of globalOptions to open repository 2023-05-26 19:38:44 +02:00
Michael Eischer 692f81ede8 cleanup prune integration test 2023-05-26 19:38:44 +02:00
Michael Eischer 7a268e4aba always access stdout/stderr via globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer 4b3a0b4104 read JSON/Quiet flag from the passed in globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer cebce52c16 test: add helper to capture stdout for integration tests 2023-05-26 19:38:44 +02:00
Michael Eischer 675a49a95b Restructure integration tests
The tests are now split into individual files for each command. The
separation isn't perfect as many tests make use of multiple commands. In
particular `init`, `backup`, `check` and `list` are used by a larger
number of test cases.

Most tests now reside in files name cmd_<name>_integration_test.go. This
provides a certain indication which commands have significant test
coverage.
2023-05-26 19:38:43 +02:00
Michael Eischer e2dba9f5c7 test: cleanup a some check calls 2023-05-26 19:34:57 +02:00
Michael Eischer 06fd6b54d7 test: print log output if testRunCheck fails 2023-05-26 19:34:57 +02:00
Michael Eischer 419e6f26b1 tests: Simplify checks that a specific number of snapshots exists 2023-05-26 19:34:57 +02:00
Michael Eischer c3212ab6a6 test: use standard logging methods from testing for the test helpers
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
2023-05-26 19:34:57 +02:00
Michael Eischer 658aa4c0f7
Merge pull request #4339 from greatroar/ioctl
backend, termstatus: fix ioctl calls for 64-bit big-endian platforms
2023-05-26 19:34:14 +02:00
Michael Eischer 998cf5a7f8
Merge pull request #4333 from MichaelEischer/staticcheck
Update golangci-lint and resolve all errors
2023-05-26 19:34:00 +02:00
Michael Eischer 7eec91f841 add changelog 2023-05-26 19:23:51 +02:00
greatroar 51dc80be5b backend: Use IoctlSetPointerInt for tcsetpgrp
This function casts its argument to int32 before passing it to the
system call, so that big-endian CPUs read the lower rather than the
upper 32 bits of the pid.

This also gets rid of the last import of "unsafe" in the Unix build.
I changed syscall to x/sys/unix while I was at it, to remove one more
import line. The constants and types there are aliases for their syscall
counterparts.
2023-05-25 17:31:51 +02:00
greatroar ddbc0c1b37 termstatus: Fix IsProcessBackground for 64-bit big-endian Linux
Fixes #4223. Verified with QEMU on linux/amd64.
2023-05-25 17:20:42 +02:00
Michael Eischer ecbf8e055c CI: upgrade golangci-lint and check full codebase 2023-05-18 21:17:53 +02:00
Michael Eischer 16ba237d8b azure: Fix missing rate limiting 2023-05-18 21:17:53 +02:00
Michael Eischer a466e945d9 stats: pass StatsOptions via parameter 2023-05-18 21:17:53 +02:00
Michael Eischer 03b9764bce init/generate: check that no parameters are passed 2023-05-18 21:17:53 +02:00
Michael Eischer 22c9276719 rest: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer 1e33b285c1 lock: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer c05f96e6b9 repair index: remove no longer used paramter 2023-05-18 21:17:53 +02:00
Michael Eischer 94752b7ee2 backup: Remove unused parameters from rejectFunc collection 2023-05-18 21:17:53 +02:00
Michael Eischer 0058745881 test: use parameter instead of hardcoded constant 2023-05-18 21:17:53 +02:00
Michael Eischer a719d10e22 termstatus: remove unused parameter from helpers 2023-05-18 21:17:53 +02:00
Michael Eischer b0a01ae68a ui/backup: remove unused parameter from ProgressPrinter interface 2023-05-18 21:17:53 +02:00
Michael Eischer 472bf5184f Replace lots of unused parameters with _
The parameters are required by the implemented function signature or interface.
2023-05-18 21:17:53 +02:00
Michael Eischer d1a5ec7839 Rename unused testing parameter to _
The parameter is an additional marker that the test helper must only be
used for tests.
2023-05-18 21:17:53 +02:00
Michael Eischer 1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer 5e4e268bdc Use _ as parameter name for unused Context
The context is required by the implemented interface.
2023-05-18 21:15:45 +02:00
Michael Eischer 3252f60df5 Ignore deprecated imports for poly1305 and openpgp 2023-05-18 21:15:45 +02:00
Michael Eischer 2fa8b96843 Don't shadow builtins 2023-05-18 21:15:45 +02:00
Michael Eischer 7a01bd3b67 Remove unnecessary else block 2023-05-18 21:15:45 +02:00
Michael Eischer 319087c056 Remove redundant if ...; err != nil; return pattern 2023-05-18 21:15:45 +02:00
Michael Eischer 6ed73ed408 CI: remove deprecated linters 2023-05-18 21:15:45 +02:00
Michael Eischer c832a492ac fuse: use syscall errnos directly to fix deprecations 2023-05-18 21:15:45 +02:00
Michael Eischer e01baeabba Use either test or rtest to refer to internal test helpers
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
Michael Eischer bfc9c6c971
Merge pull request #4331 from MichaelEischer/fix-mount-failures
Hopefully fix `TestMount` failures
2023-05-18 20:22:57 +02:00
Michael Eischer 5773b86d02 repository: Push all usage of errors.Fatal out of the package
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.

Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:27:41 +02:00
Michael Eischer a013014c24 backup: return normal error if --group-by cannot be parsed
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:23:31 +02:00
Michael Eischer f9850b79b5 rest/sftp: Remove redundant fatal from error message
This caused restic to exit with error messages like
`Fatal: parsing repository location failed: Fatal: sftp path [...]`
`Fatal: create repository at rest:http://localhost:12345/ failed: Fatal: config file already exists`
2023-05-18 17:22:26 +02:00
Michael Eischer 2f518b7241
Merge pull request #4304 from MichaelEischer/unlimited-lock-refresh
lock: Do not limit backend concurrency for lock files
2023-05-13 21:40:29 +02:00
Michael Eischer 49be202cb0
Merge pull request #4315 from MichaelEischer/restore-hardlink-progress
restore: Correctly account for hardlinks in progress bar
2023-05-13 21:37:40 +02:00
Michael Eischer 19ebc1b786 restore: Add basic test for progress bar accounting of hardlinks 2023-05-08 20:51:49 +02:00
Michael Eischer 23a122a901 restore: count files in the same way as the stats command 2023-05-08 20:51:49 +02:00
Michael Eischer e77002f841 restore: correctly count hardlinks in progress bar
For hardlinked files, only the first instance of that file increases the
amount of bytes to restore. All later instances only increase the file
count but not the restore size.
2023-05-08 20:51:49 +02:00
Michael Eischer d05f6211d1 lock: Do not limit backend concurrency for lock files
restic must be able to refresh lock files in time. However, large
uploads over slow connections can cause the lock refresh to be stuck
behind the large uploads and thus time out.
2023-05-08 19:04:46 +02:00
Michael Eischer ee3c55ea3d
Merge pull request #2876 from aawsome/new-repair-command
Add repair command
2023-05-05 23:22:24 +02:00
Michael Eischer db046c0acc
Merge pull request #4318 from MichaelEischer/status-output-truncation
Fix status output truncation
2023-05-05 23:09:59 +02:00
Michael Eischer 3e6a26e2e9
Merge pull request #4317 from MichaelEischer/debug-test-mount-failures
mount: enable debug logging for the flaky TestMount test
2023-05-05 23:08:03 +02:00
greatroar 65c5e511a1 ui/termstatus: Add test for Terminal.SetStatus 2023-05-05 22:51:41 +02:00
Michael Eischer 6d10c655a0 termstatus: test status line sanitization 2023-05-05 22:51:41 +02:00
Michael Eischer bb40e49e75 ui/termstatus: Fix truncation of status output
The last line was not truncated as expected
2023-05-05 22:45:52 +02:00
Michael Eischer fefe15d7a1
Merge pull request #4316 from MichaelEischer/fix-check-modified-data
Fix flaky TestCheckerModifiedData test
2023-05-05 22:38:28 +02:00
Michael Eischer 78e5aa6d30 repair snapshots: add basic tests 2023-05-04 23:00:46 +02:00
Michael Eischer 1cb11ad8ad mount: enable debug logging for the flaky TestMount test
The test case fails from time to time with an Input/Output error while
trying to access the snapshots directory.
2023-05-01 18:03:17 +02:00
Michael Eischer 90a663c94f
Merge pull request #4308 from greatroar/tar-header
dump: Report filename with tar.ErrFieldTooLong
2023-05-01 17:30:31 +02:00
Michael Eischer 88a7231217 report snapshot id if loading failed 2023-05-01 17:24:13 +02:00
Michael Eischer c0627dc80d check: Fix flaky TestCheckerModifiedData
The test had a 4% chance of not modified the data read from the
repository, in which case the test would fail. Change the data
manipulation to just modified each read operation.
2023-05-01 17:18:19 +02:00
Michael Eischer e71367e6b9 repair snapshots: update changelog 2023-05-01 16:06:17 +02:00
Michael Eischer 5aa37acdaa repair snapshots: cleanup command help 2023-05-01 16:06:17 +02:00
Michael Eischer 9c64a95df8 doc: rewrite troubleshooting section 2023-05-01 16:06:17 +02:00
Michael Eischer 7c8dd61e8c repair snapshots: cleanup warnings 2023-05-01 15:22:30 +02:00
Michael Eischer f6cc10578d repair snapshots: Always sanitize file nodes
If the node for a file is intact, this is a no-op.
2023-05-01 15:22:18 +02:00
Michael Eischer 4ce87a7f64 repair snapshots: port to filterAndReplaceSnapshot
The previous approach of rewriting all snapshots first, then flushing
the repository data and finally removing old snapshots has the downside
that an interrupted command execution leaves behind broken snapshots as
not all new data is already flushed.
2023-05-01 15:22:03 +02:00
Michael Eischer e17ee40a31 repair snapshots: Port to use walker.TreeRewriter 2023-05-01 15:20:36 +02:00
Michael Eischer 1bd1f3008d walker: extend TreeRewriter to support snapshot repairing
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
2023-05-01 15:20:24 +02:00
Michael Eischer 38dac78180 walker: restructure FilterTree into TreeRewriter
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
2023-05-01 15:20:12 +02:00
Michael Eischer bc2399fbd9 walker: recurse into directory based on node type
A broken directory might also not have a subtree.
2023-05-01 15:20:00 +02:00
Michael Eischer 1a9705fc95 walker: Simplify change detection in FilterTree
Now the rewritten tree is always serialized which makes sure that we
don't accidentally miss any relevant changes.
2023-05-01 15:19:48 +02:00
Michael Eischer 8c4caf09a8 repair snapshots: Do not rename repaired files
The files in a tree must be sorted in lexical order. However, this
cannot be guaranteed when appending a filename suffix. For two files

file, file.rep

where "file" is broken, this would result in

file.repaired, file.rep

which is no longer sorted.

In addition, adding a filename suffix is also prone to filename
collisions which would require a rather complex search for a
collision-free name in order to work reliably.
2023-05-01 15:19:36 +02:00
Michael Eischer 375189488c rewrite: prepare for code sharing with rewrite snapshots 2023-05-01 15:19:24 +02:00
Michael Eischer 903651c719 repair snapshots: partially synchronize code with rewrite command
Simplify CLI options:
* Rename "DeleteSnapshots" to "Forget"
* Replace "AddTag" and "Append" with hardcoded values

Change output and snapshot modifications to be more in line with the
"rewrite" command.
2023-05-01 15:19:11 +02:00
Michael Eischer 118d599d0a Rename 'rebuild-index' to 'repair index'
The old name still works, but is deprecated.
2023-05-01 15:16:44 +02:00
Michael Eischer db459eda21 move to subcommand 2023-05-01 15:15:48 +02:00
Michael Eischer a14a63cd29 modernize code 2023-05-01 15:12:09 +02:00
Alexander Weiss 947f0c345e correct typos 2023-05-01 14:56:42 +02:00
Alexander Weiss d23a2e1925 better error handling and correct nil tree behavior 2023-05-01 14:56:42 +02:00
Alexander Weiss 08ae708b3b make linter happy 2023-05-01 14:56:42 +02:00
Alexander Weiss 99a05d5ab2 Update troubleshooting documentation 2023-05-01 14:56:42 +02:00
Alexander Weiss 6557f36f61 Add changelog and docu for #2876 2023-05-01 14:56:42 +02:00
Alexander Weiss 5f58797ba7 Add repair command 2023-05-01 14:56:42 +02:00
Alexander Weiss 9cef6b4c69 Add troubleshooting documentation 2023-05-01 14:56:42 +02:00
Michael Eischer 8a78a042db
Merge pull request #4310 from restic/dependabot/go_modules/github.com/klauspost/compress-1.16.5
build(deps): bump github.com/klauspost/compress from 1.16.0 to 1.16.5
2023-05-01 14:35:57 +02:00
Michael Eischer b491af2b57
Merge pull request #4311 from restic/dependabot/go_modules/golang.org/x/crypto-0.8.0
build(deps): bump golang.org/x/crypto from 0.7.0 to 0.8.0
2023-05-01 14:35:37 +02:00
dependabot[bot] d747a9c401
build(deps): bump golang.org/x/crypto from 0.7.0 to 0.8.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 11:15:38 +00:00
dependabot[bot] 888c1ae63a
build(deps): bump github.com/klauspost/compress from 1.16.0 to 1.16.5
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.16.0 to 1.16.5.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.16.0...v1.16.5)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 11:07:29 +00:00
Michael Eischer 5eeb257c95
Merge pull request #4312 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.52
build(deps): bump github.com/minio/minio-go/v7 from 7.0.50 to 7.0.52
2023-05-01 13:00:48 +02:00
Michael Eischer ae6729cf89
Merge pull request #4309 from restic/dependabot/go_modules/golang.org/x/oauth2-0.7.0
build(deps): bump golang.org/x/oauth2 from 0.6.0 to 0.7.0
2023-05-01 12:58:55 +02:00
dependabot[bot] 6abd494915
build(deps): bump github.com/minio/minio-go/v7 from 7.0.50 to 7.0.52
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.50 to 7.0.52.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.50...v7.0.52)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 01:59:50 +00:00
dependabot[bot] 7147a54ceb
build(deps): bump golang.org/x/oauth2 from 0.6.0 to 0.7.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 01:59:10 +00:00
Michael Eischer 8d971172c4
Merge pull request #4306 from MichaelEischer/document-restic-compression-env
Document that the compression mode can be set via $RESTIC_COMPRESSION
2023-04-30 18:26:10 +02:00
Michael Eischer 37d0e323eb doc: update help output in manual_rest.rst 2023-04-30 16:12:29 +02:00
Michael Eischer face5bd7f7 Document that the compression mode can be set via $RESTIC_COMPRESSION 2023-04-30 16:11:53 +02:00
Michael Eischer 1daf928a77
Merge pull request #4305 from MichaelEischer/stracktrace-for-windows
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
2023-04-30 16:08:58 +02:00
Michael Eischer 37bab08181
Merge pull request #4302 from MichaelEischer/logging-cleanup
Tweak debug logs
2023-04-30 16:08:08 +02:00
Michael Eischer 6dc2324d2e
Merge pull request #4301 from MichaelEischer/less-restore-memory
restore: slightly reduce memory usage while restoring files
2023-04-30 16:07:43 +02:00
Michael Eischer 850cd9aace
Merge pull request #4299 from MichaelEischer/update-scripting-docs
doc: recommend `cat config` to check if a repository exists
2023-04-30 16:04:59 +02:00
greatroar b50ff04cf3 dump: Report filename with tar.ErrFieldTooLong
Updates #4307.
2023-04-27 11:45:41 +02:00
Michael Eischer c8641f4479 Merge branch 'patch-release' 2023-04-24 20:44:38 +02:00
Alexander Neumann ac7ac0cb97 Set development version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann be8be3397c Add version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann db6b4f8912 Update manpages and auto-completion 2023-04-24 20:28:37 +02:00
Alexander Neumann 1f3f042f32 Generate CHANGELOG.md for 0.15.2 2023-04-24 20:28:24 +02:00
Alexander Neumann 0aaa4e6cbe Prepare changelog for 0.15.2 2023-04-24 20:28:24 +02:00
Leo R. Lundgren 0bac935dac doc: Polish changelogs 2023-04-23 22:10:04 +02:00
Michael Eischer 306a29980a Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
2023-04-23 15:50:40 +02:00
Michael Eischer 1e6e9f9bd0 tweak changelogs 2023-04-23 12:46:55 +02:00
greatroar f342db7666 ui/termstatus: Quote funny filenames
Fixes #2260, #4191.
2023-04-23 12:45:40 +02:00
Michael Eischer 41cc320145
Merge pull request #4166 from MichaelEischer/cache-mkdirall
cache: Create missing intermediate directories when caching a file
2023-04-23 11:44:18 +02:00
Michael Eischer cdb0fb9c06 tweak debug logs 2023-04-23 11:38:06 +02:00
Michael Eischer 94cbc6392d restore: slightly reduce memory usage while restoring files
The information which target files are contained in a pack file is no
longer necessary after processing a pack.
2023-04-23 11:33:21 +02:00
Michael Eischer 78a1757e5a Cancel current command if cache becomes unusable
If the cache suddenly disappears, the current command will now fail.
2023-04-23 11:31:15 +02:00
Michael Eischer 22562d2132
Merge pull request #4300 from MichaelEischer/less-flaky-tests
Increase timeouts for lock refresh tests
2023-04-23 11:24:59 +02:00
Michael Eischer 51d823348d
Merge pull request #4286 from MichaelEischer/backend-cleanup-logging
Normalize backend logging and connection limiting
2023-04-23 11:24:26 +02:00
Michael Eischer 831f593b87 backend/sema: Add tests 2023-04-22 13:20:20 +02:00
Michael Eischer 179e11c2ae Increase timeouts for lock refresh tests
When saving files to the local backend, in some cases the used fsync
calls are slow enough to cause the tests to time out. Thus, increase the
test timeouts as a stopgap measure until we can use the mem backend for
these tests.
2023-04-22 12:45:59 +02:00
Michael Eischer ebba233a3a backend/sema: rename constructor to NewBackend 2023-04-22 12:32:57 +02:00
rawtaz 8479390d7c
Merge pull request #3939 from Fabien-jrt/resticprofile
doc: Add resticprofile to scheduling section
2023-04-22 01:24:34 +02:00
Fabien-Jrt 756f43d5f9 doc: Add resticprofile to scheduling section 2023-04-22 01:04:48 +02:00
Michael Eischer affd04c125 doc: recommend `cat config` to check if a repository exists
This will be much faster if a large number of snapshots exists.
2023-04-21 22:56:31 +02:00
Michael Eischer f27750e270 backend/sema: rename type to connectionLimitedBackend 2023-04-21 22:53:09 +02:00
Michael Eischer 1dd873b706
Merge pull request #4293 from MichaelEischer/fix-groups-by
Fix parent snapshot selection for relative paths
2023-04-21 22:47:03 +02:00
Michael Eischer 7a60d9e54f
Merge pull request #4288 from MichaelEischer/log-warnings-to-debug-log
Add warnings via Warnf to the debug log
2023-04-21 22:39:00 +02:00
Michael Eischer 3001dd8c2b Add test to verify that the backup parent is correctly selected 2023-04-21 22:35:02 +02:00
Michael Eischer 4503aea0df
Merge pull request #4296 from MichaelEischer/rewrite-reorder-output
rewrite: log snapshot saved before removal of the old snapshot
2023-04-19 22:00:32 +02:00
Michael Eischer 09cddb8927 rewrite: log snapshot saved before removal of the old snapshot
The snapshot was already saved before removing the old snapshot. Only
the log messages were printed in the wrong order.
2023-04-17 21:00:45 +02:00
Michael Eischer 913eab3361
Merge pull request #4234 from thndrbrrr/forget-opts-neg1-means-forever-issue-2565
restic forget --keep-* options will interpret -1 as "forever"
2023-04-14 23:18:47 +02:00
Michael Eischer a9c7c12276
Merge pull request #4265 from HeikoSchlittermann/chg/progress-to-stderr
generate: do not write progress to STDOUT
2023-04-14 23:18:23 +02:00
Michael Eischer 85eef232e6
Merge pull request #4232 from thndrbrrr/copy-chunker-params-verbose-msg
init: Add --copy-chunker-params verbose msg
2023-04-14 23:17:56 +02:00
Michael Eischer 07a44a88f2 Fix snapshot filtering for relative paths in the backup command
The snapshot filtering internally converts relative paths to absolute
ones to ensure that the parent snapshots selection works for backups of
relative paths.
2023-04-14 22:58:34 +02:00
Michael Eischer 48e065d971 Sync dependency upgrades from master
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob still uses v0.5.1
as upgrading it would increase the minimum Go version on Solaris to
1.20.
2023-04-14 22:58:34 +02:00
Michael Eischer a06d927dce
Merge pull request #4291 from greatroar/widechars
ui/termstatus: Optimize Truncate
2023-04-14 22:48:34 +02:00
Michael Eischer fd3ed9e2f4
Merge pull request #4192 from greatroar/quote
ui/termstatus: Quote funny filenames
2023-04-14 22:39:09 +02:00
Michael Eischer 6042df075f migrations: Fix S3 backend detection 2023-04-14 22:32:16 +02:00
Michael Eischer c934c99d41 gs: replace usage of context.Background() 2023-04-14 22:32:15 +02:00
Michael Eischer 616926d2c1 gs: use IsNotExist to check error 2023-04-14 22:32:15 +02:00
Michael Eischer 05abc6d6f5 backend: deduplicate implementation of Delete() method 2023-04-14 22:32:15 +02:00
Michael Eischer 45244fdf68 backend: remove parameter validation tests
These parameter validations have been factored out into
SemaphoreBackend.
2023-04-14 22:32:15 +02:00
Michael Eischer 803640ba4b backend: remove a few unnecessary debug logs 2023-04-14 22:32:15 +02:00
Michael Eischer 8e1e3844aa backend: factor out connection limiting and parameter validation
The SemaphoreBackend now uniformly enforces the limit of concurrent
backend operations. In addition, it unifies the parameter validation.

The List() methods no longer uses a semaphore. Restic already never runs
multiple list operations in parallel.

By managing the semaphore in a wrapper backend, the sections that hold a
semaphore token grow slightly. However, the main bottleneck is IO, so
this shouldn't make much of a difference.

The key insight that enables the SemaphoreBackend is that all of the
complex semaphore handling in `openReader()` still happens within the
original call to `Load()`. Thus, getting and releasing the semaphore
tokens can be refactored to happen directly in `Load()`. This eliminates
the need for wrapping the reader in `openReader()` to release the token.
2023-04-14 22:32:15 +02:00
Michael Eischer 8b5ab5b59f dryrun: fix outdated comments 2023-04-14 22:32:15 +02:00
Michael Eischer 4703473ec5 backend: extract most debug logs into logger backend 2023-04-14 22:32:15 +02:00
Michael Eischer 8bfc2519d7 backend: Deduplicate sanity checks for parameters of Load() method
The check is now handled by backend.DefaultLoad. This also guarantees
consistent behavior across all backends.
2023-04-14 22:32:15 +02:00
Michael Eischer ba16904eed backup: Add test to verify parent snapshot selection for relative paths 2023-04-14 22:21:43 +02:00
Michael Eischer 2841a87cc6 Fix snapshot filtering for relative paths in the backup command
The snapshot filtering internally converts relative paths to absolute
ones to ensure that the parent snapshots selection works for backups of
relative paths.
2023-04-14 21:53:55 +02:00
Michael Eischer fab4a8a4d2 Properly initialize the --group-by option for backup tests 2023-04-14 21:53:01 +02:00
greatroar 3b24c15c3d fuse: Mix inode hashes in a non-symmetric way
Since 0.15 (#4020), inodes are generated as hashes of names, xor'd with
the parent inode. That means that the inode of a/b/b is

	h(a/b/b) = h(a) ^ h(b) ^ h(b) = h(a).

I.e., the grandchild has the same inode as the grandparent. GNU find
trips over this because it thinks it has encountered a loop in the
filesystem, and fails to search a/b/b. This happens more generally when
the same name occurs an even number of times.

Fix this by multiplying the parent by a large prime, so the combining
operation is not longer symmetric in its arguments. This is what the FNV
hash does, which we used prior to 0.15. The hash is now

	h(a/b/b) = h(b) ^ p*(h(b) ^ p*h(a))

Note that we already ensure that h(x) is never zero.

Collisions can still occur, but they should be much less likely to occur
within a single path.

Fixes #4253.
2023-04-14 20:50:39 +02:00
greatroar 4304e01ca2 fuse: Report fuse.Attr.Blocks correctly
Fixes #4239.
2023-04-14 20:50:08 +02:00
Torben Giesselmann 6d6c04abef forget: Simplify usage text 2023-04-14 10:05:23 -07:00
greatroar 49e32f3f8a ui/termstatus: Optimize Truncate
x/text/width.LookupRune has to re-encode its argument as UTF-8,
while LookupString operates on the UTF-8 directly.
The uint casts get rid of a bounds check.

Benchmark results, with b.ResetTimer introduced first:

name               old time/op  new time/op  delta
TruncateASCII-8    69.7ns ± 1%  55.2ns ± 1%  -20.90%  (p=0.000 n=20+18)
TruncateUnicode-8   350ns ± 1%   171ns ± 1%  -51.05%  (p=0.000 n=20+19)
2023-04-14 11:13:39 +02:00
greatroar 9412f37e50 ui/termstatus: Quote funny filenames
Fixes #2260, #4191.
2023-04-14 09:36:02 +02:00
Ian Muge 593eb710b4 added changelog 2023-04-13 22:54:36 +02:00
greatroar 97274ecabd cmd, restic: Refactor and fix snapshot filtering
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
2023-04-13 22:51:45 +02:00
Michael Eischer 74f7dd0b38 Make help for --verbose less confusing
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
2023-04-13 22:46:37 +02:00
Michael Eischer 21ad357c10 add linux/riscv64 builds 2023-04-13 22:45:35 +02:00
Heiko Schlittermann (HS12-RIPE) 087cf7e114
generate: write progress to STDOUT if this is a terminal
This allows usage as:

  eval "$(restic generated --bash-completion /dev/stdout)"
2023-04-12 22:39:59 +02:00
Michael Eischer e604939e72 Debug log status code if execution is interrupted
Currently, there is no clear indication in the debug log whether restic
exited or not.
2023-04-12 21:58:35 +02:00
Michael Eischer 37aca6bec0 Add warnings via Warnf to the debug log 2023-04-12 21:37:37 +02:00
Michael Eischer bdcafbc11c
Merge pull request #4282 from TorstenC/patch-1
040_backup.rst line 228 - #4280 --exclude-caches
2023-04-11 22:34:16 +02:00
Torsten C ba33e41068 040_backup.rst line 228 - #4280 --exclude-caches
Proposal for issue #4280
2023-04-11 22:33:10 +02:00
Michael Eischer 4661f45a8c
Merge pull request #4220 from Skaronator/master
Add jq to container image
2023-04-11 22:26:02 +02:00
Michael Eischer 2091fc0dde
Merge pull request #4107 from jooola/feature-wait-for-unlock
Add a global option --retry-lock
2023-04-11 22:21:01 +02:00
Michael Eischer 9238dcc81a
Merge pull request #4201 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.0.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 0.5.1 to 1.0.0
2023-04-11 22:20:38 +02:00
Torben Giesselmann 71537da4b0 init: Add --copy-chunker-params verbose msg 2023-04-09 14:59:07 -07:00
Torben Giesselmann ce51d2f3c0 forget: Update usage text 2023-04-09 12:59:15 -07:00
Torben Giesselmann 403b7ca2be forget: Update documentation 2023-04-09 12:57:37 -07:00
Torben Giesselmann 309cf0586a Add changelog entry 2023-04-09 12:05:15 -07:00
Torben Giesselmann 9f9e91eb0d Fix comment to comply with linter 2023-04-09 11:47:10 -07:00
Michael Eischer 1f43003cc1 Merge branch 'master' into feature-wait-for-unlock 2023-04-07 20:32:21 +02:00
Michael Eischer 8ce5f29758 lock: increase test timeout tolerances to avoid test failures 2023-04-07 20:07:05 +02:00
Michael Eischer 8afc117aa3
Merge pull request #4285 from MichaelEischer/debug-lock-refresh-failures
lock: Improve debug logging in the test
2023-04-07 20:06:15 +02:00
Michael Eischer cf1cc1fb72 lock: Print stacktrace if TestLockSuccessfulRefresh fails 2023-04-07 19:51:31 +02:00
Michael Eischer 64233ca0a7 lock: Improve debug logging in the test 2023-04-07 19:43:00 +02:00
jo ea59896bd6
Add a global option --retry-lock
Fixes restic#719

If the option is passed, restic will wait the specified duration of time
and retry locking the repo every 10 seconds (or more often if the total
timeout is relatively small).

- Play nice with json output
- Reduce wait time in lock tests
- Rework timeout last attempt
- Reduce test wait time to 0.1s
- Use exponential back off for the retry lock
- Don't pass gopts to lockRepo functions
- Use global variable for retry sleep setup
- Exit retry lock on cancel
- Better wording for flag help
- Reorder debug statement
- Refactor tests
- Lower max sleep time to 1m
- Test that we cancel/timeout in time
- Use non blocking sleep function
- Refactor into minDuration func

Co-authored-by: Julian Brost <julian@0x4a42.net>
2023-04-07 16:24:14 +02:00
Michael Eischer 806a0cdce3 bump minimum go version on Solaris to 1.20 2023-04-07 15:40:57 +02:00
Michael Eischer faa83db9e4 azure: Adapt code to API change 2023-04-07 15:05:55 +02:00
dependabot[bot] 9358a5fb37
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 0.5.1 to 1.0.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/armcore/v0.5.1...v1.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 11:03:25 +00:00
Michael Eischer 71c9516b26
Merge pull request #4247 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.4.0
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.3.1 to 1.4.0
2023-04-07 13:01:04 +02:00
Michael Eischer 17ac91fba3
Merge pull request #4283 from restic/dependabot/go_modules/google.golang.org/api-0.116.0
build(deps): bump google.golang.org/api from 0.114.0 to 0.116.0
2023-04-07 12:58:54 +02:00
Michael Eischer 26a3c47c5c
Merge pull request #4255 from greatroar/fuse-hash
fuse: Mix inode hashes in a non-symmetric way
2023-04-07 12:56:57 +02:00
Michael Eischer 7896e50301
Merge pull request #4271 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.50
build(deps): bump github.com/minio/minio-go/v7 from 7.0.49 to 7.0.50
2023-04-07 12:47:30 +02:00
dependabot[bot] 9584cbda90
build(deps): bump google.golang.org/api from 0.114.0 to 0.116.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.114.0 to 0.116.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.114.0...v0.116.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 10:27:38 +00:00
dependabot[bot] cacc48fc09
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.3.1 to 1.4.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.1...sdk/azcore/v1.4.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-07 10:27:30 +00:00
Michael Eischer 7fbaca577b
Merge pull request #4272 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.30.1
build(deps): bump cloud.google.com/go/storage from 1.29.0 to 1.30.1
2023-04-07 12:26:20 +02:00
Michael Eischer 1f9dd84d1e
Merge pull request #3991 from mark-herrmann/feature/restore-progress
restore: show progress
2023-04-07 12:24:55 +02:00
Michael Eischer c8ff5592b8
Merge pull request #4273 from ducalex/gitignore-restic-exe
Added restic.exe to the .gitignore
2023-04-07 12:10:09 +02:00
Mark Herrmann f875a8843d restore: Add progress bar
Co-authored-by: Mark Herrmann <mark.herrmann@mailbox.org>
2023-04-07 12:08:23 +02:00
Alex Duchesne 0ed5c20c57 Added restic.exe to the .gitignore 2023-03-28 23:07:25 -04:00
dependabot[bot] 21edbdc3ac
build(deps): bump cloud.google.com/go/storage from 1.29.0 to 1.30.1
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.29.0 to 1.30.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.29.0...spanner/v1.30.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-26 20:14:09 +00:00
dependabot[bot] 220d937975
build(deps): bump github.com/minio/minio-go/v7 from 7.0.49 to 7.0.50
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.49 to 7.0.50.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.49...v7.0.50)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-26 20:13:34 +00:00
rawtaz 5f13bbc118
Merge pull request #4270 from MichaelEischer/monthly-dependabot
Switch back to monthly dependabot updates
2023-03-26 22:09:21 +02:00
Michael Eischer 9672670756
Merge pull request #4240 from greatroar/fuse-blocks
fuse: Report fuse.Attr.Blocks correctly
2023-03-26 21:57:53 +02:00
Michael Eischer 5ac24a9744 Switch back to monthly dependabot updates 2023-03-26 21:55:27 +02:00
Michael Eischer 99e247caa6
Merge pull request #4259 from restic/dependabot/github_actions/actions/setup-go-4
build(deps): bump actions/setup-go from 3 to 4
2023-03-26 21:53:47 +02:00
Michael Eischer 0c705e07db
Merge pull request #4249 from restic/dependabot/go_modules/golang.org/x/oauth2-0.6.0
build(deps): bump golang.org/x/oauth2 from 0.5.0 to 0.6.0
2023-03-26 10:38:53 +02:00
Michael Eischer 024d01d85b
Merge pull request #4266 from restic/linkfix
doc: Correct broken links
2023-03-22 22:59:54 +01:00
Leo R. Lundgren 0666fa11b8 doc: Correct broken links 2023-03-22 17:37:58 +01:00
rawtaz caa0e89114
Merge pull request #4246 from greatroar/doc
doc: Fix links, focus less on implementation details
2023-03-22 14:47:31 +01:00
rawtaz 46a7072f3f
Merge pull request #4264 from GuitarBilly/patch-1
Update 110_talks.rst - Add changelog podcast.
2023-03-22 14:13:40 +01:00
GuitarBilly 1f12915b0c Update 110_talks.rst
Update 110_talks.rst

align date to 2nd of April as agreed.
( match podcast publication date )
2023-03-22 13:54:27 +01:00
greatroar a0885d5d69 fuse: Mix inode hashes in a non-symmetric way
Since 0.15 (#4020), inodes are generated as hashes of names, xor'd with
the parent inode. That means that the inode of a/b/b is

	h(a/b/b) = h(a) ^ h(b) ^ h(b) = h(a).

I.e., the grandchild has the same inode as the grandparent. GNU find
trips over this because it thinks it has encountered a loop in the
filesystem, and fails to search a/b/b. This happens more generally when
the same name occurs an even number of times.

Fix this by multiplying the parent by a large prime, so the combining
operation is not longer symmetric in its arguments. This is what the FNV
hash does, which we used prior to 0.15. The hash is now

	h(a/b/b) = h(b) ^ p*(h(b) ^ p*h(a))

Note that we already ensure that h(x) is never zero.

Collisions can still occur, but they should be much less likely to occur
within a single path.

Fixes #4253.
2023-03-21 17:33:18 +01:00
dependabot[bot] 5dccab701a
build(deps): bump actions/setup-go from 3 to 4
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-20 02:18:58 +00:00
Torben Giesselmann 5069c9edd9 Represent -1 as "all" in ExpirePolicy's Stringer 2023-03-15 15:07:51 -07:00
Torben Giesselmann 1a584cb16e Refactor policy sum calculation & duration parsing
- Convert policy sum calculation to function and move to tests.
- Remove parseDuration(...) and use ParseDurationOrPanic(...) instead.
2023-03-14 19:29:08 -07:00
Torben Giesselmann 84ede6ad7a forget: Prevent neg. values in --keep-within* opts 2023-03-14 19:20:03 -07:00
Torben Giesselmann b7f03d01b8 Add helper function for Duration parsing
Tests in cmd_forget_test.go need the same convenience function
that was implemented in snapshot_policy_test.go.

Function parseDuration(...) was moved to testing.go  and renamed to
ParseDurationOrPanic(...).
2023-03-14 19:16:24 -07:00
dependabot[bot] eaceaca113
build(deps): bump golang.org/x/oauth2 from 0.5.0 to 0.6.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-13 02:07:39 +00:00
greatroar a7ac9a4769 doc: Fix some links, focus less on implementation details
The godoc for filepath.Match has the syntax, which is what is important
for writing patterns. Use pkg.go.dev instead of golang.org/pkg.

For #4245. Not all links fixed yet.
2023-03-13 00:23:15 +01:00
Michael Eischer 23ed03a267
Merge pull request #4243 from restic/update-go
Run tests with Go 1.20
2023-03-12 18:15:54 +01:00
Alexander Neumann aac8c5a7ca
Merge pull request #4244 from MichaelEischer/fix-doc-typos
Fix documentation typos and build with Sphinx 6
2023-03-12 15:04:26 +01:00
Michael Eischer 7c8a401d97
Merge pull request #4176 from LXGaming/fix/impostor
Fix scan_finished JSON MessageType
2023-03-12 12:14:40 +01:00
Alexander Neumann d83332315c Update golangci-lint 2023-03-12 12:05:59 +01:00
Alexander Neumann c2703e5024 Increase timeout for golangci-lint 2023-03-12 12:02:22 +01:00
Michael Eischer eb7dbc88b5
Merge pull request #4210 from aawsome/patch-1
docs: Corrections/extra information in design.rst
2023-03-12 12:00:25 +01:00
Michael Eischer e02aadf1d2 doc: use extlink for links to issues 2023-03-12 11:55:36 +01:00
Michael Eischer c4e6b198ae doc: fix building with Sphinx 6 2023-03-12 11:45:26 +01:00
Michael Eischer f47c8eebb7 doc: fix broken link in rewrite command docs 2023-03-12 11:44:56 +01:00
Alexander Neumann 996e2ac7c5 Run tests with Go 1.20 2023-03-12 11:41:22 +01:00
Michael Eischer a67d3781a3 doc: apply review comments and improve link formatting 2023-03-12 11:37:37 +01:00
Michael Eischer 153a73ebba
Merge pull request #4242 from MichaelEischer/fix-mixed-index-in-design-docs
doc: Remove mixed pack files from index example
2023-03-12 11:24:21 +01:00
Michael Eischer f9d6e3a035
Merge pull request #4236 from restic/dependabot/go_modules/golang.org/x/crypto-0.7.0
build(deps): bump golang.org/x/crypto from 0.5.0 to 0.7.0
2023-03-11 20:42:38 +01:00
dependabot[bot] 51656e8764
build(deps): bump golang.org/x/crypto from 0.5.0 to 0.7.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.5.0 to 0.7.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.5.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-11 19:17:59 +00:00
Michael Eischer 9e23200bff
Merge pull request #4219 from ianmuge/feat_upgrade_minio_for_new_region
(feat) Upgrade Minio to 7.0.49
2023-03-11 20:17:03 +01:00
Michael Eischer b884643b40
Merge pull request #4224 from restic/dependabot/go_modules/github.com/klauspost/compress-1.16.0
build(deps): bump github.com/klauspost/compress from 1.15.15 to 1.16.0
2023-03-11 20:14:55 +01:00
Ian Muge 5be4845710 added changelog 2023-03-11 20:04:12 +01:00
Ian Muge cfa3c6abc5 upgrade Minio to 7.0.49 2023-03-11 20:04:12 +01:00
Michael Eischer f499e66032
Merge pull request #4235 from restic/dependabot/go_modules/google.golang.org/api-0.111.0
build(deps): bump google.golang.org/api from 0.109.0 to 0.111.0
2023-03-11 20:02:03 +01:00
Michael Eischer 00575ecffe doc: Remove mixed pack files from index example
The example given for the format of an index shows a mixed pack file.
Mixing tree and data blobs has been deprecated for a long time. Thus,
change the pack to only contain "data" blobs.
2023-03-11 19:10:31 +01:00
greatroar cb5694d136 fuse: Report fuse.Attr.Blocks correctly
Fixes #4239.
2023-03-07 22:14:07 +01:00
dependabot[bot] 100b06d806
build(deps): bump google.golang.org/api from 0.109.0 to 0.111.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.109.0 to 0.111.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.109.0...v0.111.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-06 02:11:38 +00:00
Torben Giesselmann 667536cea4 forget: Allow neg. values in "--keep-within*" opts 2023-03-05 14:18:08 -08:00
Torben Giesselmann ba183c44c3 forget: Add test with regular and "forever" opts 2023-03-05 13:56:16 -08:00
Torben Giesselmann 32e6a438be forget: Add test for "keep all hourly snapshots" 2023-03-04 22:13:34 -08:00
Torben Giesselmann b77b0749fa forget: Treat -1 as forever for all "last n" opts 2023-03-04 19:01:37 -08:00
Torben Giesselmann 6aca7dac21 forget: Verify forget opts 2023-03-03 19:10:11 -08:00
Torben Giesselmann 8161605f1b snapshot_group: Fix typo 2023-03-03 19:07:57 -08:00
dependabot[bot] b78607c9d8
build(deps): bump github.com/klauspost/compress from 1.15.15 to 1.16.0
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.15 to 1.16.0.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.15...v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-27 02:52:10 +00:00
Niklas Wagner c1101ede19 Add jq to container image 2023-02-21 14:01:43 +01:00
Michael Eischer f646406822
Merge pull request #4182 from BenWiederhake/dev-with-cache-description
check: Document behavior of --with-cache a bit better
2023-02-19 19:41:38 +01:00
Ben Wiederhake 9888443f5c check: Document behavior of --with-cache a bit better
Also see https://forum.restic.net/t/how-to-speed-up-tiny-incremental-checks/5905
2023-02-19 17:45:47 +01:00
Michael Eischer cf6dfd6d36
Merge pull request #4212 from greatroar/snapshotfilter
cmd, restic: Refactor and fix snapshot filtering
2023-02-19 15:20:18 +01:00
greatroar a7786c67f1 cmd, restic: Refactor and fix snapshot filtering
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
2023-02-19 15:04:25 +01:00
Michael Eischer 15b7d9c80b
Merge pull request #4213 from restic/dependabot/go_modules/golang.org/x/net-0.7.0
build(deps): bump golang.org/x/net from 0.6.0 to 0.7.0
2023-02-19 12:42:01 +01:00
dependabot[bot] ee4128281e
build(deps): bump golang.org/x/net from 0.6.0 to 0.7.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-19 11:32:55 +00:00
Michael Eischer 3c8aefa0cb
Merge pull request #4205 from restic/dependabot/go_modules/golang.org/x/oauth2-0.5.0
build(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0
2023-02-19 12:31:43 +01:00
aawsome e2df73b0ac
Update doc/design.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-02-17 10:07:53 +01:00
aawsome f4329a20f6
docs: Corrections/extra information in design.rst 2023-02-16 22:41:30 +01:00
dependabot[bot] 11ebc0c5db
build(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-13 02:54:09 +00:00
Michael Eischer f137be42fe
Merge pull request #4081 from MichaelEischer/backup-group-by
Add support for `--group-by` to the backup command
2023-02-11 11:38:13 +01:00
Michael Eischer 0ce182f044 document backup --group-by 2023-02-11 00:55:11 +01:00
Alex Thomson 1b50faf03e Add changelog 2023-02-10 23:45:20 +01:00
Alex Thomson 4cbbf5d952 Fix scan_finished JSON MessageType
Corrected the scan_finished JSON MessageType from status to verbose_status
2023-02-10 23:45:20 +01:00
Michael Eischer 2885db7902 backup: add group-by option 2023-02-10 23:18:14 +01:00
Michael Eischer acb40d2b94 Refactor group-by to parse options into a struct 2023-02-10 23:18:14 +01:00
Michael Eischer fa73b50b45
Merge pull request #4198 from MichaelEischer/better-verbose-description
Make help for --verbose less confusing
2023-02-09 22:28:18 +01:00
Michael Eischer 2d700c3887 Make help for --verbose less confusing
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
2023-02-06 22:11:21 +01:00
Michael Eischer 91251f2d57
Merge pull request #4196 from restic/dependabot/github_actions/docker/build-push-action-4
build(deps): bump docker/build-push-action from 3 to 4
2023-02-06 21:54:06 +01:00
Michael Eischer 3df4ec7c61
Merge pull request #4195 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.3.1
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.3.0 to 1.3.1
2023-02-06 21:20:33 +01:00
Michael Eischer b1d3a1a5e3
Merge pull request #4194 from restic/dependabot/go_modules/google.golang.org/api-0.109.0
build(deps): bump google.golang.org/api from 0.108.0 to 0.109.0
2023-02-06 21:17:57 +01:00
dependabot[bot] 4f31c2699d
build(deps): bump docker/build-push-action from 3 to 4
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3 to 4.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:13:17 +00:00
dependabot[bot] 96b1ff5e38
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.3.0...sdk/azcore/v1.3.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:07:34 +00:00
dependabot[bot] 6b5ffce9dc
build(deps): bump google.golang.org/api from 0.108.0 to 0.109.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.108.0 to 0.109.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.108.0...v0.109.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 02:06:57 +00:00
Michael Eischer 590eb9efd7
Merge pull request #4180 from MichaelEischer/add-riscv
Add linux/riscv64 builds
2023-01-30 22:57:39 +01:00
Michael Eischer 55c4ca66f7 add linux/riscv64 builds 2023-01-30 22:24:12 +01:00
Michael Eischer 56ad761b19
Merge pull request #4177 from greatroar/loadunpacked
repository: Reuse buffers in Repository.LoadUnpacked
2023-01-30 22:12:13 +01:00
greatroar d129baba7a repository: Reuse buffers in Repository.LoadUnpacked
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
2023-01-30 22:01:01 +01:00
Alexander Neumann febb32b5b4 Set development version for 0.15.1 2023-01-30 20:43:52 +01:00
Alexander Neumann 7d4b7ad9cb Add version for 0.15.1 2023-01-30 20:43:47 +01:00
Alexander Neumann a883bb6596 Generate CHANGELOG.md for 0.15.1 2023-01-30 20:43:15 +01:00
Alexander Neumann 91acef90b2 Prepare changelog for 0.15.1 2023-01-30 20:43:15 +01:00
Michael Eischer b2b7727b31
Merge pull request #4178 from rawtaz/changelogs
Polish changelogs
2023-01-27 22:46:46 +01:00
Leo R. Lundgren 0e4c9a5421 Polish changelogs 2023-01-27 22:27:57 +01:00
Michael Eischer 49fa8fe6dd
Merge pull request #4175 from fergus-dall/deadlock-fix
Fix deadlock in Lock.Stale
2023-01-26 22:45:13 +01:00
Michael Eischer 12f167ee79
Merge pull request #4167 from aneesh-n/progress-percent-eta-fix
ui/backup: Fix percent and eta in backup progress
2023-01-26 22:42:10 +01:00
Michael Eischer bb018fbc3e
Merge pull request #4163 from MichaelEischer/fix-windows-self-upgrade
self-upgrade: Fix handling of `--output` on windows
2023-01-26 22:40:36 +01:00
Michael Eischer 3b24e0ac55
Merge pull request #4154 from MichaelEischer/cleanup-prune-tests
deduplicate prune tests a bit
2023-01-26 22:39:15 +01:00
Fergus Dall 04da31af2b Fix deadlock in Lock.Stale
With debug logging enabled this method would take a lock and then
format the lock as a string. Since PR #4022 landed the string
formatting method has also taken the lock, so this deadlocks.

Instead just record the lock ID, as is done elsewhere.
2023-01-27 03:21:37 +11:00
Alexander Neumann 65923e9c26
Merge pull request #4152 from MichaelEischer/invalid-locks
Improve handling of invalid locks
2023-01-25 08:19:53 +01:00
Michael Eischer b903081804
Merge pull request #4169 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.15
build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
2023-01-23 22:35:52 +01:00
Michael Eischer beb1e872cc
Merge pull request #4170 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.29.0
build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
2023-01-23 22:25:09 +01:00
Michael Eischer db350c0430
Merge pull request #4171 from restic/dependabot/go_modules/google.golang.org/api-0.108.0
build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
2023-01-23 22:23:51 +01:00
dependabot[bot] 716a5dd20d
build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.107.0 to 0.108.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.107.0...v0.108.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:51 +00:00
dependabot[bot] dbd07ade98
build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.1 to 1.29.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.28.1...spanner/v1.29.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:33 +00:00
dependabot[bot] 7adf1e5d37
build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.14 to 1.15.15.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.14...v1.15.15)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:15 +00:00
Michael Eischer 8f94eb5420 add changelog for less strict lock handling 2023-01-22 15:54:07 +01:00
Michael Eischer 8aaba83719 add changelog for self-upgrade bug on windows 2023-01-22 15:39:42 +01:00
Michael Eischer e16a6d4c50 self-update: add basic test for extractToFile 2023-01-22 15:39:42 +01:00
Michael Eischer 34e67e3510 self-update: Fix handling of `--output` on windows
The code always assumed that the upgrade happens in place. Thus writing
the upgrade to a separate file fails, when trying to remove the file
stored at that location.
2023-01-22 15:39:42 +01:00
Michael Eischer c527c05590 add changelog for missing backup ETA 2023-01-22 15:32:55 +01:00
Aneesh Nireshwalia ed23edeb62 ui/backup: Fix percent and eta in backup progress
Added missing call to scanFinished=true.
This was causing the percent and eta to never get
printed for backup progress even after the scan was finished.
2023-01-21 13:25:08 -07:00
Michael Eischer 0f398b82e3
Merge pull request #4145 from greatroar/index-encode
index: Optimize generatePackList
2023-01-21 01:01:12 +01:00
greatroar 99755c634b index: Optimize generatePackList
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
2023-01-14 20:41:07 +01:00
Michael Eischer f5f13f6648
Merge pull request #4153 from MichaelEischer/fix-quiet-prune
prune: Don't show "packs processed" for quiet runs
2023-01-14 20:11:34 +01:00
Michael Eischer 00216d54a1 deduplicate prune tests a bit 2023-01-14 19:05:45 +01:00
Michael Eischer 1f3f68b2c0 prune: Don't show "packs processed" for quiet runs 2023-01-14 18:48:32 +01:00
Michael Eischer 57acc769b4 lock: Ignore empty lock files
These may be left behind by backends which do not guarantee atomic
uploads.
2023-01-14 18:15:46 +01:00
Michael Eischer 20ad14e362 lock: add help message how to recover from invalid locks 2023-01-14 18:04:22 +01:00
Michael Eischer c995b5be52 lock: cleanup error message
The error message is now `Fatal: unable to create lock in backend:
[...]` instead of `unable to create lock in backend: Fatal: [...]`.
2023-01-14 17:57:02 +01:00
Michael Eischer 1adf28a2b5 repository: properly return invalid data error in LoadUnpacked
The retry backend does not return the original error, if its execution
is interrupted by canceling the context. Thus, we have to manually
ensure that the invalid data error gets returned.

Additionally, use the retry backend for some of the repository tests, as
this is the configuration which will be used by restic.
2023-01-14 17:57:02 +01:00
Michael Eischer 6d9675c323 repository: cleanup error message on invalid data
The retry printed the filename twice:
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: load(<lock/04804cba82>): invalid data returned
```
now the warning has changed to
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: invalid data returned
```
2023-01-14 17:57:02 +01:00
Michael Eischer 551b31ce3c
Merge pull request #4146 from MichaelEischer/update-blazer
Update blazer to fix `b2_download_file_by_name: 404`
2023-01-14 15:23:32 +01:00
Michael Eischer ec99507e4c update blazer
This removes a stray warning, that was printed when checking the status
of non-existent files.
2023-01-14 14:56:25 +01:00
Michael Eischer 5f97f534b1
Merge pull request #4150 from restic/dependabot/go_modules/google.golang.org/api-0.107.0
build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
2023-01-14 14:55:15 +01:00
Michael Eischer ed11bbd0e2
Merge pull request #4151 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.47
build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
2023-01-14 14:50:01 +01:00
dependabot[bot] 5bb9cb056d
build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.46 to 7.0.47.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.46...v7.0.47)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:58 +00:00
dependabot[bot] cd9bd22563
build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.106.0 to 0.107.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.106.0...v0.107.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:42 +00:00
Michael Eischer ecc826ef7d
Merge pull request #4149 from MichaelEischer/more-frequent-dependabot
CI: let dependabot check for updates weekly
2023-01-14 14:37:30 +01:00
Michael Eischer fb43cbab49
Merge pull request #4129 from greatroar/cleanup
cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser
2023-01-14 12:17:25 +01:00
Michael Eischer 41d31b1e27
Merge pull request #4116 from MichaelEischer/extract-progress-updater
ui/progress: Extract progress updater
2023-01-14 12:07:26 +01:00
Michael Eischer f6ea5c5865 CI: let dependabot check for updates weekly 2023-01-14 12:02:42 +01:00
Michael Eischer 4a7a6b06af ui/backup: Use progress.Updater for progress updates 2023-01-14 01:20:43 +01:00
Michael Eischer e499bbe3ae progress: extract progress updating into Updater struct
This allows reusing the code to create periodic progress updates.
2023-01-14 01:13:08 +01:00
Michael Eischer 52682b1c7b
Merge pull request #4111 from MichaelEischer/extract-stdio-wrapper
backup: extract StdioWrapper from ProgressPrinters
2023-01-14 01:12:23 +01:00
Michael Eischer c15b4bceae backup: extract StdioWrapper from ProgressPrinters
The StdioWrapper is not used at all by the ProgressPrinters. It is
called a bit earlier than previously. However, as the password prompt
directly accessed stdin/stdout this doesn't cause problems.
2023-01-14 00:58:13 +01:00
Alexander Neumann 74348be3fa Set development version for 0.15.0 2023-01-12 20:51:23 +01:00
Alexander Neumann 37d0e1fe58 Add version for 0.15.0 2023-01-12 20:51:19 +01:00
Alexander Neumann da196aa43e Update manpages and auto-completion 2023-01-12 20:51:19 +01:00
Alexander Neumann 099774c2aa Generate CHANGELOG.md for 0.15.0 2023-01-12 20:50:45 +01:00
Alexander Neumann cd2f53e3f9 Prepare changelog for 0.15.0 2023-01-12 20:50:44 +01:00
Alexander Neumann 0c5a55d1bd
Merge pull request #4142 from restic/update-deps
Upgrade dependencies
2023-01-12 20:23:25 +01:00
Michael Eischer 9ddca65f6d update dependencies 2023-01-11 23:22:10 +01:00
Michael Eischer 06fee601bc
Merge pull request #4141 from restic/doc-no-scan-correction
doc: Correct heading level for --no-scan
2023-01-11 22:01:05 +01:00
Leo R. Lundgren 1cb920cc57 doc: Correct heading level for --no-scan 2023-01-11 21:47:38 +01:00
Michael Eischer 8f53ffb921
Merge pull request #4140 from restic/doc-manual
doc: Update manual page with --no-scan
2023-01-11 21:42:42 +01:00
Michael Eischer 351cbb4f94
Merge pull request #4139 from restic/doc-no-scan
doc: Move and update documentation for --no-scan
2023-01-11 21:41:20 +01:00
Leo R. Lundgren b8b5508d15 doc: Update manual page with --no-scan 2023-01-11 02:27:43 +01:00
Leo R. Lundgren c5542ddcd2 doc: Move and update documentation for --no-scan 2023-01-11 00:09:24 +01:00
rawtaz dffb8e0c14
Merge pull request #4138 from MichaelEischer/doc-sparse
doc: add description for restore --sparse
2023-01-10 23:39:51 +01:00
Michael Eischer b151fa498a doc: add description for restore --sparse 2023-01-10 23:27:42 +01:00
Michael Eischer c354b55e62
Merge pull request #4136 from restic/doc-small-files
doc: Clarify text about tuning backups for small files
2023-01-08 22:00:09 +01:00
Michael Eischer 375953a001
Merge pull request #4088 from MichaelEischer/doc-cifs-backup-source
doc: reading from CIFS can be a problem on linux
2023-01-08 21:56:48 +01:00
Leo R. Lundgren 6306797238 doc: Clarify text about tuning backups for small files 2023-01-08 21:49:55 +01:00
Michael Eischer ef9164fcbb doc: reading from CIFS can be a problem on linux 2023-01-08 21:47:34 +01:00
rawtaz e2bcfd68dd
Merge pull request #4135 from restic/changelogs
Polish changelogs
2023-01-08 20:04:24 +01:00
Leo R. Lundgren 33fb351386 Polish changelogs 2023-01-08 19:48:51 +01:00
Michael Eischer c9840da4f8
Merge pull request #4134 from MichaelEischer/fix-verbose-help-text
Correct maximum verbosity level in help message
2023-01-07 22:24:04 +01:00
Michael Eischer 732184a849 Correct maximum verbosity level in help message
The maximum for `--verbose=n` is n=2. Internally it is translated into a
scale from 0 to 3. However, the default (without verbose) is 1, thus the
verbosity level can only be increased two times.
2023-01-07 22:02:13 +01:00
Michael Eischer 24178c97e9
Merge pull request #4117 from MichaelEischer/prune-dry-run-help
prune: make it clearer when prune is used in dry-run mode
2023-01-04 23:18:53 +01:00
Michael Eischer 7a36306901 forget: Clarify log message for `--dry-run --prune` 2023-01-04 00:44:46 +01:00
Michael Eischer b404ad4eaa prune: make it clearer when prune is used in dry-run mode 2023-01-04 00:44:46 +01:00
Michael Eischer e02a10c58a
Merge pull request #4109 from MichaelEischer/fix-prune-uncompressed-accounting
prune: Fix calculation of remaining uncompressed data
2023-01-03 23:28:10 +01:00
Michael Eischer 81dc8c8d13 prune: Fix calculation of remaining uncompressed data
Only the repacking of *un*compressed packs reduces the amount of
uncompressed data. Previously the counter even overflowed for fully
compressed repositories.
2023-01-03 22:34:36 +01:00
greatroar 72922a79ed cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser 2023-01-03 19:03:36 +01:00
Michael Eischer 89a8006578
Merge pull request #4104 from philaris/fix_max_uint32_uid_gid_to_zero
in tar dump, convert uid, gid of value -1 to zero
2023-01-02 22:28:28 +01:00
Panagiotis Cheilaris 3b516d4b70 convert uid/gid -1 to 0 only in 32-bit tar dump
Only for a 32-bit build of restic, convert a uid or gid value of -1 to 0.
2022-12-30 18:12:12 +01:00
Michael Eischer 0de3b24756
Merge pull request #4110 from MichaelEischer/remove-exitf
Remove Exitf function
2022-12-29 12:07:51 +01:00
Michael Eischer 0fbff39ae8
Merge pull request #4108 from MichaelEischer/cleanup-check-output
Cleanup check output
2022-12-29 11:59:18 +01:00
Michael Eischer 68b1f30733 Remove Exitf function
Commands should use the normal shutdown path. In addition, the Exitf
function was only used by `dump` and `restore` but not any other command
which introduces the risk of inconsistent behavior.
2022-12-28 21:42:38 +01:00
Panagiotis Cheilaris a86a56cf3b fix lint issue with function name 'tarId'
See https://github.com/golang/lint/issues/89 and
https://github.com/golang/lint/issues/124
2022-12-28 18:46:58 +01:00
Panagiotis Cheilaris 050ed616ae be more explicit with uid or gid of value -1 2022-12-28 18:44:36 +01:00
Michael Eischer 8430399fce check: Partially fix garbled output
When reporting an error for a tree, the output message can overlap with
the progress bar output, e.g. `error for tree e91ef6fb:napshots`.

The fix only applies for this specific message and does not work on
Windows.
2022-12-28 17:47:27 +01:00
Michael Eischer aea96b7d86 check: Slightly improve help message
If a repository has both pack/index related warnings and errors, then
the help message is quite misleading. Reword it slightly to be more
clear.
2022-12-28 17:46:06 +01:00
Michael Eischer bcae28afb4
Merge pull request #4100 from klemensn/tag-self-update
Reinstate selfupdate tag to make builds without self-update work
2022-12-28 15:49:11 +01:00
Panagiotis Cheilaris 10fa5cde0a in tar dump, convert uid, gid of value -1 to zero 2022-12-27 16:36:04 +01:00
Klemens Nanni 61e7386384
Bugfix: Make distribution package builds without self-update work 2022-12-26 21:52:24 +04:00
Klemens Nanni 94f6e7d4a6
Reinstate selfupdate tag to make builds without self-update work
Revert what seems to be a typo introduced as part of the fix for #2041
in 2018 7d0f2eaf24.

`xbuild` does not look like a go build/tag keyword to me, I failed to
find documentation for it and using `go install -tags '!selfupdate' ...`
has no effect, i.e. self-update code is still compiled.

`+build` however works;  updating the OpenBSD port/binary package
security/restic to apply this PR works as expected:

```
	$ restic help | grep self
	$ restic self-update
	unknown command "self-update" for "restic"
```

(Using `go:build` now as per restic's style and gofmt.)

Previously, using `restic-0.14.0p1` on OpenBSD/amd64 7.2-current would
check for a newer version and probably attempt replacing the system wide
root-owned executable (on a read-only filesystem) as unprivileged user:

```
	$ restic version
	restic 0.14.0 compiled with go1.19.2 on openbsd/amd64
	$ restic help | grep self
	  self-update   Update the restic binary
	$ restic self-update
	writing restic to /usr/local/bin/restic
	find latest release of restic at GitHub
	restic is up to date
```

(It never tried to actually write besaid path;  doing so would fail, so
the current message can be considered misleading.)
2022-12-26 21:46:22 +04:00
Michael Eischer 90fb6f70b4
Merge pull request #4089 from greatroar/errors
Clean up error handling further
2022-12-24 10:41:56 +01:00
Michael Eischer 29b8500254
Merge pull request #4090 from restic/upgrade-dependencies
Upgrade dependencies
2022-12-23 22:34:39 +01:00
Michael Eischer 705cabb304
Merge pull request #3981 from MichaelEischer/prune-uncompressed-stats
prune: report how much data must be repacked to compressed the repo
2022-12-23 22:34:04 +01:00
Michael Eischer a6f3ae5790
Merge pull request #4094 from googol42/master
remove duplicated init
2022-12-23 22:33:01 +01:00
Andreas Dominik Preikschat ea37240597 remove duplicated init
the documentation contained the `init` command twice
2022-12-20 17:24:56 +01:00
Michael Eischer bd2f6aaac3 azure: downgrade azblob dependency due to build breakages on Solaris 2022-12-17 23:35:07 +01:00
Michael Eischer 583372956b Upgrade dependencies
Nothing special has changed.
2022-12-17 15:23:11 +01:00
greatroar 1678392a6d checker: Make ErrLegacyLayout a value, not a type 2022-12-17 09:41:07 +01:00
greatroar d9002f050e backend: Don't Wrap errors from url.Parse
The messages from url.Error.Error already start with the word "parse".
2022-12-17 09:41:07 +01:00
greatroar b150dd0235 all: Replace some errors.Wrap calls by errors.WithStack
Mostly changed the ones that repeat the name of a system call, which is
already contained in os.PathError.Op. internal/fs.Reader had to be
changed to actually return such errors.
2022-12-17 09:41:07 +01:00
Michael Eischer cccc17e4e9
Merge pull request #4086 from blackpiglet/modify_access_denied_code
Fix: change error code in function isAccessDenied to AccessDenied
2022-12-16 21:55:52 +01:00
Michael Eischer 2723159ed4
Merge pull request #3931 from kjetilho/feature/optional_scanner
add --no-scan to backup command
2022-12-16 21:42:22 +01:00
Xun Jiang/Bruce Jiang ecc62c8be2 Update changelog/unreleased/issue-4085
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Xun Jiang cc5325d22b Fix: change error code in function isAccessDenied to AccessDenied
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Michael Eischer da0e45cf40
Merge pull request #4083 from greatroar/cleanup
repository: Remove empty cleanup functions in tests
2022-12-16 21:39:30 +01:00
Kjetil Torgrim Homme 14aa6f2a00 add --disable-scanner to backup command
The scanner process has only cosmetic effect for the progress printer,
and can be disabled without impacting functionality when the user does
not need an estimate of completion.

In many cases the scanner process can provide beneficial priming of
the file system cache, so as general advice it should not be disabled.
However, tests have shown that backup of NFS and fuse based filesystems,
where stat(2) is relatively expensive, can be significantly faster
without the scanner.
2022-12-16 21:29:59 +01:00
Michael Eischer 7bdb985dde
Merge pull request #4079 from MichaelEischer/rewrite-set-original
rewrite: Always set the Original field in a rewritten snapshot
2022-12-13 22:56:20 +01:00
Michael Eischer 1bfe98bdc0
Merge pull request #2398 from DanielG/b2-hide-file
b2: Fallback to b2_hide_file when delete returns unauthorized
2022-12-13 22:52:23 +01:00
Michael Eischer 1c071a462e
Merge pull request #4084 from ekarlso/azure-stat-fix
fix: Make create not error out when ContainerNotFound
2022-12-13 22:49:05 +01:00
Michael Eischer 25d22d5241
Merge pull request #4082 from MichaelEischer/unbuffered-logger-for-testing
Don't buffer the golang `log` package output when running tests
2022-12-13 22:45:50 +01:00
Endre Karlson 7dd33c0ecc azure: Make create not error out when ContainerNotFound 2022-12-11 22:57:23 +01:00
greatroar c0b5ec55ab repository: Remove empty cleanup functions in tests
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
Michael Eischer 2e3d4640be Don't buffer the golang log output when running tests 2022-12-10 16:08:27 +01:00
Michael Eischer 38b2e9b42c rewrite: Always set the Original field in a rewritten snapshot
The Original field is meant to remember the original snapshot id if e.g.
changing its tags. It was only set by the `rewrite` command if it was
not set previously. However, a rewritten snapshot is potentially rather
different from the original snapshot. Thus just always set the Original
field. This also makes it easier to later on detect and potentially
remove the original snapshots.
2022-12-10 12:47:00 +01:00
Michael Eischer 049a105ba5
Merge pull request #4077 from greatroar/cleanup
test: Use testing.T.Cleanup to remove tempdirs
2022-12-09 22:17:46 +01:00
Michael Eischer 4b98b5562d
Merge pull request #4075 from greatroar/sftp-enospc
sftp: Fix ENOSPC check
2022-12-09 22:00:13 +01:00
greatroar f90bf84ba7 test: Use testing.T.Cleanup to remove tempdirs 2022-12-09 14:23:55 +01:00
greatroar 83d23b3ae8 Changelog for ENOSPC handling bug 2022-12-09 08:50:30 +01:00
Michael Eischer eae7366563
Merge pull request #4028 from ekarlso/use-az-blob-sdk
Switch to azblob sdk
2022-12-07 21:58:03 +01:00
Endre Karlson 25648e2501 azure: Switch to azblob sdk 2022-12-07 21:46:07 +01:00
greatroar 62520bb7b4 sftp: Fix ENOSPC check
We now check for space that is not reserved for the root user on the
remote, and the check is no longer in a defer block because it wouldn't
fire. Some change in the surrounding code may have led the deferred
function to capture the wrong err variable.

Fixes #3336.
2022-12-07 21:06:46 +01:00
rawtaz 4ba31df08f
Merge pull request #4074 from greatroar/lobaro-docker
doc: Remove ref to Lobaro's Docker image
2022-12-04 18:00:08 +01:00
greatroar 5efcbe143c doc: Remove ref to Lobaro's Docker image
It hasn't been updated for a while and has restic 0.12.0. Fixes #4002.
2022-12-04 16:20:42 +01:00
Michael Eischer 0df585dd99
Merge pull request #4066 from sedlund/fix#4033
fix#4033 cmd: copy no longer lists skipped existing snapshots by default
2022-12-03 19:22:11 +01:00
Michael Eischer 223da7344e
Merge pull request #4070 from restic/fix-cloud-tests
Fix cloud tests
2022-12-03 19:21:25 +01:00
Michael Eischer 2b67862420 backend/test: check that IsNotExist actually works 2022-12-03 18:56:55 +01:00
Michael Eischer 2f934f5803 gs: check against the correct error in IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer 04d101fa94 gs/s3: remove useless os.IsNotExist check 2022-12-03 18:49:54 +01:00
Michael Eischer 579cd6dc64 azure: fix totally broken IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer 3ebdadc58f
Merge pull request #4069 from greatroar/cleanup
cache, prune, restic: Cleanup
2022-12-03 17:50:52 +01:00
Michael Eischer bc8b2455b9
Merge pull request #4064 from MichaelEischer/flaky-abort-early-on-error
archiver: Fix flaky TestArchiverAbortEarlyOnError
2022-12-03 17:43:44 +01:00
Michael Eischer 60c6a09324
Merge pull request #4065 from MichaelEischer/flaky-rclone-failed-start
rclone: treat "file already closed" as command startup error
2022-12-03 17:42:56 +01:00
Michael Eischer 8bf6b2b80d
Merge pull request #4067 from MichaelEischer/remove-backend-test-method
Remove `Test()` method from Backend
2022-12-03 17:40:55 +01:00
Michael Eischer 78ea69082a
Merge pull request #4068 from MichaelEischer/debug-lock-refresh-test
Add more debug logging to `TestLockSuccessfulRefresh`
2022-12-03 17:38:21 +01:00
Scott Edlund cbe73ace3f
Update changelog/unreleased/issue-4033
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-12-03 20:07:37 +08:00
greatroar 63bed34608 restic: Clean up restic.IDs type
IDs.Less can be rewritten as

	string(list[i][:]) < string(list[j][:])

Note that this does not copy the ID's.

The Uniq method was no longer used.

The String method has been reimplemented without first copying into a
separate slice of a custom type.
2022-12-03 12:38:20 +01:00
greatroar 0c749dd358 prune: Pass fewer options around 2022-12-03 12:14:04 +01:00
greatroar d45a2475e1 cache: Rewrite unnecessary if-else 2022-12-03 12:13:54 +01:00
Michael Eischer 6b5d6b9f2c Add more debug logging to TestLockSuccessfulRefresh
The test fails from time to time. Add some more logging to hopefully get
an idea where things go wrong.
2022-12-03 12:05:38 +01:00
Michael Eischer 648edeca40 retry: Do not retry Stat() if file does not exist
In non test/debug code, Stat() is used exclusively to check whether a
file exists. Thus, do not retry if a file is reported as not existing.
2022-12-03 11:42:48 +01:00
Michael Eischer 40ac678252 backend: remove Test method
The Test method was only used in exactly one place, namely when trying
to create a new repository it was used to check whether a config file
already exists.

Use a combination of Stat() and IsNotExist() instead.
2022-12-03 11:28:10 +01:00
sedlund 06ee0339aa fix#4033 cmd: copy no longer lists skipped existing snapshots by default 2022-12-03 09:55:39 +08:00
Michael Eischer 57d8eedb88
Merge pull request #4020 from greatroar/fuse-inode
fuse: Better inode generation
2022-12-02 22:28:15 +01:00
Michael Eischer ca1803cacb
Merge pull request #4063 from MichaelEischer/replace-ioutil-usage
Replace ioutil usage
2022-12-02 21:49:40 +01:00
Michael Eischer 0af89a5738
Merge pull request #3132 from metalsp0rk/init-json
Init command JSON output
2022-12-02 21:49:22 +01:00
Michael Eischer 364a396fd6 init: use standard name `message_type` to distinguish JSON messages 2022-12-02 21:33:03 +01:00
Michael Eischer 9a9f559806 init: cleanup json print code 2022-12-02 21:33:03 +01:00
Kyle Brennan 933c9af328 create changelog entry for issue-3124 and pull-3132 2022-12-02 21:32:30 +01:00
Kyle Brennan a6ae79b39e support json output for init command 2022-12-02 21:32:30 +01:00
Michael Eischer f3d964a8c1 rclone: treat "file already closed" as command startup error
Since #3940 the rclone backend returns the commands exit code if it
fails to start. The list of expected errors was missing the "file
already closed"-error which can occur if the http test request first
learns about the closed pipe to rclone before noticing the canceled
context.

Go internally makes sure that a file descriptor is unusable once it was
closed, thus this cannot have unintended side effects (like accidentally
reading from the wrong file due to a reused file descriptor).
2022-12-02 20:46:02 +01:00
Michael Eischer a9972dbe7d archiver: Fix flaky TestArchiverAbortEarlyOnError
Saving the blobs of a file by now happens asynchronously to the
processing in the FileSaver. Thus we have to account for the blobs
queued for saving.
2022-12-02 20:07:34 +01:00
Michael Eischer f755233210 Replace usages of ioutil.ReadDir
This changes the return type to []fs.DirEntry. However, as we only use
the filenames anyways, this doesn't make a difference.
2022-12-02 19:54:27 +01:00
Michael Eischer fa20a78bb6
Merge pull request #4056 from greatroar/cleanup
backend, fs, options: Minor cleanup
2022-12-02 19:44:54 +01:00
Michael Eischer ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00
greatroar 65612d797c backend, options: Prefer strings.Cut to SplitN
Also realigned the various "split into host🪣prefix"
implementations.
2022-12-02 19:19:14 +01:00
Michael Eischer 2d5e28e777
Merge pull request #4059 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.45
build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
2022-12-01 21:20:57 +01:00
dependabot[bot] 4fefa2ade2
build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.44 to 7.0.45.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.44...v7.0.45)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-01 01:03:42 +00:00
Michael Eischer 3c5d1eabe9
Merge pull request #4051 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.12
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
2022-11-28 21:47:55 +01:00
Michael Eischer bec391ee26
Merge pull request #4053 from greatroar/xattr
Upgrade pkg/xattr to version with Solaris FIFO fix
2022-11-28 21:06:19 +01:00
greatroar daafcaf380 Upgrade pkg/xattr to version with Solaris FIFO fix
This version doesn't have a release tag yet, but it's 0.4.9 + one patch.

Fixes #4003.
2022-11-28 20:43:51 +01:00
Alexander Neumann 1d7e7fcd6b
Merge pull request #4049 from MichaelEischer/fix-rewrite-docs
rewrite: fix link anchors in documentation
2022-11-28 19:39:38 +01:00
dependabot[bot] 57d59c71e3
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.15.12.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.15.12)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-28 18:39:35 +00:00
Alexander Neumann bb83c78ee5
Merge pull request #4047 from MichaelEischer/clean-ci-configuration
Cleanup CI configuration
2022-11-28 19:38:33 +01:00
greatroar 60aa87bbab fs: Remove explicit type check in extendedStat
Without comma-ok, the runtime inserts the same check with a similar
enough panic message:

    interface conversion: interface {} is nil, not *syscall.Stat_t
2022-11-27 19:58:06 +01:00
Michael Eischer 34609bca0e
Merge pull request #4050 from greatroar/lruv2
bloblru: Upgrade to hashicorp/golang-lru/v2
2022-11-27 17:37:14 +01:00
greatroar e5d597fd22 bloblru: Upgrade to hashicorp/golang-lru/v2
The new genericized LRU cache no longer needs to have the IDs separately
allocated:

name   old time/op    new time/op    delta
Add-8     494ns ± 2%     388ns ± 2%  -21.46%  (p=0.000 n=10+9)

name   old alloc/op   new alloc/op   delta
Add-8      176B ± 0%      152B ± 0%  -13.64%  (p=0.000 n=10+10)

name   old allocs/op  new allocs/op  delta
Add-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2022-11-27 17:18:13 +01:00
Michael Eischer 0eddc89e98 doc: design.rst: Fix highlighting for index snippet
JSON does not support comments. As JSON is a subset of Javascript, use
the latter instead.
2022-11-27 17:01:27 +01:00
Michael Eischer 41b0f1d43a doc: fix link to amazon s3 section 2022-11-27 17:01:22 +01:00
Michael Eischer 6a793db9ca rewrite: fix link anchors in documentation 2022-11-27 16:38:10 +01:00
Michael Eischer 05cebc1c4b
Merge pull request #4044 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.28.0
build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
2022-11-27 15:23:06 +01:00
Michael Eischer ce39727846
Merge pull request #4036 from restic/dependabot/go_modules/github.com/pkg/profile-1.7.0
build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
2022-11-27 15:22:16 +01:00
Michael Eischer 9aa06ce959 CI: remove option to configure command used to install go tools
With the minimum required go version of 1.18, we always use `go
install`.
2022-11-27 15:07:29 +01:00
Michael Eischer 5968971313 CI: remove dependabot ignore for bazil.org/fuse
We've switched to a fork of the original library, thus the ignore is no
longer necessary.
2022-11-27 15:06:30 +01:00
dependabot[bot] 95374767de
build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
Bumps [github.com/pkg/profile](https://github.com/pkg/profile) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/pkg/profile/releases)
- [Commits](https://github.com/pkg/profile/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/pkg/profile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:02:32 +00:00
dependabot[bot] c100a62ebf
build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.25.0 to 1.28.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.25.0...spanner/v1.28.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:01:36 +00:00
Michael Eischer c41a1b66e1
Merge pull request #4037 from restic/dependabot/go_modules/google.golang.org/api-0.103.0
build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
2022-11-27 15:00:16 +01:00
dependabot[bot] 705aed0ecb
build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.93.0 to 0.103.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.93.0...v0.103.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 13:45:08 +00:00
Michael Eischer 28d6de648c
Merge pull request #4040 from restic/dependabot/go_modules/github.com/spf13/cobra-1.6.1
build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
2022-11-27 14:44:25 +01:00
Michael Eischer bb40b55d1c
Merge pull request #4038 from restic/dependabot/go_modules/github.com/cenkalti/backoff/v4-4.2.0
build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
2022-11-27 14:13:39 +01:00
dependabot[bot] a24c1e99a6
build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.1.3 to 4.2.0.
- [Release notes](https://github.com/cenkalti/backoff/releases)
- [Commits](https://github.com/cenkalti/backoff/compare/v4.1.3...v4.2.0)

---
updated-dependencies:
- dependency-name: github.com/cenkalti/backoff/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:41 +00:00
dependabot[bot] fd56ead4a8
build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:23 +00:00
Michael Eischer cc679c6494
Merge pull request #4041 from MichaelEischer/require-go-1.18
Require go 1.18
2022-11-27 13:56:43 +01:00
greatroar c9c7671c58 fuse: Clean up inode generation 2022-11-27 13:53:42 +01:00
Michael Eischer 530f129a39 rest: remove workaround for content-length handling bug 2022-11-27 13:18:44 +01:00
Michael Eischer 8ad231bcad bump version numbers in instructions to reproduce binaries 2022-11-27 13:18:44 +01:00
Michael Eischer a1eb923876 remove no longer necessary conditional compiles 2022-11-27 13:18:44 +01:00
Michael Eischer bcdfc2a8ea CI: allow dependabot update of oauth2
Our minimum go version is new enough to allow updating the library.
2022-11-27 13:18:44 +01:00
Michael Eischer 686b0b2a3e update the minimum required go version to 1.18 2022-11-27 13:18:43 +01:00
Michael Eischer 69a2e81bd3
Merge pull request #4039 from restic/dependabot/go_modules/github.com/google/go-cmp-0.5.9
build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
2022-11-26 17:39:27 +01:00
dependabot[bot] 278e93f738
build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-26 11:35:05 +00:00
Michael Eischer 747d2ecd7b
Merge pull request #4042 from restic/skip-cloud-test-for-dependabot
CI: skip cloud tests for dependabot pull requests
2022-11-26 12:34:16 +01:00
Michael Eischer 98c6ca9d8f CI: skip cloud tests for dependabot pull requests 2022-11-26 12:23:55 +01:00
Michael Eischer 9113b2620f
Merge pull request #4024 from MichaelEischer/macos-fuse
mount: switch to anacrolix fork of bazil/fuse
2022-11-26 12:15:14 +01:00
Michael Eischer f115d64634
Merge pull request #4022 from MichaelEischer/race-checker
CI: Run the golang race checker
2022-11-26 12:13:50 +01:00
Michael Eischer 923c06cea0
Merge pull request #4025 from MichaelEischer/update-minio
Update minio library to add `credential_process` support
2022-11-25 23:21:57 +01:00
Michael Eischer f4d3ed77c4 update minio library 2022-11-25 22:36:21 +01:00
greatroar 189e0fe5a9 fuse: Better inode generation
Hard links to the same file now get the same inode within the FUSE
mount. Also, inode generation is faster and, more importantly, no longer
allocates.

Benchmarked on Linux/amd64. Old means the benchmark with

        sink = fs.GenerateDynamicInode(1, sub.node.Name)

instead of calling inodeFromNode. Results:

name                   old time/op    new time/op    delta
Inode/no_hard_links-8     137ns ± 4%      34ns ± 1%   -75.20%  (p=0.000 n=10+10)
Inode/hard_link-8        33.6ns ± 1%     9.5ns ± 0%   -71.82%  (p=0.000 n=9+8)

name                   old alloc/op   new alloc/op   delta
Inode/no_hard_links-8     48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8         0.00B          0.00B           ~     (all equal)

name                   old allocs/op  new allocs/op  delta
Inode/no_hard_links-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8          0.00           0.00           ~     (all equal)
2022-11-16 08:35:01 +01:00
Michael Eischer 32ffcd86a2
Merge pull request #3993 from MichaelEischer/backup-json-full-snapshot-id
backup: print full snapshot id in JSON summary
2022-11-12 20:42:35 +01:00
Michael Eischer f032a9d0ad prune: report how much data must be repacked to compressed the repo
prune now reports the remaining size of pack files containing
uncompressed blobs. The displayed value is suitable for use with `--max-repack-size`.
2022-11-12 20:20:23 +01:00
Michael Eischer 66818a8f98
Merge pull request #3980 from MichaelEischer/prune-compression-stats
prune: Correctly count used/duplicate blobs for partially compressed repos
2022-11-12 20:06:56 +01:00
Michael Eischer 4b5234924b
Merge pull request #2875 from fgma/issue2699
issue2699: restore symlinks on windows when run as admin user
2022-11-12 20:06:45 +01:00
Michael Eischer 726a1969cd
Merge pull request #2731 from dionorgua/rewrite-snapshot
Implement 'rewrite' command to exclude files from existing snapshots
2022-11-12 20:06:35 +01:00
Michael Eischer bb0fa76c06 Cleanup exclude pattern collection 2022-11-12 19:55:22 +01:00
Michael Eischer 537cfe2e4c rewrite: Fix check that an exclude pattern was passed
The old check did not consider files containing case insensitive
excludes. The check is now implemented as a function of the
excludePatternOptions struct to improve cohesion.
2022-11-12 19:55:22 +01:00
Leo R. Lundgren f175da2756 rewrite: Polish documentation 2022-11-12 19:55:22 +01:00
Leo R. Lundgren f86ef4d3dd rewrite: Polish code and add missing messages 2022-11-12 19:55:22 +01:00
Leo R. Lundgren c15bedccc0 rewrite: Revert unrelated documentation change 2022-11-12 19:55:22 +01:00
Michael Eischer f88acd4503 rewrite: Fail if a tree contains an unknown field
In principle, the JSON format of Tree objects is extensible without
requiring a format change. In order to not loose information just play
it safe and reject rewriting trees for which we could loose data.
2022-11-12 19:55:22 +01:00
Michael Eischer 11b8c3a158 rewrite: add documentation 2022-11-12 19:55:22 +01:00
Michael Eischer ec0c91e233 rewrite: Add tests for further ways to use the command 2022-11-12 19:55:22 +01:00
Michael Eischer 0224e276ec walker: Add tests for FilterTree 2022-11-12 19:55:22 +01:00
Michael Eischer 73f54cc5ea rewrite: rename --inplace to --forget 2022-11-12 19:55:22 +01:00
Michael Eischer a47d9a1c40 rewrite: use unified snapshot filter options 2022-11-12 19:55:22 +01:00
Michael Eischer b044649118 rewrite: add minimal test 2022-11-12 19:55:22 +01:00
Michael Eischer 375a3db64d rewrite: non-exclusive lock if snapshots are only added 2022-11-12 19:55:22 +01:00
Michael Eischer 327f418a9c rewrite: cleanup err handling and output 2022-11-12 19:55:22 +01:00
Michael Eischer ad14d6e4ac rewrite: use SelectByName like in the backup command 2022-11-12 19:55:22 +01:00
Michael Eischer 7ebaf6e899 rewrite: start repository uploader goroutines 2022-11-12 19:55:22 +01:00
Michael Eischer 559acea0d8 unify exclude pattern options 2022-11-12 19:55:22 +01:00
Michael Eischer 4cace1ffe9 unify exclude patterns with backup command 2022-11-12 19:55:22 +01:00
Michael Eischer 2b69a1c53b rewrite: filter all snapshots if none are specified 2022-11-12 19:55:22 +01:00
Michael Eischer f6339b88af rewrite: extract tree filtering 2022-11-12 19:55:22 +01:00
Michael Eischer c0f7ba2388 rewrite: simplify dryrun 2022-11-12 19:55:22 +01:00
Michael Eischer 4d6ab83019 rewrite: use treejsonbuilder 2022-11-12 19:55:22 +01:00
Michael Eischer 82592b88b5 rewrite: address most review comments 2022-11-12 19:55:22 +01:00
Michael Eischer b922774343 rewrite: fix compilation 2022-11-12 19:55:22 +01:00
Dmitry Nezhevenko dc29709742 Implement 'rewrite' command to exclude files from existing snapshots 2022-11-12 19:55:22 +01:00
Michael Eischer 220eaee76b mount: switch to anacrolix fork of bazil/fuse
The anacrolix fork contains the latest changes from bazil/fuse and
additionally provides support for recent versions of macFUSE.
2022-11-12 19:22:31 +01:00
Michael Eischer 6fa45d0d39
Merge pull request #4011 from greatroar/backup-stdin-password
cmd: Don't read password from stdin for backup --stdin
2022-11-12 19:18:56 +01:00
Michael Eischer bbd180ae21
Merge pull request #4017 from Rajpratik71/Rajpratik71-patch-1
feat: dependabot workflow automation for updating dependency
2022-11-12 15:48:48 +01:00
Pratik Raj bef1064b8e
chore: ignore upgrade for 'bazil/fuse' and 'golang.org/x/oauth2' 2022-11-12 19:39:16 +05:30
Michael Eischer 7b4fe7bad5
Merge pull request #4021 from greatroar/mac-fsync
backend/local: Ignore ENOTTY for fsync on Mac
2022-11-11 23:10:37 +01:00
greatroar 348e966daa backend/local: Ignore ENOTTY for fsync on Mac
Fixes #4016.
2022-11-11 22:51:51 +01:00
Michael Eischer 0e5fe4c6ab CI: run golang race checker 2022-11-11 22:15:22 +01:00
Michael Eischer 13fbc96ed3 lock: Synchronize `Refresh()` and `Stale()`
The lock test creates a lock and checks that it is not stale. However,
it is possible that the lock is refreshed concurrently, which updates
the lock timestamp. Checking the timestamp in `Stale()` without
synchronization results in a data race. Thus add a lock to prevent
concurrent accesses.
2022-11-11 21:52:53 +01:00
Michael Eischer e1ba7ab684 lock: Don't copy the lock when checking for process existence
The lock test creates a lock and checks that it is not stale. This also
tests whether the corresponding process still exists. However, it is
possible that the lock is refreshed concurrently, which updates the lock
timestamp. Calling `processExists()` with a value receiver, however,
creates an unsynchronized copy of this field. Thus call the method using
a pointer receiver.
2022-11-11 21:45:55 +01:00
Michael Eischer dc060356c2 mount: only start next test after mount command cleanup is complete
The test did not wait for the mount command to fully shutdown all
running goroutines. This caused the go race detector to report a data
race related to lock refreshes.

==================
WARNING: DATA RACE
Write at 0x0000021bdfdb by goroutine 667:
  github.com/restic/restic/internal/backend/retry.TestFastRetries()
      /restic/restic/internal/backend/retry/testing.go:7 +0x18f
  github.com/restic/restic/cmd/restic.withTestEnvironment()
      /restic/restic/cmd/restic/integration_helpers_test.go:175 +0x183
  github.com/restic/restic/cmd/restic.TestMountSameTimestamps()
      /restic/restic/cmd/restic/integration_fuse_test.go:202 +0xac
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /usr/lib/go/src/testing/testing.go:1493 +0x47

Previous read at 0x0000021bdfdb by goroutine 609:
  github.com/restic/restic/internal/backend/retry.(*Backend).retry()
      /restic/restic/internal/backend/retry/backend_retry.go:72 +0x9e
  github.com/restic/restic/internal/backend/retry.(*Backend).Remove()
      /restic/restic/internal/backend/retry/backend_retry.go:149 +0x17d
  github.com/restic/restic/internal/cache.(*Backend).Remove()
      /restic/restic/internal/cache/backend.go:38 +0x11d
  github.com/restic/restic/internal/restic.(*Lock).Unlock()
      /restic/restic/internal/restic/lock.go:190 +0x249
  github.com/restic/restic/cmd/restic.refreshLocks.func1()
      /restic/restic/cmd/restic/lock.go:86 +0xae
  runtime.deferreturn()
      /usr/lib/go/src/runtime/panic.go:476 +0x32
  github.com/restic/restic/cmd/restic.lockRepository.func2()
      /restic/restic/cmd/restic/lock.go:61 +0x71

[...]

Goroutine 609 (finished) created at:
  github.com/restic/restic/cmd/restic.lockRepository()
      /restic/restic/cmd/restic/lock.go:61 +0x488
  github.com/restic/restic/cmd/restic.lockRepo()
      /restic/restic/cmd/restic/lock.go:25 +0x219
  github.com/restic/restic/cmd/restic.runMount()
      /restic/restic/cmd/restic/cmd_mount.go:126 +0x1f8
  github.com/restic/restic/cmd/restic.testRunMount()
      /restic/restic/cmd/restic/integration_fuse_test.go:61 +0x1ce
  github.com/restic/restic/cmd/restic.checkSnapshots.func1()
      /restic/restic/cmd/restic/integration_fuse_test.go:90 +0x124
==================
2022-11-11 21:43:01 +01:00
Michael Eischer 32c9667990
Merge pull request #4019 from MichaelEischer/fix-file-saver-race
archiver: Fix race condition resulting in files containing null IDs
2022-11-11 20:52:33 +01:00
Michael Eischer d268552a0a
Merge pull request #4014 from MichaelEischer/fix-debug-examine
debug: fix crash in `debug examine --reupload-blobs`
2022-11-10 20:37:32 +01:00
Michael Eischer 5756c96c9f archiver: Fix race condition resulting in files containing null IDs
In some rare cases files could be created which contain null IDs (all
zero) in their content list. This was caused by a race condition between
growing the `Content` slice and inserting the blob IDs into it. In some
cases the blob ID was written to the old slice, which a short time
afterwards was replaced with a larger copy, that did not yet contain the
blob ID.
2022-11-10 20:19:37 +01:00
Pratik Raj df614fff26
feat: dependabot workflow automation for updating dependency
Signed-off-by: Pratik Raj <Rajpratik71@gmail.com>
2022-11-10 16:02:03 +05:30
Michael Eischer 11a4bb051e debug: fix crash in `debug examine --reupload-blobs` 2022-11-09 22:13:17 +01:00
Michael Eischer 5f9ac2b165
Merge pull request #4010 from MichaelEischer/file-saver-sanity-check
archiver: Check that saved file does not have null IDs in content
2022-11-08 23:07:32 +01:00
Michael Eischer b1d1202b1d archiver: Check that saved file does not have null IDs in content
Null IDs in the file content indicate that something went wrong. Thus
fails before saving the affected file.
2022-11-08 22:57:41 +01:00
greatroar 5dceadeb72 cmd: Don't read password from stdin for backup --stdin 2022-11-06 14:55:57 +01:00
Michael Eischer 1ccab95bc4 b2: Support file hiding instead of deleting them permanently
Automatically fall back to hiding files if not authorized to permanently
delete files. This allows using restic with an append-only application
key with B2.  Thus, an attacker cannot directly delete backups with the
API key used by restic.

To use this feature create an application key without the deleteFiles
capability. It is recommended to restrict the key to just one bucket.
For example using the b2 command line tool:

    b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles

Suggested-by: Daniel Gröber <dxld@darkboxed.org>
2022-11-05 20:10:45 +01:00
Michael Eischer 24a2e5cab9
Merge pull request #4008 from MichaelEischer/tweak-lock-refresh-test
lock: Tweak timeouts for lock refresh test
2022-11-05 10:53:13 +01:00
Michael Eischer 403390479c
Merge pull request #3997 from greatroar/fuse-hash
fuse: Better check for whether snapshots changed
2022-11-05 10:52:11 +01:00
Michael Eischer d29abc1a31
Merge pull request #4007 from MichaelEischer/hide-compression-level-for-v1-repo
Only print compression level starting from repository version 2
2022-11-05 10:33:25 +01:00
greatroar c091e43b33 fuse: Better check for whether snapshots changed
We previously checked whether the set of snapshots might have changed
based only on their number, which fails when as many snapshots are
forgotten as are added. Check for the SHA-256 of their id's instead.
2022-11-05 09:32:45 +01:00
Michael Eischer aaac63da8d lock: Tweak timeouts for lock refresh test
For some reason the test fails from time to time. Increase the timeouts
to hopefully avoid this issue.
2022-11-04 22:48:18 +01:00
Michael Eischer fd4d23460f only print compression level starting from repository version 2 2022-11-04 22:40:07 +01:00
Alexander Neumann 8dd95b710e
Merge pull request #3992 from MichaelEischer/err-on-invalid-compression
Return error if RESTIC_COMPRESSION env variable is invalid
2022-11-04 19:41:34 +01:00
Alexander Neumann 783b8781a7
Merge pull request #4000 from restic/min-go-version
build: Correct checks for minimum Go version
2022-11-04 10:31:02 +01:00
Alexander Neumann 543649f2f2
Merge pull request #4001 from restic/docker-go-version
docker: Increase Go version to 1.19
2022-11-04 10:30:11 +01:00
Leo R. Lundgren 0a4cddb34d docker: Increase Go version to 1.19 2022-11-03 22:59:59 +01:00
Leo R. Lundgren 333c2c6ed4 build: Correct checks for minimum Go version 2022-11-03 22:50:07 +01:00
rawtaz 92df039e5d
Merge pull request #3996 from MichaelEischer/fix-ui-progress
backup: fix stuck status bar
2022-11-02 21:48:16 +01:00
Michael Eischer 9354262b1b backup: fix stuck status bar
The status bar got stuck once the first error was reported, the scanner
completed or some file was backed up. Either case sets a flag that the
scanner has started.

This flag is used to hide the progress bar until the flag is set. Due to
an inverted condition, the opposite happened and the status stopped
refreshing once the flag was set.

In addition, the scannerStarted flag was not set when the scanner just
reported progress information.
2022-11-02 21:31:13 +01:00
Michael Eischer 06141ce1f4 backup: print full snapshot id in JSON summary 2022-10-31 19:03:42 +01:00
Michael Eischer 59a90943bb
Merge pull request #3983 from greatroar/formatting
Centralize and fix formatting of bytes, percentages, durations
2022-10-31 18:52:24 +01:00
greatroar 5ab3e6276a ui: Fix FormatBytes at exactly 1024 time a unit
1024 would be displayed as "1024 bytes" instead of "1.000 KiB", etc.
2022-10-31 18:39:28 +01:00
rawtaz 4f1fae9c98
Merge pull request #3982 from MichaelEischer/show-compression-mode
Show selected compression level when opening repository
2022-10-30 21:29:42 +01:00
Michael Eischer 8fe159cc5a enable ysmlink tests for windows 2022-10-30 18:43:04 +01:00
Michael Eischer 3499c6354e
Merge pull request #3955 from MichaelEischer/async-futurefile-completion
Improve archiver performance for small files
2022-10-30 18:38:04 +01:00
Michael Eischer 144257f8bd restore symlink timestamps on windows 2022-10-30 11:04:04 +01:00
Michael Eischer c0f34af9db backup: hide files from status which are read completely but not saved
As the FileSaver is asynchronously waiting for all blobs of a file to be
stored, the number of active files is higher than the number of files
from which restic is reading concurrently. Thus to not confuse users,
only display files in the status from which restic is currently reading.
2022-10-30 10:29:12 +01:00
Michael Eischer a571fc4aa1 add changelog for faster backups with small files 2022-10-30 10:29:12 +01:00
Michael Eischer b52a8ff05c ui: Properly clear lines no longer used for status
Previously, the old status text remained until it was overwritten.
2022-10-30 10:29:12 +01:00
Michael Eischer b4de902596 archiver: Asynchronously complete FutureFile
After reading and chunking all data in a file, the FutureFile still has
to wait until the FutureBlobs are completed. This was done synchronously
which results in blocking the file saver and prevents the next file from
being read.

By replacing the FutureBlob with a callback, it becomes possible to
complete the FutureFile asynchronously.
2022-10-30 10:29:11 +01:00
Michael Eischer 47e05080a9
Merge pull request #3990 from MichaelEischer/lock-refresh-test
lock: add test to check that refreshing works
2022-10-30 10:15:44 +01:00
Michael Eischer c7ace314f6
Merge pull request #3989 from greatroar/eachbypack
More compact data structure for Index.EachByPack
2022-10-30 00:02:55 +02:00
greatroar 0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
greatroar 137f0bc944 repository: Fix benchmarkSaveAndEncrypt 2022-10-29 23:09:17 +02:00
Michael Eischer 01f0db4e56 return error if RESTIC_COMPRESSION env variable is invalid 2022-10-29 22:03:39 +02:00
Michael Eischer 7c87fb941c
Merge pull request #3986 from greatroar/counter
ui/progress: Load both values in a single Lock/Unlock
2022-10-29 21:50:55 +02:00
Michael Eischer 3b0bb02a68
Merge pull request #3977 from greatroar/progress
ui/backup: Replace channels with a mutex
2022-10-29 21:33:04 +02:00
Michael Eischer 0d260cfd82 enable symlink test on windows 2022-10-29 21:26:34 +02:00
fgma 8e5eb1090c issue2699: restore symlinks on windows when run as admin user 2022-10-29 21:19:33 +02:00
rawtaz af3f7c866f
Merge pull request #3988 from FelixBurkhard/FelixBurkhard-patch-1
Clearify what Azure account name means
2022-10-29 13:32:44 +02:00
Michael Eischer 24267e9a9d lock: add test to check that refreshing works 2022-10-29 11:26:00 +02:00
Michael Eischer 8e51e1e605 shorten 'repository opened' output 2022-10-29 11:22:00 +02:00
FelixBurkhard 575d26ec87
Clearify what Azure account name means
When reading it first it was not clear to me the 'account name' meant the name of the
Azure Storage Account and not the Azure account itself.
2022-10-29 00:27:43 +02:00
greatroar 2dafda9164 ui/progress: Load both values in a single Lock/Unlock
We always need both values, except in a test, so we don't need to lock
twice and risk scheduling in between.

Also, removed the resetting in Done. This copied a mutex, which isn't
allowed. Static analyzers tend to trip over that.
2022-10-25 07:55:24 +02:00
Michael Eischer f8910bc4ff
Merge pull request #3985 from saltsa/fix_lock_refresh
Fix bug in lock refresh monitoring
2022-10-24 22:59:18 +02:00
Joonas Aunola b06427c9f6 fix Unix to UnixNano 2022-10-23 23:40:21 +03:00
greatroar 006380199e cmd, ui: Deduplicate formatting utilities 2022-10-23 13:40:07 +02:00
greatroar 04216eb9aa ui/backup: Replace channels with a mutex
The channel-based algorithm had grown quite complicated. This is easier
to reason about and likely to be more performant with very many
CompleteBlob calls.
2022-10-23 13:28:41 +02:00
Michael Eischer 4fea3a413d show selected compression level when opening repository 2022-10-22 20:18:46 +02:00
Michael Eischer ba58ccbe07 prune: add remark about non-deterministic blob selection 2022-10-22 19:46:10 +02:00
Michael Eischer 05651d6d4f prune: Correctly count used/duplicate blobs for partially compressed repos
Counting the first occurrence of a duplicate blob as used and counting
all other as duplicates, independent of which instance of the blob is
kept, is only accurate if all copies of the blob have the same size. This
is no longer the case for a repository containing both compressed and
uncompressed blobs.

Thus for duplicated blobs first count all instances as duplicates and
then subtract the actually used instance later on.
2022-10-22 19:24:36 +02:00
Michael Eischer b57d42905c
Merge pull request #3899 from MichaelEischer/less-prune-mem
Optimize prune memory usage
2022-10-22 18:56:02 +02:00
Michael Eischer d966c52707 prune: allow gc of set of repacked blobs before index rebuild 2022-10-22 18:45:12 +02:00
Michael Eischer 1e2794fa55 add prune memory optimization changelog 2022-10-22 18:45:12 +02:00
Michael Eischer 68c9cb9c6a prune: Shrink keepBlobs set if possible
As long as only a small fraction of the data in a repository is
rewritten, the keepBlobs set will be rather small after cleaning it up.
As golang maps do not shrink their memory usage, just copy the contents
over to a new map. However, only copy the map if the cleanup removed at
least half the entries.
2022-10-22 18:45:12 +02:00
Michael Eischer c4fc5c97f9 prune: Use a single CountedBlobSet to track blobs
The set covers necessary, existing and duplicate blobs. This removes the
duplicate sets used to track whether all necessary blobs also exist.
This reduces the memory usage of prune by about 20-30%.
2022-10-22 18:45:12 +02:00
Michael Eischer b21241ec1c restic: Add CountedBlobSet type
This allows maintaining a usage counter for each blob.
2022-10-22 18:45:12 +02:00
Michael Eischer ee6688a9f6
Merge pull request #3915 from plumbeo/compression-stats
restic stats: print uncompressed size in mode raw-data
2022-10-21 22:10:29 +02:00
Michael Eischer 27634a1a68
Merge pull request #3978 from MichaelEischer/fix-negative-pattern-example
Remove misleading wildcard from negative exclude pattern example
2022-10-21 22:04:30 +02:00
Michael Eischer aa77702e49
Merge pull request #3971 from MichaelEischer/parallel-list
Unify ForAllIndex/Snapshot/Lock functions
2022-10-21 21:58:33 +02:00
Michael Eischer 6877aaa8aa
Merge pull request #3967 from MichaelEischer/archiver-extract-exclude-options
backup: extract exclude pattern options
2022-10-21 21:50:00 +02:00
Michael Eischer 2e9ee8577a
Merge pull request #3970 from MichaelEischer/split-retry-backend
Split backend package into smaller parts
2022-10-21 21:49:46 +02:00
Michael Eischer 59d46bb3f5 backup: extract exclude pattern options
This is a preparation to make the exclude options usable for the
upcoming `rewrite` command.
2022-10-21 21:40:59 +02:00
Michael Eischer 5c7a9a739a backend: Split RetryBackend into own package
The RetryBackend tests depend on the mock backend. When the Backend
interface is eventually split from the restic package, this will lead to
a dependency cycle between backend and backend/mock. Thus split the
RetryBackend into a separate package to avoid this problem.
2022-10-21 21:38:17 +02:00
Michael Eischer 32603d49c4 backend: remove unused ErrorBackend 2022-10-21 21:36:05 +02:00
Michael Eischer 8c18c65b3b backend: remove unused Paths variable 2022-10-21 21:36:05 +02:00
Michael Eischer 4ccd5e806b backend: split layout code into own subpackage 2022-10-21 21:36:05 +02:00
Michael Eischer b361284f28
Merge pull request #3979 from MichaelEischer/backup-less-time-now
backup: reduce calls to time.Now
2022-10-21 21:33:34 +02:00
Michael Eischer 738b2a0445 parallelize more List usages 2022-10-21 21:26:45 +02:00
Michael Eischer ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer 8e2695be0b
Merge pull request #3973 from MichaelEischer/speedup-integration-tests
speed-up integration tests by reducing the RetryBackend timeout
2022-10-21 21:17:35 +02:00
Michael Eischer 35d968bcde
Merge pull request #3969 from MichaelEischer/key-by-id
Port restic.Find to return IDs and identify keys by restic.ID
2022-10-21 21:15:40 +02:00
Michael Eischer 4133fee6f9
Merge pull request #3972 from MichaelEischer/fix-flaky-lock-cancel-test
lock: fix flaky TestLockFailedRefresh
2022-10-21 21:12:34 +02:00
Michael Eischer c8c8391b21
Merge pull request #3974 from greatroar/cleanup
More cleanups and a micro-optimization
2022-10-21 21:11:37 +02:00
Michael Eischer ee7c28f5e6 backup: reduce calls to time.Now
Archiver.Save queries the current time multiple times. This commit
removes one of these calls as they showed up while profiling a backup of
a nearly unchanged dataset containing 3 million files.
2022-10-21 20:55:01 +02:00
Michael Eischer 3e60d38a23 Remove misleading wildcard from negative exclude pattern example
There is no need to use a special wildcard `**` to demonstrate negative
patterns. Actually, it is both slower than the simpler variant and seems
to confuse users.
2022-10-21 20:48:45 +02:00
greatroar 9adae5521d cache: Call interface method once 2022-10-21 14:32:46 +02:00
greatroar 201e5c7e74 backup: Clean up progress reporting code 2022-10-21 13:48:30 +02:00
plumbeo a6f83e0011 Add changelog 2022-10-17 15:38:42 +02:00
plumbeo bc945d0bf0 restic stats: add more compression statistics
Calculate and display compression ratio, space saving and progress
2022-10-17 15:38:38 +02:00
greatroar b513597546 internal/restic: Make FileType a uint8 instead of a string
The string form was presumably useful before the introduction of
layouts, but right now it just makes call sequences and garbage
collection more expensive (the latter because every string contains
a pointer to be scanned).
2022-10-16 10:59:01 +02:00
greatroar 22147e1e02 all: Minor cleanups
if x { return true } return false => return x

	fmt.Sprintf("%v", x) => fmt.Sprint(x) or x.String()

The fmt.Sprintf idiom is still used in the SecretString tests, where it
serves security hardening.
2022-10-16 10:50:39 +02:00
greatroar d03460010f internal/restic: Fix ID.UnmarshalJSON, ParseID
ID.UnmarshalJSON accepted non-JSON input with ' as the string delimiter.
Also, the error message for non-hex input was less informative than it
could be and it performed too many checks.

Changed ParseID to keep the error messages consistent.
2022-10-16 10:39:52 +02:00
Michael Eischer aa39bf3cf6 backend/test: remove duplicate test
The test is identical to the tests for the mem backend.
2022-10-15 23:15:07 +02:00
Michael Eischer 28e1c4574b mem: use cheaper hash for backend 2022-10-15 23:14:33 +02:00
Michael Eischer c3400d3c55 backend: speedup RetryBackend tests 2022-10-15 23:13:44 +02:00
Michael Eischer 99547518cd lock: fix flaky TestLockFailedRefresh
The comparison of the current time and the last lock refresh were using
seconds represented as integers. As the test only waits for up to one
second, the associated number truncation can cause the test to take
longer than once second and thus to fail.

Switch to nanoseconds to avoid this problem. This also slightly speeds
up the test.
2022-10-15 22:36:32 +02:00
Michael Eischer e10420553b speed-up integration tests by reducing the RetryBackend timeout
On my machine this decreases the runtime for `./cmd/restic` from 9.5s to
6.5s.
2022-10-15 22:29:58 +02:00
Michael Eischer 367f35db27
Merge pull request #3968 from MichaelEischer/cleanup-complete-blob
backup: Remove unused filename parameter from CompleteBlob callback
2022-10-15 16:11:16 +02:00
Michael Eischer 8d62a7adb4 identify keys by ID and not name 2022-10-15 16:07:43 +02:00
Michael Eischer 02634dce7a restic: change Find to return ids
That way consumers no longer have to manually convert the returned name
to an id.
2022-10-15 16:06:54 +02:00
Michael Eischer 964977677f backup: Remove unused filename parameter from CompleteBlob callback 2022-10-15 15:21:17 +02:00
Michael Eischer 258b487d8f
Merge pull request #3951 from MichaelEischer/rework-snapshot-filter
Rework snapshot filtering
2022-10-15 14:47:47 +02:00
Michael Eischer de9bc031df add changelog for ls handling of missing snapshots 2022-10-15 13:34:50 +02:00
Michael Eischer 246d3032ae restic: Don't list snapshots if FindSnapshot gets full id 2022-10-15 13:34:34 +02:00
Michael Eischer d8c00b9726 add comment 2022-10-15 13:34:21 +02:00
Michael Eischer a3113c6097 restic: Change FindSnapshot functions to return the snapshot 2022-10-15 13:34:04 +02:00
Michael Eischer b50f48594d restic: cleanup arguments of findLatestSnapshot 2022-10-15 13:33:48 +02:00
Michael Eischer 61e827ae4f restic: hide findLatestSnapshot 2022-10-15 13:33:32 +02:00
Michael Eischer fcad5e6f5d backup: use unified FindFilteredSnapshot 2022-10-15 13:33:29 +02:00
Michael Eischer 0aa73bbd39 ls: proper error handling for non-existent snapshot
Use restic.FindFilteredSnapshot to resolve the snapshot ID. This ensures
consistent behavior for all commands using initSingleSnapshotFilterOptions.
2022-10-15 13:32:00 +02:00
Michael Eischer a81f0432e9 restic: Add unified method to resolve a single snapshot 2022-10-15 13:31:45 +02:00
Michael Eischer 95a1bb4261 restic: Rework error handling of FindFilteredSnapshots and handle snapshotIDs
FindFilteredSnapshots no longer prints errors during snapshot loading on
stderr, but instead passes the error to the callback to allow the caller
to decide on what to do.

In addition, it moves the logic to handle an explicit snapshot list from
the main package to restic.
2022-10-15 13:31:26 +02:00
Michael Eischer cff22a5f01 dump: use correct help text for filter options 2022-10-15 13:31:10 +02:00
Michael Eischer 7a6dcb4831
Merge pull request #3966 from MichaelEischer/cleanup-walker-test
walker: Convert tests to use TreeJSONBuilder
2022-10-15 11:25:11 +02:00
Michael Eischer 7cf042118f walker: Convert tests to use TreeJSONBuilder
The old code marshalled the tree blobs different than other places in
restic. The hashed tree blob did not contain a final newline character.
2022-10-15 11:04:13 +02:00
Michael Eischer cea7191995
Merge pull request #3959 from MichaelEischer/buffered-backup-progress
backup: Use buffered channels to collect backup status
2022-10-15 10:57:19 +02:00
Michael Eischer ba688aad20
Merge pull request #3961 from greatroar/cleanup
Misc. cleanup
2022-10-14 21:49:35 +02:00
Michael Eischer 9c290a8093
Merge pull request #3960 from greatroar/errors
errors: Drop WithMessage
2022-10-14 21:41:28 +02:00
greatroar 0e155fd9a6 internal/restic: Fix UID/GID parsing
The helper function uidGidInt used strconv.ParseInt instead of
ParseUint, so it silently ignored some invalid user/group IDs.

Also, improve the error message. "Invalid UID" is more informative than
having "ParseInt" twice (*strconv.NumError displays the function name).

Finally, the user.User struct can be passed by pointer to get reduce
code size.
2022-10-14 18:21:00 +02:00
greatroar e0b743c64d internal/restic: Remove unused ID.EqualString 2022-10-14 18:20:11 +02:00
greatroar 6922360179 ui/backup: Remove unused ProgressReporter type, Progress field 2022-10-14 14:36:19 +02:00
greatroar d4aadfa389 all: Drop ctxhttp
This package is no longer needed, since we can use the stdlib's
http.NewRequestWithContext.

backend/rclone already did, but it needed a different error check due to
a difference between net/http and ctxhttp.

Also, store the http.Client by value in the REST backend (changed to a
pointer when ctxhttp was introduced) and use errors.WithStack instead
of errors.Wrap where the message was no longer accurate. Errors from
http.NewRequestWithContext will start with "net/http" or "net/url", so
they're easy to identify.
2022-10-14 14:33:49 +02:00
greatroar 16849d5361 internal/archiver: Missing argument to errors.Errorf 2022-10-14 14:18:52 +02:00
greatroar 09c14f33c8 internal/checker: Pass Error.Error pointer receiver 2022-10-14 14:13:32 +02:00
greatroar feb790f497 internal/restic: Use errors.New when no formatting is needed 2022-10-14 14:07:20 +02:00
greatroar ba44666704 errors: Drop WithMessage 2022-10-14 14:06:47 +02:00
Michael Eischer 1a6160d152
Merge pull request #3880 from MichaelEischer/archiver-savedir-cleanup
archiver: Improve handling of "file xxx already present" error
2022-10-08 21:48:14 +02:00
Michael Eischer 21b1d7a880
Merge pull request #3948 from MichaelEischer/split-index
repository: split index into a separate package
2022-10-08 21:41:57 +02:00
Michael Eischer 5278ab51c8 archiver: Check that duplicates are only ignored if identical 2022-10-08 21:38:36 +02:00
Michael Eischer 403b01b788 backup: Only return a warning for duplicate directory entries
The backup command failed if a directory contains duplicate entries.
Downgrade the severity of this problem from fatal error to a warning.
This allows users to still create a backup.
2022-10-08 21:38:21 +02:00
Michael Eischer d7d7b4ab27 archiver: refactor TreeSaverTest 2022-10-08 21:29:32 +02:00
Michael Eischer 8e38c43c27 archiver: let FutureNode.Take return an error if no data is available
This ensures that we cannot accidentally store an invalid node.
2022-10-08 21:28:39 +02:00
Michael Eischer 2b88cd6eab archiver: Restructure SaveTree to work like SaveDir
SaveTree did not use the TreeSaver but rather managed the tree
collection and upload itself. This prevents using the parallelism
offered by the TreeSaver and duplicates all related code. Using the
TreeSaver can provide some speed-ups as all steps within the backup tree
now rely on FutureNodes. This can be especially relevant for backups
with large amounts of explicitly specified files.

The main difference between SaveTree and SaveDir is, that only the
former can save tree blobs in which nodes have a different name than the
actual file on disk. This is the result of resolving name conflicts
between multiple files with the same name. The filename that must be
used within the snapshot is now passed directly to
restic.NodeFromFileInfo. This ensures that a FutureNode already contains
the correct filename.
2022-10-08 21:28:39 +02:00
Michael Eischer 2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Michael Eischer 5760ba6989
Merge pull request #3949 from MichaelEischer/simplify-mixedpacks
repository: remove IsMixedPack and add replacement for checker
2022-10-08 21:14:14 +02:00
Michael Eischer 5ee25e669a
Merge pull request #3940 from MichaelEischer/better-rclone-error
Better error message if connection to rclone fails
2022-10-08 21:14:00 +02:00
Michael Eischer 5600f11696 rclone: Fix stderr handling if command exits unexpectedly
According to the documentation of exec.Cmd Wait() must not be called
before completing all reads from the pipe returned by StdErrPipe(). Thus
return a context that is canceled once rclone has exited and use that as
a precondition to calling Wait(). This should ensure that all errors
printed to stderr have been copied first.
2022-10-08 20:16:06 +02:00
Michael Eischer b8acad4da0 rclone: return rclone error instead of canceled context
When rclone fails during the connection setup this currently often
results in a context canceled error. Replace this error with the exit
code from rclone.
2022-10-08 20:15:24 +02:00
Michael Eischer d3ebec8f21 backup: Use buffered channels to collect backup status
When backing up many small files, the unbuffered channels frequently
cause the FileSaver to block when reporting progress information. Thus,
add buffers to these channels to avoid unnecessary scheduling.

As the status information is purely informational, it doesn't matter
that the status reporting shutdown is somewhat racy and could miss a few
final updates.
2022-10-08 18:20:41 +02:00
Michael Eischer f9d4e0c2af
Merge pull request #3958 from greatroar/errors
errors: Drop Cause in favor of Go 1.13 error handling
2022-10-08 18:06:35 +02:00
Michael Eischer 119e6aee01
Merge pull request #3957 from greatroar/typo
cmd: Typo in --read-concurrency description
2022-10-08 14:41:35 +02:00
greatroar 07e5c38361 errors: Drop Cause in favor of Go 1.13 error handling
The only use cases in the code were in errors.IsFatal, backend/b2,
which needs a workaround, and backend.ParseLayout. The last of these
requires all backends to implement error unwrapping in IsNotExist.
All backends except gs already did that.
2022-10-08 13:08:08 +02:00
greatroar 4eae4d3e1a cmd: Typo in --read-concurrency description 2022-10-08 11:27:39 +02:00
Michael Eischer 83cb58b4f3
Merge pull request #3956 from MichaelEischer/fix-lock-refresh
lock: Use the correct duration to check for expired locks
2022-10-07 22:58:10 +02:00
Michael Eischer 7c5d63a794 lock: Use the correct duration to check for expired locks 2022-10-07 22:39:53 +02:00
Michael Eischer 8b7c952f17
Merge pull request #3953 from keachi/typo
Fix typo
2022-10-07 22:18:32 +02:00
Michael Eischer e43d2d45f7
Merge pull request #3952 from hoelzro/master
Update copy documentation to use --from-repo option
2022-10-07 22:18:06 +02:00
Rob Hoelz 03e9a26018 Update copy documentation to use --from-repo option
Removing the last references to the deprecated --repo2 option
2022-10-07 22:00:12 +02:00
tr 43cc01d63e
doc: Fix typo 2022-10-05 21:03:14 +02:00
Michael Eischer 7112a132c3
Merge pull request #3950 from MichaelEischer/misc-cleanups
Cleanups for cmd_debug/repository and remove dead code from restic package
2022-10-03 12:46:32 +02:00
Michael Eischer 4bb5240720 repository: remove unused PrefixLength 2022-10-03 12:15:53 +02:00
Michael Eischer 999fe29976 repository: hide prepareCache 2022-10-03 12:15:53 +02:00
Michael Eischer 9197c63007 debug: use repository.ListPack wrapper 2022-10-03 12:09:08 +02:00
Michael Eischer ddcf549eba repository: remove IsMixedPack and add replacement for checker
Repositories with mixed packs are probably quite rare by now. When
loading data blobs from a mixed pack file, this will no longer trigger
caching that file. However, usually tree blobs are accessed first such
that this shouldn't make much of a difference.

The checker gets a simpler replacement.
2022-10-03 12:03:59 +02:00
Michael Eischer a61fbd287a
Merge pull request #3569 from MichaelEischer/strict-locking
Strict repository lock handling
2022-10-03 00:44:44 +02:00
Michael Eischer 6d2d297215 pass global context through cobra 2022-10-03 00:19:46 +02:00
Michael Eischer 49126796d0 lock: fix timer expiry monitoring during standby
Monotonic timers are paused during standby. Thus these timers won't fire
after waking up. Fall back to periodic polling to detect too large clock
jumps. See https://github.com/golang/go/issues/35012 for a discussion of
go timers during standby.
2022-10-03 00:19:46 +02:00
Michael Eischer 401e432e9d lock: Do not ignore invalid lock files
While searching for lock file from concurrently running restic
instances, restic ignored unreadable lock files. These can either be
in fact invalid or just be temporarily unreadable. As it is not really
possible to differentiate between both cases, just err on the side of
caution and consider the repository as already locked.

The code retries searching for other locks up to three times to smooth
out temporarily unreadable lock files.
2022-10-03 00:19:46 +02:00
Michael Eischer aeed420e1a add changelog 2022-10-03 00:19:46 +02:00
Michael Eischer 9959190e39 lock: Add integration test
The tests check that the wrapped context is properly canceled whenever
the repository is unlock or when the lock refresh fails.
2022-10-03 00:19:46 +02:00
Michael Eischer c3538b063a lock: Use repository interface instead of struct 2022-10-03 00:19:46 +02:00
Michael Eischer d92957dd78 lock: Implement strict lock expiry monitoring
Restic continued e.g. a backup task even when it failed to renew the
lock or failed to do so in time. For example if a backup client enters
standby during the backup this can allow other operations like `prune`
to run in the meantime (after calling `unlock`). After leaving standby
the backup client will continue its backup and upload indexes which
refer pack files that were removed in the meantime.

This commit introduces a goroutine explicitly monitoring for locks that
are not refreshed in time. To simplify the implementation there's now a
separate goroutine to refresh the lock and monitor for timeouts for each
lock. The monitoring goroutine would now cause the backup to fail as the
client has lost it's lock in the meantime.

The lock refresh goroutines are bound to the context used to lock the
repository initially. The context returned by `lockRepo` is also
cancelled when any of the goroutines exits. This ensures that the
context is cancelled whenever for any reason the lock is no longer
refreshed.
2022-10-03 00:19:46 +02:00
Michael Eischer 928914f821 Prepare for context bound to lock lifetime 2022-10-03 00:19:46 +02:00
Michael Eischer 985722b102 Remove ctx from globalOptions
Previously the global context was either accessed via gopts.ctx,
stored in a local variable and then used within that function or
sometimes both. This makes it very hard to follow which ctx or a wrapped
version of it reaches which method.

Thus just drop the context from the globalOptions struct and pass it
explicitly to every command line handler method.
2022-10-03 00:19:46 +02:00
Michael Eischer ab819b2344 key: Cleanup method signatures 2022-10-03 00:19:46 +02:00
Michael Eischer d0668b695d Remove unnecessary context.WithCancel calls
The gopts.ctx is cancelled when the main() method of restic exits.
2022-10-03 00:19:46 +02:00
Michael Eischer 7ce4cb7908
Merge pull request #3947 from MichaelEischer/fix-cache-verify-test
cache: Fix file descriptor leak in TestBackendRemoveBroken
2022-10-03 00:19:26 +02:00
Michael Eischer 430ab32941 cache: Fix file descriptor leak in TestBackendRemoveBroken 2022-10-03 00:06:44 +02:00
Michael Eischer e99ad39b34
Merge pull request #2750 from metalsp0rk/min-packsize
Add `backup --file-read-concurrency` flag
2022-10-02 23:11:47 +02:00
Michael Eischer 2e606ca70b backup: rework read concurrency 2022-10-02 22:55:14 +02:00
Kyle Brennan 4a501d7118 backup: add option for file read concurrency 2022-10-02 22:51:45 +02:00
Michael Eischer 9ec7eee803
Merge pull request #3521 from MichaelEischer/redownload-broken-files
Redownload files with wrong hash
2022-10-02 22:50:03 +02:00
Michael Eischer b25d0773b6
Merge pull request #3944 from MichaelEischer/fix-linter-errors
CI: ignore warning about missing package comment
2022-09-27 21:41:55 +02:00
Michael Eischer 5265550ff3 CI: ignore warning about missing package comment 2022-09-27 21:31:37 +02:00
Michael Eischer e89fc2a29d
Merge pull request #3943 from MichaelEischer/find-match-only-valid-ids
ignore filenames which are not IDs when expanding a prefix
2022-09-27 20:56:48 +02:00
Michael Eischer 67e4620cd6
Merge pull request #3938 from restic/errdot
rclone/sftp: Improve handling of ErrDot errors
2022-09-27 20:33:42 +02:00
Michael Eischer 5d3c5b9e50 restic: ignore filenames which are not IDs when expanding a prefix
Some backends generate additional files for each existing file, e.g.

1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef.sha256

For some commands this leads to an "multiple IDs with prefix" error when
trying to reference a snapshot.
2022-09-27 20:30:40 +02:00
Leo R. Lundgren ebe9f2c969 rclone/sftp: Improve handling of ErrDot errors
Restic now yields a more informative error message when exec.ErrDot occurs.
2022-09-25 16:19:03 +02:00
Michael Eischer d114e483c4 Add changelog for corrupt data downloads 2022-09-25 11:55:09 +02:00
Michael Eischer 34c1a83340 cache: Drop cache entry if it cannot be processed
Failing to process data requested from the cache usually indicates a
problem with the returned data. Assume that the cache entry is somehow
damaged and retry downloading it once.
2022-09-25 11:55:09 +02:00
Michael Eischer aa3b1925b4 cache: Simplify loadFromCacheOrDelegate 2022-09-25 11:35:35 +02:00
Michael Eischer 5c6b6edefe retry index, lock and snapshot loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer 822422ef03 retry key loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer d6575f53ca
Merge pull request #3942 from MichaelEischer/split-cross-compile-test
Split cross compile test
2022-09-24 22:27:08 +02:00
Michael Eischer 78d2312ee9
Merge pull request #3854 from MichaelEischer/sparsefiles
restore: Add support for sparse files
2022-09-24 22:04:02 +02:00
Michael Eischer 46b30b9826 split cross compilation into three parts
The cross compilation tasks are currently the slowest part of the CI
runs. Splitting it into three parts should reduce its time to roughly
that of the windows CI run.
2022-09-24 22:00:25 +02:00
Michael Eischer bd191ec60b update golang-ci to version 1.49 2022-09-24 22:00:08 +02:00
Michael Eischer 519059cca4 update ci actions 2022-09-24 21:59:36 +02:00
Michael Eischer 19afad8a09 restore: support sparse restores also on windows 2022-09-24 21:39:39 +02:00
Michael Eischer 0f89f443c7 update sparse restore changelog 2022-09-24 21:39:39 +02:00
Michael Eischer c147422ba5 repository: special case SaveBlob for all zero chunks
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.

The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.

This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
2022-09-24 21:39:39 +02:00
Michael Eischer 34fe1362da restorer: move zeroPrefixLen to restic package 2022-09-24 21:39:39 +02:00
Michael Eischer a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
2022-09-24 21:39:39 +02:00
Michael Eischer 5b6a77058a Enable sparseness only conditionally
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
greatroar 3047bf611c Changelog entry for sparse file restoring 2022-09-24 21:18:48 +02:00
greatroar 5d4568d393 Write sparse files in restorer
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
2022-09-24 21:18:48 +02:00
Michael Eischer eb83402d39
Merge pull request #3935 from miles170/master
Only display the message if there were locks to be removed
2022-09-24 20:53:13 +02:00
Michael Eischer ef58ddd7b1
Merge pull request #3923 from MichaelEischer/fix-flaky-cache-test
cache: fix flaky TestFileSaveConcurrent on windows
2022-09-24 20:52:55 +02:00
Michael Eischer 7fc178aaf4 internal/cache: extend description of cache sharing test failure 2022-09-24 13:07:01 +02:00
Miles Liu 1acbda18f8
Only display the message if there were locks to be removed
`restic unlock` now only shows `successfully removed locks` if there were locks to be removed.
In addition, it also reports the number of the removed lock files.
2022-09-24 19:02:24 +08:00
Michael Eischer da1a359c8b
Merge pull request #3927 from MichaelEischer/faster-index-each
Speed up MasterIndex.Each
2022-09-24 12:35:23 +02:00
Michael Eischer 041a51512a
Merge pull request #3780 from jkmw/fix/2578
Remove existing path before restoring a symlink
2022-09-24 12:34:42 +02:00
Michael Eischer 1ebd57247a repository: optimize MasterIndex.Each
Sending data through a channel at very high frequency is extremely
inefficient. Thus use simple callbacks instead of channels.

> name                old time/op  new time/op  delta
> MasterIndexEach-16   6.68s ±24%   0.96s ± 2%  -85.64%  (p=0.008 n=5+5)
2022-09-24 12:21:59 +02:00
Michael Eischer 825b95e313 repository: add benchmark for MasterIndex.Each 2022-09-24 12:21:59 +02:00
greatroar 1220fe9650 internal/cache: Concurrent use of cache not working on Windows 2022-09-17 19:49:44 +02:00
Jerome Küttner ef618bdd3f use os.Remove if path already exists on symlink restore 2022-09-14 08:14:31 +02:00
Michael Eischer b48766d7b8
Merge pull request #3928 from restic/rawtaz-doc-b2-s3
doc: Clarify S3 recommendation for B2 slightly
2022-09-13 20:59:50 +02:00
rawtaz 20f1913ef7
doc: Clarify S3 recommendation for B2 slightly
This gives slightly more background to the recommendation for using restic's S3 backend with Backblaze B2.
2022-09-12 17:48:59 +02:00
rawtaz d79e61ce5d
Merge pull request #3925 from hgraeber/add-powershell-completion
Add powershell completion
2022-09-11 01:04:57 +02:00
Herbert Graeber 988b386e8b Add powershell completion
- Add code for powersehll complition available in cobra
- Add documentation for powershell completion
- Add changelog for pr3925
2022-09-11 00:44:12 +02:00
rawtaz 14d09a6081
Merge pull request #3912 from MichaelEischer/cleanup-snapshot-filter-options
Clean up snapshot filter options
2022-09-11 00:18:42 +02:00
Michael Eischer 381da0443a tweak snapshot filter descriptions 2022-09-10 23:50:20 +02:00
Michael Eischer 8b9778d537
Merge pull request #3900 from MichaelEischer/b2-init-timeout
Add timeout for the initial connection to B2
2022-09-10 23:28:59 +02:00
Michael Eischer 17c27400f8
Merge pull request #3921 from MichaelEischer/filter-cleanup-error-handling
filter: deduplicate error handling for pattern validation
2022-09-10 23:24:50 +02:00
Michael Eischer f76643bd2e
Merge pull request #3894 from MichaelEischer/filter-mount-exit-code
Mount should return exit code 0 after pressing Ctrl-C
2022-09-10 23:22:01 +02:00
Michael Eischer be9ccc186e
Merge pull request #3875 from MichaelEischer/fix-fuse-context-cancel
mount: Fix input/output errors for canceled syscalls
2022-09-10 23:20:29 +02:00
Michael Eischer 2363e5c083
Merge pull request #3913 from MichaelEischer/better-migrate-error-message
migrate: Report why an migration cannot be applied
2022-09-09 23:37:25 +02:00
Michael Eischer 8e0ca80547 filter: deduplicate error handling for pattern validation 2022-09-09 23:12:41 +02:00
plumbeo d66e755ac7 Change uncompressed size calculation to account for the encryption overhead 2022-09-08 10:15:19 +02:00
plumbeo 837b816358 restic stats: print uncompressed size in mode raw-data 2022-09-05 17:38:32 +02:00
Michael Eischer d6309961c5 deduplicate the snapshot filter cli option setup 2022-09-04 10:27:33 +02:00
Michael Eischer 8b4dd70013 migrate: Report why an migration cannot be applied
Just returning that `Migration upgrade cannot be applied: check failed`
is not too useful when running `migrate upgrade_repo_v2`.
2022-09-03 11:49:31 +02:00
Michael Eischer 7689d6c679 normalize help text for host, tag and path options 2022-09-03 00:06:38 +02:00
Michael Eischer 6c69f08a7b
Merge pull request #3905 from DRON-666/haspaths-linear
Reduce quadratic time complexity of `Snapshot.HasPaths`
2022-08-30 20:35:56 +02:00
Michael Eischer 3e70bac56e
Merge pull request #3898 from MichaelEischer/fix-copy-hang
don't hang when `copy` uses a single connection
2022-08-30 20:23:39 +02:00
DRON-666 2a630c51c1 Add changelog 2022-08-30 20:22:07 +02:00
DRON-666 d0f1060df7 Fix quadratic time complexity of Snapshot.HasPaths 2022-08-30 04:38:17 +03:00
Michael Eischer f481ad64c8
Merge pull request #3904 from lbausch/add-newline
Add newline to keep prompt intact
2022-08-29 21:43:18 +02:00
Lorenz Bausch 7ddd803e46
Add newline to keep prompt intact 2022-08-29 17:37:49 +02:00
Michael Eischer e5b2c4d571 b2: sniff the error that caused init retry loops 2022-08-28 17:46:03 +02:00
Michael Eischer dc2db2de5e b2: cancel connection setup after a minute
If the connection to B2 fails, the library enters an endless loop.
2022-08-28 14:56:17 +02:00
Michael Eischer 7682149c9d repository: cleanup copy connection count check 2022-08-28 11:40:56 +02:00
Michael Eischer b03277ead5 repository: don't hang when copying using a single connection 2022-08-28 11:40:31 +02:00
Michael Eischer 1b233c4e2e
Merge pull request #2661 from creativeprojects/issue-1734
"Everything is ok" message after retrying
2022-08-28 11:04:59 +02:00
Fred 4042db5169 Add changelog 2022-08-27 22:36:19 +02:00
Fred be6baaec12 Add success callback to the backend 2022-08-27 22:27:15 +02:00
Fred baf58fbaa8 Add unit tests 2022-08-27 22:21:06 +02:00
Fred d629333efe Add function to notify of success after retrying 2022-08-27 22:21:06 +02:00
Alexander Neumann c169e37139
Merge pull request #3895 from MichaelEischer/refactor-cat-key
cat: Simplify implementation of 'cat key'
2022-08-27 18:40:46 +02:00
Michael Eischer 1b4af0c6e5 cat: Simplify implementation of 'cat key' 2022-08-26 23:21:51 +02:00
Michael Eischer 3174641ca4 add changelog for mount exit code filtering 2022-08-26 23:17:04 +02:00
Michael Eischer 5478ab22c5 mount: return exit code 0 after receiving a SIGINT 2022-08-26 23:07:07 +02:00
Michael Eischer d768c1c3e4 Allow cleanup handlers to filter the exit code 2022-08-26 23:04:59 +02:00
Michael Eischer 908f7441fe
Merge pull request #3885 from MichaelEischer/delete-fixes
Improve reliability of upload retries and B2 file deletions
2022-08-26 22:30:50 +02:00
Michael Eischer 4c90d91d4d backend: Test that failed uploads are not removed for backends with atomic replace 2022-08-26 21:20:52 +02:00
Michael Eischer 694dfa026a add changelog for reliable B2 deletes 2022-08-26 21:20:46 +02:00
MichaelEischer 582167d671
Merge pull request #3882 from MichaelEischer/sftp-init-single-connection
sftp: Only connect once to server during `init`
2022-08-26 21:13:28 +02:00
MichaelEischer 3822ded0b3
Merge pull request #3877 from MichaelEischer/no-env-in-help
Do not include the actual values of environment variables in help output
2022-08-26 20:59:54 +02:00
Michael Eischer cf0a8d7758 sftp: Only connect once for repository creation
This is especially useful if ssh asks for a password or if closing the
initial connection could return an error due to a problematic server
implementation.
2022-08-26 20:50:40 +02:00
Michael Eischer dd7cd5b9b3 fuse: remove unused context parameter 2022-08-26 20:48:48 +02:00
Michael Eischer a0c1ae9f90 mount: Correctly return context.Canceled for interrupted syscalls
bazil/fuse expects us to return context.Canceled to signal that a
syscall was successfully interrupted. Returning a wrapped version of
that error however causes the fuse library to signal an EIO (input/output
error). Thus unwrap context.Canceled errors before returning them.
2022-08-26 20:48:48 +02:00
Michael Eischer 5d0649faaf Update help output in docs 2022-08-26 20:44:01 +02:00
Michael Eischer faa4597af1 Set name for option values of cli 2022-08-26 20:42:34 +02:00
Michael Eischer 6ed157aee6 Do not include the actual values of environment variables in help output
This results in printing a `(default: $ENV) (default: value)` suffix for
the corresponding options which looks strange. In addition, some of the
environment variables might contain secrets which should not be
displayed.
2022-08-26 20:39:54 +02:00
MichaelEischer f7808245aa
Merge pull request #3878 from MichaelEischer/cheaper-cache-load
cache: Just try to open cache entry without calling stat first
2022-08-26 20:33:36 +02:00
MichaelEischer bee15dd555
Merge pull request #3879 from MichaelEischer/mem-optimize
Some random (minor) memory-allocation optimizations
2022-08-26 20:33:02 +02:00
MichaelEischer 0e1d082b12
Merge pull request #3886 from MichaelEischer/recommend-s3-over-b2
doc: recommend usage of B2's S3 API
2022-08-26 20:29:05 +02:00
Alexander Neumann d464543171 Update repo version table 2022-08-25 21:30:25 +02:00
Alexander Neumann 6b40456db7 Set development version for 0.14.0 2022-08-25 19:55:05 +02:00
Alexander Neumann 1bc87e1718 Add version for 0.14.0 2022-08-25 19:55:00 +02:00
Alexander Neumann c1a5da56e3 Update manpages and auto-completion 2022-08-25 19:55:00 +02:00
Alexander Neumann 193c62dfc3 Generate CHANGELOG.md for 0.14.0 2022-08-25 19:54:02 +02:00
Alexander Neumann a825e0d409 Prepare changelog for 0.14.0 2022-08-25 19:54:01 +02:00
rawtaz b824d8cdcc
Merge pull request #3891 from MichaelEischer/fix-secondary-repo-typo
Fix typo in the environment variable name for `--from-password-file`
2022-08-24 22:38:47 +02:00
Michael Eischer a2e89234fc Fix typo in the environment variable name for --from-password-file 2022-08-24 22:25:18 +02:00
MichaelEischer b4ae05627f
Merge pull request #3876 from MichaelEischer/document-aws-session-token
doc: document aws session token
2022-08-24 22:20:26 +02:00
MichaelEischer bd7bca2b51
Merge pull request #3889 from restic/prepare-0-14
Polish changelog entries
2022-08-24 21:28:44 +02:00
MichaelEischer be90a565cc
Merge pull request #3887 from MichaelEischer/rclone-permanent-error
rclone: Return a permanent error if rclone already exited
2022-08-24 21:19:00 +02:00
Michael Eischer 7a5d29ce24 Further changelog polishing 2022-08-24 21:13:14 +02:00
MichaelEischer d198a77d86
Merge pull request #3888 from restic/doc-prepare-repo
doc: Improve/clarify preparing and versions of repositories
2022-08-24 20:40:49 +02:00
Leo R. Lundgren cb5a61c46e doc: Improve/clarify preparing and versions of repositories 2022-08-24 02:08:46 +02:00
Leo R. Lundgren ee6b9dc492 Polish changelog entries 2022-08-24 01:23:51 +02:00
Michael Eischer 506d92e87c rclone: Return a permanent error if rclone already exited
rclone can exit early for example when the connection to rclone is
relayed for example via ssh: `-o rclone.program='ssh user@example.org
forced-command'`
2022-08-23 22:05:04 +02:00
MichaelEischer ad6eabbfa5
Merge pull request #3883 from MichaelEischer/update-dependencies
Update dependencies
2022-08-23 21:44:38 +02:00
Michael Eischer 7681a63fdb restic: Cleanup xattr error handling for Solaris
Since xattr 0.4.8 (https://github.com/pkg/xattr/pull/68) returns ENOTSUP
similar to Linux.
2022-08-23 21:25:15 +02:00
Michael Eischer 99e4ccbd94 remain compatible with go 1.15 2022-08-23 21:25:14 +02:00
Michael Eischer 22f46c18f9 downgrade bazil/fuse again to retain macOS support 2022-08-23 21:24:47 +02:00
Michael Eischer 6db979b3a6 update dependencies 2022-08-23 21:24:46 +02:00
MichaelEischer 9cdc8da10f
Merge pull request #3881 from MichaelEischer/cleanup-release-helper
Cleanup release helper
2022-08-23 21:10:57 +02:00
MichaelEischer b51e73e78f
Merge pull request #3884 from MichaelEischer/fix-index-saving-progress
repository: Do not report ignored packs in EachByPack
2022-08-23 21:10:07 +02:00
MichaelEischer 98dcd0a887
Merge pull request #3874 from MichaelEischer/disk-wear-note
Add note that larger packs increase disk wear
2022-08-21 22:18:08 +02:00
Michael Eischer e0d6bf525c doc: fix typo 2022-08-21 19:12:19 +02:00
Michael Eischer 5d0523e2f1 Add note that larger packs increase disk wear 2022-08-21 19:12:19 +02:00
Michael Eischer c586a5e20f doc: recommend usage of B2's S3 API 2022-08-21 11:39:03 +02:00
Michael Eischer 623556bab6 b2: Increase list size to maximum
Just request as many files as possible in one call to reduce the number
of network roundtrips.
2022-08-21 11:20:03 +02:00
Michael Eischer de0162ea76 backend/retry: Overwrite failed uploads instead of deleting them
For backends which are able to atomically replace files, we just can
overwrite the old copy, if it is necessary to retry an upload. This has
the benefit of issuing one operation less and might be beneficial if a
backend storage, due to bugs or similar, could mix up the order of the
upload and delete calls.
2022-08-21 11:14:53 +02:00
Michael Eischer fc506f8538 b2: Repeat deleting until all file versions are removed
When hard deleting the latest file version on B2, this uncovers earlier
versions. If an upload required retries, multiple version might exist
for a file. Thus to reliably delete a file, we have to remove all
versions of it.
2022-08-21 11:11:00 +02:00
Michael Eischer cc4728d287 repository: Do not report ignored packs in EachByPack
Ignored packs were reported as an empty pack by EachByPack. The most
immediate effect of this is that the progress bar for rebuilding the
index reports processing more packs than actually exist.
2022-08-21 10:38:40 +02:00
Michael Eischer c3374b3ea5 helper: download modules as first step
There's no use in running that step in parallel.
2022-08-20 12:11:54 +02:00
Michael Eischer 7f0929e519 helper: Reduce number of parallel builds a bit
The go compiler is already parallelized. The high concurrency caused my
podman container to hit a resource limit.
2022-08-20 12:10:48 +02:00
Michael Eischer ed94678820 helper: cleanups 2022-08-20 12:10:29 +02:00
Michael Eischer e530d422a0 helper: don't setup cmd paths twice 2022-08-20 12:09:42 +02:00
Michael Eischer 7a992fc794 repository: Reduce buffer reallocations in ForAllIndexes
Previously the buffer was grown incrementally inside `repo.LoadUnpacked`.
But we can do better as we already know how large the index will be.
Allocate a bit more memory to increase the chance that the buffer can be
reused in the future.
2022-08-19 21:13:40 +02:00
Michael Eischer 77b1980d8e repository: MasterIndex.Packs: reduce allocations 2022-08-19 21:10:43 +02:00
Michael Eischer 6ff9517e45 repository: MasterIndex.ListPacks / Index.EachByPack allow earlier GC
Allow earlier garbage collection of some of the intermediate data
structures.
2022-08-19 21:06:33 +02:00
Michael Eischer ce902aac67 cache: Just try to open cache entry without calling stat first
Instead of first checking whether a file is in the repository cache and
then opening it, we just can open the file. This saves one stat call. If
the file is in the cache, everything is fine and otherwise the code
follows its normal fallback path.
2022-08-19 20:59:06 +02:00
Michael Eischer b6c86ababe doc: document aws session token 2022-08-19 20:41:15 +02:00
MichaelEischer 0d9ac78437
Merge pull request #3873 from MichaelEischer/gofmt-comments
gofmt comments
2022-08-19 19:54:30 +02:00
MichaelEischer 7e96a5af62
Merge pull request #3872 from MichaelEischer/fuse-fix
mount: Only remember successful snapshot refreshes
2022-08-19 19:21:29 +02:00
Michael Eischer f414db987d gofmt all files
Apparently the rules for comment formatting have changed with go 1.19.
2022-08-19 19:12:26 +02:00
Michael Eischer 522406b4f0 mount: Only remember successful snapshot refreshes
If the context provided by the fuse library is canceled before the index
was loaded this could lead to missing snapshots.
2022-08-19 19:07:07 +02:00
MichaelEischer dbca93da28
Merge pull request #3742 from MichaelEischer/from-repo
copy: replace --repo2 with --from-repo
2022-08-19 19:01:04 +02:00
MichaelEischer b4dfab002a
Merge pull request #3691 from greatroar/tag-filenames
Sanitize tags when used as filenames by restic mount
2022-08-19 18:33:44 +02:00
Michael Eischer 2758d76b77 copy: replace --repo2 with --from-repo
`init` and `copy` use `--repo2` with two different meaning which has
proven to be confusing for users. `--from-repo` now consistently marks a
source repository from which data is read. `--repo` is now always the
target/destination repository.
2022-08-19 18:33:26 +02:00
Michael Eischer af50fe9ac0 mount: Map slashes in tags to underscores
Suggested-by: greatroar <>
2022-08-19 18:17:57 +02:00
MichaelEischer 4cccffab58
Merge pull request #3862 from restic/3861-forget-invalid-policy
forget: Error when invalid unit is given in duration policy
2022-08-18 20:45:26 +02:00
Michael Eischer 2ea6c82cf6 comment cleanup
gofmt reformatted the comment
2022-08-18 20:15:38 +02:00
Michael Eischer bb27f7408c forget: Fail test if duration parsing error is missing 2022-08-18 20:14:09 +02:00
MichaelEischer c4b3a154ba
Merge pull request #3865 from restic/rawtaz-doc-comp-typo
doc: Fix typo in compression section
2022-08-18 19:37:44 +02:00
rawtaz 2cb2aa31cd
doc: Fix typo in compression section 2022-08-12 22:17:38 +02:00
Leo R. Lundgren 6f517858e8 forget: Error when invalid unit is given in duration policy 2022-08-10 13:37:26 +02:00
rawtaz f0bb4f8708
Merge pull request #3857 from restic/rawtaz-gcs-account
doc: Update links to GCS documentation
2022-08-08 22:52:30 +02:00
Michael Eischer 40c8755b13 doc: Update more links to GCS documentation 2022-08-08 20:00:44 +02:00
rawtaz f673068dbb
doc: Update link to GCS documentation
Updates the link to Google Cloud Storage documentation about creating a service account key.
2022-08-08 12:51:43 +02:00
rawtaz f26231c9e6
Merge pull request #3852 from MichaelEischer/snapshots-processed
stats: Add snapshots count to json output
2022-08-08 00:05:15 +02:00
MichaelEischer 80e3efffef
Merge pull request #3855 from mattxtaz/master
Fix typo with double percentage in help text
2022-08-07 22:44:36 +02:00
mattxtaz 01ab36336f Fix typo with double percentage in help text 2022-08-07 20:21:05 +01:00
Michael Eischer 6e92d852a8 stats: Add snapshots count to json output 2022-08-07 15:44:09 +02:00
MichaelEischer 9ad3ad5972
Merge pull request #3850 from lbausch/go1.19
Update tests to Go 1.19
2022-08-07 14:56:17 +02:00
MichaelEischer 2930a102de
Merge pull request #3731 from metalsp0rk/feature/min-packsize-flag
Feature: min packsize flag
2022-08-07 14:54:45 +02:00
MichaelEischer 8fa64a8f99
Merge pull request #3036 from greatroar/refactor-fuse
Clean up internal/fuse
2022-08-07 14:46:30 +02:00
Michael Eischer f3fdc66b32
restic: Use stable sorting in snapshot policy
sort.Sort is not guaranteed to be stable. Go 1.19 has changed the
sorting algorithm which resulted in changes of the sort order. When
comparing snapshots with identical timestamp but different paths and
tags lists, there is not meaningful order among them. So just keep their
order stable.
2022-08-07 14:10:40 +02:00
Lorenz Bausch b82f4824f0
Bump golangci-lint version 2022-08-07 14:10:39 +02:00
Lorenz Bausch 0b9b4c52ad
Update tests to Go 1.19 2022-08-07 14:10:39 +02:00
Michael Eischer 0b7291b8b2 mount: Fix parent inode used by snapshots dir 2022-08-07 13:03:32 +02:00
greatroar cfa80e2c6b mount: remove unused inode field from root node 2022-08-07 13:03:26 +02:00
MichaelEischer 74ae76036f
Merge pull request #2913 from aawsome/mount-snapshot-slashes
mount: Make snapshots dir structure customizable
2022-08-07 12:27:59 +02:00
MichaelEischer 09497aec02
Merge pull request #3826 from MichaelEischer/debug-log-for-release
Debug log for release build
2022-08-07 12:15:27 +02:00
Michael Eischer 83b4c50ee3 Mention --snapshot-template and --time-template in changelog 2022-08-07 12:13:06 +02:00
Michael Eischer caa17988a3 fuse: Redesign snapshot dirstruct
Cleanly separate the directory presentation and the snapshot directory
structure. SnapshotsDir now translates the dirStruct into a format
usable by the fuse library and contains only minimal special case rules.
All decisions have moved into SnapshotsDirStructure which now creates a
fully preassembled tree data structure.
2022-08-07 12:13:06 +02:00
Michael Eischer a3e48da3a3 Add changelog for DEBUG_LOG available in release builds 2022-08-05 23:49:39 +02:00
Michael Eischer b3cdee66a9 update documentation to reflect DEBUG_LOG for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer 1ed775e3a8 debug: support roundtripper logging also for release builds
Different from debug builds do not use the eofDetectRoundTripper if
logging is disabled.
2022-08-05 23:49:39 +02:00
Michael Eischer 38becfc436 debug: enable debug support for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer 82c268c917 Remove unused hooks mechanism 2022-08-05 23:49:39 +02:00
Michael Eischer 7266f07c87 repository: StreamPack in parts if there are too large gaps
For large pack sizes we might be only interested in the first and last
blob of a pack file. Thus stream a pack file in multiple parts if the
gaps between requested blobs grow too large.
2022-08-05 23:48:36 +02:00
Michael Eischer 55a11c1396 Reword prune --repack-small description 2022-08-05 23:48:36 +02:00
Michael Eischer eaf43607f9 Add note that pack-size is not an exact limit 2022-08-05 23:48:36 +02:00
Michael Eischer 7f3b2be1e8 s3: Disable multipart uploads below 200MB 2022-08-05 23:48:36 +02:00
Michael Eischer 176b387d98 Always repack very small pack files 2022-08-05 23:48:36 +02:00
Michael Eischer 324935cb80 Only repack small files if there are multiple of them 2022-08-05 23:48:34 +02:00
Michael Eischer 1b076cda97 rename option to --pack-size 2022-08-05 23:47:43 +02:00
Michael Eischer d7e2892048 Add changelog for packsize option 2022-08-05 23:47:43 +02:00
Michael Eischer 8a44258b6f update restic help snippets in documentation 2022-08-05 23:47:43 +02:00
Michael Eischer 420ddc03c9 rework pack size parameter documentation 2022-08-05 23:47:43 +02:00
Kyle Brennan e43be84eb8 document minPackSize 2022-08-05 23:47:41 +02:00
Kyle Brennan 1e3f05c3f1 repository: prevent header overfill 2022-08-05 23:47:12 +02:00
Michael Eischer 6a6d313c9a prune: reduce priority of repacking small packs 2022-08-05 23:47:12 +02:00
Kyle Brennan 0269381b8d prune: add repack-small parameter 2022-08-05 23:47:12 +02:00
Michael Eischer 0a6fa602c8 add option for setting min pack size 2022-08-05 23:47:12 +02:00
Michael Eischer 2db7733ee3 fuse: remove unused MetaDir 2022-08-05 23:46:46 +02:00
Michael Eischer f678f7cb04 fuse: cleanup test 2022-08-05 23:46:46 +02:00
Alexander Weiss 1751afae26 Make snapshots dirs in mount command customizable 2022-08-05 23:46:46 +02:00
Alexander Weiss 57f4003f2f Generalize fuse snapshot dirs implemetation
+ allow "/" in tags and snapshot template
2022-08-05 23:46:46 +02:00
Alexander Weiss 696c18e031 Add possibility to set snapshot ID (used in test) 2022-08-05 23:46:46 +02:00
MichaelEischer 04a8ee80fb
Merge pull request #3829 from MichaelEischer/prune-refactor
Split prune into slightly small functions
2022-08-05 23:29:52 +02:00
MichaelEischer 15679be858
Merge pull request #3841 from MichaelEischer/compression-env
Add environment variable RESTIC_COMPRESSION
2022-08-05 23:20:04 +02:00
MichaelEischer 7d14b1baf1
Merge pull request #3845 from greatroar/solaris-xattr
internal/restic: Handle EINVAL for xattr on Solaris
2022-08-04 22:04:52 +02:00
greatroar ad6ac680af internal/restic: Handle EINVAL for xattr on Solaris
Also make the errors a bit less verbose by not prepending the operation,
since pkg/xattr already does that. Old errors looked like

    Listxattr: xattr.list /myfiles/.zfs/snapshot: invalid argument
2022-08-01 12:45:17 +02:00
MichaelEischer 846d021db5
Merge pull request #3840 from greatroar/sftp-init
Speed up restic init over slow SFTP links
2022-07-31 19:54:59 +02:00
greatroar b9fa6e05bd Add changelog for #3837/#3840 2022-07-30 22:53:52 +02:00
Michael Eischer 73053674d9 repository: Test fallback to existing blobs 2022-07-30 17:37:07 +02:00
Michael Eischer e85a21eda2 prune: move code 2022-07-30 17:37:07 +02:00
Michael Eischer 623770eebb repository: try to recover from invalid blob while repacking
If a blob that should be kept is invalid, Repack will now try to request
the blob using LoadBlob. Only return an error if that fails.
2022-07-30 17:37:07 +02:00
Michael Eischer d0590b7841 prune: Add internal integrity check
After repacking every blob that should be kept must have been repacked.
We have seen a few cases in which a single blob went missing, which
could have been caused by a bitflip somewhere. This sanity check might
help catch some of these cases.
2022-07-30 17:37:07 +02:00
Michael Eischer 5cbde03eae prune: split into smaller functions 2022-07-30 17:37:07 +02:00
Alexander Weiss 7643237da5 prune: separate collecting/printing/pruning 2022-07-30 17:37:07 +02:00
Michael Eischer 5723c72eb1 Add environment variable RESTIC_COMPRESSION 2022-07-30 16:21:53 +02:00
greatroar 2bdc40e612 Speed up restic init over slow SFTP links
pkg/sftp.Client.MkdirAll(d) does a Stat to determine if d exists and is
a directory, then a recursive call to create the parent, so the calls
for data/?? each take three round trips. Doing a Mkdir first should
eliminate two round trips for 255/256 data directories as well as all
but one of the top-level directories.

Also, we can do all of the calls concurrently. This may reintroduce some
of the Stat calls when multiple goroutines try to create the same
parent, but at the default number of connections, that should not be
much of a problem.
2022-07-30 13:09:08 +02:00
greatroar 23ebec717c Remove stale comments from backend/sftp
The preExec and postExec functions were removed in
0bdb131521 from 2018.
2022-07-30 13:07:25 +02:00
MichaelEischer 4ffd479ba4
Merge pull request #3773 from MichaelEischer/efficient-dir-json
Reduce memory usage for large directories/files
2022-07-23 17:47:32 +02:00
Michael Eischer 2ba14160de Add changelog for the optimized tree serialization 2022-07-23 14:49:08 +02:00
Michael Eischer 4a10ebed15 archiver: reduce memory usage for large files
FutureBlob now uses a Take() method as a more memory-efficient way to
retrieve the futures result. In addition, futures are now collected
while saving the file. As only a limited number of blobs can be queued
for uploading, for a large file nearly all FutureBlobs already have
their result ready, such that the FutureBlob object just consumes
memory.
2022-07-23 14:45:07 +02:00
Michael Eischer b817681a11 archiver: Incrementally serialize tree nodes
That way it is not necessary to keep both the Nodes forming a Tree and
the serialized JSON version in memory.
2022-07-23 14:45:07 +02:00
Michael Eischer c206a101a3 archiver: unify FutureTree/File into futureNode
There is no real difference between the FutureTree and FutureFile
structs. However, differentiating both increases the size of the
FutureNode struct.

The FutureNode struct is now only 16 bytes large on 64bit platforms.
That way is has a very low overhead if the corresponding file/directory
was not processed yet.

There is a special case for nodes that were reused from the parent
snapshot, as a go channel seems to have 96 bytes overhead which would
result in a memory usage regression.
2022-07-23 14:45:07 +02:00
Michael Eischer 32f4997733 archiver: remove unused fileInfo from progress callback 2022-07-23 14:16:23 +02:00
Michael Eischer dcb00fd2d1 archiver: cleanup Saver interface 2022-07-23 14:16:23 +02:00
Michael Eischer 79321a195c archiver: remove dead attribute from FutureNode 2022-07-23 14:16:23 +02:00
MichaelEischer 049f4c4144
Merge pull request #3730 from MichaelEischer/stricter-check
Let `check` warn about legacy variants of the repo format version 1
2022-07-23 14:14:50 +02:00
Michael Eischer 3bf53da672 Add changelog for stricter checks 2022-07-23 11:21:26 +02:00
Michael Eischer 5a6f2f9fa0 Fix S3 legacy layout migration 2022-07-23 11:19:32 +02:00
Michael Eischer 04e49924fb checker: Fix S3 legacy layout detection 2022-07-23 11:19:32 +02:00
Michael Eischer 768c890fcb check: Deprecate `--check-unused`
Unused blobs are not a problem but rather expected to exist now that
prune by default does not remove every unused blob. However, the option
has caused questions from users whether a repository is damaged or not,
so just remove that option.

Note that the remaining code is left intact as it is still useful for
our test cases.
2022-07-23 11:19:32 +02:00
Michael Eischer fcb3ddf181 check: Complain about usage of s3 legacy layout 2022-07-23 11:19:32 +02:00
Michael Eischer 8b8bd4e8ac check: complain about mixed pack files 2022-07-23 11:19:32 +02:00
MichaelEischer 443cc49afd
Merge pull request #3830 from MichaelEischer/cleanup-repo
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
MichaelEischer 1f5369e072
Merge pull request #3831 from MichaelEischer/move-code
Move code out of the restic package and consolidate backend specific code
2022-07-23 10:33:05 +02:00
MichaelEischer 827ab02eea
Merge pull request #3661 from rgammans/azure_sas_support
add SAS authentication option for Azure repos
2022-07-23 10:32:03 +02:00
MichaelEischer e9c39442fb
Merge pull request #3827 from MichaelEischer/backup-doc
backup: clarify usage string
2022-07-23 10:31:33 +02:00
Michael Eischer 9729e6d7ef backend: extract readerat from restic package 2022-07-17 15:29:09 +02:00
Michael Eischer c44b21d366 restorer: extract hardlinks index from restic package 2022-07-17 13:45:42 +02:00
Michael Eischer 8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
Michael Eischer a0cef9f247 limiter: move to internal/backend 2022-07-17 13:40:15 +02:00
Michael Eischer 163ab9c025 mock: move to internal/backend 2022-07-17 13:40:06 +02:00
Michael Eischer 89d3ce852b repository: extract Load/StoreJSONUnpacked
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer fbcbd5318c repository: extract LoadTree/SaveTree
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
MichaelEischer d9ea1e9ee2
Merge pull request #3290 from aawsome/prune-handle-duplicates
prune: Handle duplicate blobs more efficiently
2022-07-17 11:51:54 +02:00
Michael Eischer 715d457aad prune: code cleanups 2022-07-17 11:41:56 +02:00
Michael Eischer 9be1bd2acc prune: handle very high duplication of some blobs
Suggested-By: Alexander Weiss <alex@weissfam.de>
2022-07-17 11:39:56 +02:00
Alexander Weiss 7478cbf70e prune: Enhance treatment of duplicates 2022-07-17 00:22:23 +02:00
Michael Eischer b2043e8198 backup: clarify usage string
Using the `--files-from` options it is possible to run `backup` without
specifying any source paths directly on the command line.
2022-07-17 00:03:22 +02:00
Michael Eischer 5639c41b6a azure: Strip ? prefix from sas token 2022-07-16 23:55:18 +02:00
Roger Gammans 64a7ec5341 azure: add SAS authentication option 2022-07-16 23:55:18 +02:00
MichaelEischer 6cbeb4a9f9
Merge pull request #3825 from restic/rawtaz-doc-gdrive
doc: Add note about using rclone for Google Drive
2022-07-16 19:21:59 +02:00
rawtaz f5c219f5a2
doc: Add note about using rclone for Google Drive
It wasn't clear that Google Cloud Storage and Google Drive are two different services and that one should use the rclone backend for the latter. This commit adds a note with this information.
2022-07-16 13:22:38 +02:00
MichaelEischer d71b29221b
Merge pull request #3822 from JsBergbau/doc-max-compression
Added hint for --compression max in migration process
2022-07-12 23:02:45 +02:00
Michael Eischer 71ff6b77f0 doc: Rework hint to repack with max compression 2022-07-12 21:24:40 +02:00
MichaelEischer 6970d05d47
Merge pull request #3741 from lbausch/repository-wording
Wording: change repo to repository
2022-07-12 21:15:33 +02:00
MichaelEischer 3934480da4
Merge pull request #3819 from lbausch/restore-validate-patterns
restore: validate include/exclude patterns
2022-07-12 20:58:41 +02:00
MichaelEischer 71a0157c2c
Merge pull request #3820 from lbausch/patch-1
Fix wording in changelog template
2022-07-12 20:51:52 +02:00
Michael Eischer 2aad6f24b5 doc: update sample help output 2022-07-12 20:49:01 +02:00
Michael Eischer ec4dfa3c66 Wording: replace further repo occurrences with repository 2022-07-12 20:48:01 +02:00
JsBergbau 8d3f04aefa
Added hint for --compression max in migration process
Added hint for --compression max in migration process. Since this is a onetime process users should be aware of this and consider this step.
2022-07-12 00:08:26 +02:00
Lorenz Bausch b609523582
Add changelog entry 2022-07-09 22:24:56 +02:00
lbausch ac96a4138d
Fix wording in changelog template 2022-07-09 22:13:54 +02:00
Lorenz Bausch 7e36ec279d
Test restore fails when using invalid patterns 2022-07-08 20:09:26 +02:00
Lorenz Bausch be524f0b78
Add testRunRestoreAssumeFailure function 2022-07-08 20:09:25 +02:00
Lorenz Bausch 9a7db6675c
Restore: validate provided patterns 2022-07-08 20:09:25 +02:00
Lorenz Bausch d6e3c7f28e
Wording: change repo to repository 2022-07-08 20:05:35 +02:00
MichaelEischer 98a3125ce4
Merge pull request #3816 from mattxtaz/master
Fix minor typo in docs
2022-07-05 21:30:26 +02:00
mattxtaz 87d899c099 Fix minor typo in docs 2022-07-05 20:06:27 +01:00
Jerome Küttner 6f3883c9d2 add changelog 2022-07-05 08:47:48 +02:00
Alexander Neumann 545220803b
Merge pull request #3813 from MichaelEischer/fix-blob-saver-data-race
Fix data race in blob_saver
2022-07-04 08:10:01 +02:00
Michael Eischer ce89018902 Fix data race in blob_saver
After the `BlobSaver` job is submitted, the buffer can be released and
reused by another `FileSaver` even before `BlobSaver.Save` returns. That
FileSaver will the change `buf.Data` leading to wrong backup statistics.

Found by `go test -race ./...`:

WARNING: DATA RACE
Write at 0x00c0000784a0 by goroutine 41:
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:176 +0x789
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91

Previous read at 0x00c0000784a0 by goroutine 29:
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save()
      /home/michael/Projekte/restic/restic/internal/archiver/blob_saver.go:57 +0x1dd
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save-fm()
      <autogenerated>:1 +0xac
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:191 +0x855
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91
2022-07-03 14:47:53 +02:00
MichaelEischer b6a38d43b3
Merge pull request #3611 from MichaelEischer/auto-concurrency
Adjust worker goroutines to number of backend connections
2022-07-03 12:33:32 +02:00
Michael Eischer 3af9c2cc58 Document automatic CPU/IO-concurrency 2022-07-03 12:19:26 +02:00
Michael Eischer 6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
MichaelEischer cd50feb66f
Merge pull request #3489 from MichaelEischer/async-pack-uploads
Asynchronously upload pack files
2022-07-03 11:56:05 +02:00
Michael Eischer 74df9d5998 Add changelog for async pack uploads 2022-07-03 11:34:01 +02:00
Michael Eischer dbb5860dc9 Document connections and compression option 2022-07-03 11:19:24 +02:00
Michael Eischer 753e56ee29 repository: Limit to a single pending pack file
Use only a single not completed pack file to keep the number of open and
active pack files low. The main change here is to defer hashing the pack
file to the upload step. This prevents the pack assembly step to become
a bottleneck as the only task is now to write data to the temporary pack
file.

The tests are cleaned up to no longer reimplement packer manager
functions.
2022-07-02 22:42:34 +02:00
Michael Eischer fa25d6118e archiver: Reduce tree saver concurrency
Large amount of tree savers have no obvious benefit, however they can
increase the amount of (potentially large) trees kept in memory.
2022-07-02 22:42:34 +02:00
Michael Eischer bba1e81719 archiver: Limit blob saver count to GOMAXPROCS
Now with the asynchronous uploaders there's no more benefit from using
more blob savers than we have CPUs. Thus use just one blob saver for
each CPU we are allowed to use.
2022-07-02 22:42:34 +02:00
Michael Eischer 120ccc8754 repository: Rework blob saving to use an async pack uploader
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.

Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.

To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Michael Eischer bba4c69a2a tag: Remove unnecessary flush call 2022-07-02 22:42:23 +02:00
MichaelEischer 3e1de52e0a
Merge pull request #3805 from greatroar/global
cmd/restic, limiter: Move config knowledge to internal packages
2022-07-02 21:56:35 +02:00
MichaelEischer 621023a50b
Merge pull request #3772 from MichaelEischer/fix-mixed-index
rebuild-index: correctly rebuild index for mixed packs
2022-07-02 20:10:02 +02:00
MichaelEischer 90e9c5c4cc
Merge pull request #3729 from MichaelEischer/full-ids-in-check
Include full IDs in check output
2022-07-02 20:09:39 +02:00
MichaelEischer 7137034517
Merge pull request #3811 from restic/fix-secret-string-crash
Don't crash if SecretString is uninitialized
2022-07-02 19:56:36 +02:00
Michael Eischer cdaf9b4f26 Don't crash if SecretString is uninitialized 2022-07-02 19:44:28 +02:00
Michael Eischer 5e0f1c3cef check: remove dead code 2022-07-02 19:28:57 +02:00
Michael Eischer 0df022fa6d check: Print full ids
The short ids are not always unique. In addition, recovering from
damages is easier when having the full ids as that makes it easier to
access the corresponding files.
2022-07-02 19:28:57 +02:00
Michael Eischer 04c23fa95d rebuild-index: correctly rebuild index for mixed packs
For mixed packs, data and tree blobs were stored in separate index
entries. This results in warning from the check command and maybe other
problems.
2022-07-02 19:24:02 +02:00
MichaelEischer bb5f196b09
Merge pull request #3733 from restic/improve-stats
Improve stats
2022-07-02 19:07:31 +02:00
MichaelEischer c16f989d4a
Merge pull request #3470 from MichaelEischer/sanitize-debug-log
Sanitize debug log
2022-07-02 19:00:54 +02:00
Michael Eischer 00d7fcff96 extend compression feature changelog entry 2022-07-02 18:55:59 +02:00
Michael Eischer a6e9e08034 Account for pack header overhead at each entry
This will miss the pack header crypto overhead and the length field,
which only amount to a few bytes per pack file.
2022-07-02 18:55:58 +02:00
Michael Eischer 856d5e4303 stats: return storage size for raw-data mode
raw-data summed up the size of the blob plaintexts. However, with
compression this makes little sense as the storage size in the
repository is lower due to compression. Thus sum up the actual size each
blob takes in the repository.
2022-07-02 18:55:12 +02:00
Alexander Neumann 6c4ceaf1e7 Print number of bytes added to the repo
This includes optional compression and crypto overhead.
2022-07-02 18:55:12 +02:00
Alexander Neumann 99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
MichaelEischer fdc53a9d32
Merge pull request #3787 from MichaelEischer/refactor-repository
repository: (Mostly) index-related cleanups
2022-07-02 18:54:04 +02:00
Michael Eischer 6923353c43 redact swift auth token in debug output 2022-07-02 18:47:35 +02:00
Michael Eischer 5a11d14082 redacted keys/token in backend config debug log 2022-07-02 18:47:35 +02:00
Michael Eischer 0936d864a4 redact http authorization header in debug log output 2022-07-02 18:47:35 +02:00
Michael Eischer ec7c9ce88b drop unused repository.Loader interface 2022-07-02 18:39:59 +02:00
Michael Eischer 2cd7e90ad1 repository: cleanup 2022-07-02 18:39:59 +02:00
Michael Eischer c1a8fa4290 repository: remove unused packIDToIndex field 2022-07-02 18:39:59 +02:00
Michael Eischer e68c3a4e62 repository: simplify CreateIndexFromPacks 2022-07-02 18:39:59 +02:00
Michael Eischer 1974ad7ce2 repository: hide MasterIndex.FinalizeFullIndexes / FinalizeNotFinalIndexes 2022-07-02 18:39:59 +02:00
Michael Eischer ef53ca4a5a repository: remove MasterIndex.All() 2022-07-02 18:39:59 +02:00
Michael Eischer bf81bf0795 repository: Properly set id for finalized index
As MergeFinalIndex and index uploads can occur concurrently, it is
necessary for MergeFinalIndex to check whether the IDs for an index were
already set before merging it. Otherwise, we'd loose the ID of an index
which is set _after_ uploading it.
2022-07-02 18:39:59 +02:00
Michael Eischer e0a7852b8b repository: remove unused (Master)Index.Count 2022-07-02 18:39:58 +02:00
Michael Eischer 8ef2968f28 repository: remove unused index.ListPack 2022-07-02 18:39:12 +02:00
Michael Eischer e4f20dea61 repository: inline index.encode 2022-07-02 18:39:12 +02:00
Michael Eischer fe5a8e137a repository: remove unused index.Store 2022-07-02 18:39:12 +02:00
Michael Eischer 628ae799ca repository: make flushPacks private 2022-07-02 18:39:12 +02:00
Michael Eischer ed8aa15376 repository: add Save method to MasterIndex interface 2022-07-02 18:38:56 +02:00
Michael Eischer a77d5c4d11 repository: index saving belongs into the MasterIndex 2022-07-02 18:38:56 +02:00
MichaelEischer 19641bf828
Merge pull request #3810 from greatroar/revert-3786
Revert "restic prune: Merge three loops over the index"
2022-07-01 23:13:39 +02:00
greatroar a0fa9c6e9f Revert "restic prune: Merge three loops over the index"
This reverts commit 8bdfcf779f.
Should fix #3809. Also needed to make #3290 apply cleanly.
2022-06-30 15:27:34 +02:00
greatroar 90d2c0502b cmd/restic, limiter: Move config knowledge to internal packages
The GlobalOptions struct now embeds a backend.TransportOptions, so it
doesn't need to construct one in open and create. The upload and
download limits are similarly now a struct in internal/limiter that is
embedded in GlobalOptions.
2022-06-22 18:29:58 +02:00
MichaelEischer bc96879d41
Merge pull request #3785 from MichaelEischer/replace-tomb-usage
Remove usage of tomb package
2022-06-19 14:42:48 +02:00
MichaelEischer 307f14604f
Merge pull request #3795 from greatroar/sema
backend: Move semaphores to a dedicated package
2022-06-18 17:12:01 +02:00
MichaelEischer 19581dbc18
Merge pull request #3786 from greatroar/prune
restic prune: Merge three loops over the index
2022-06-18 16:54:50 +02:00
greatroar 8bdfcf779f restic prune: Merge three loops over the index
There were three loops over the index in restic prune, to find
duplicates, to determine sizes (in pack.Size) and to generate packInfos.
These three are now one loop. This way, prune doesn't need to construct
a set of duplicate blobs, pack.Size doesn't need to contain special
logic for prune's use case (the onlyHdr argument) and pack.Size doesn't
need to construct a map only to have it immediately transformed into a
different map.

Some quick testing on a 160GiB local repo doesn't show running time or
memory use of restic prune --dry-run changing significantly.
2022-06-18 10:40:33 +02:00
greatroar 910d917b71 backend: Move semaphores to a dedicated package
... called backend/sema. I resisted the temptation to call the main
type sema.Phore. Also, semaphores are now passed by value to skip a
level of indirection when using them.
2022-06-18 10:01:58 +02:00
MichaelEischer 2c893fe43c
Merge pull request #3798 from greatroar/errors
all: Move away from pkg/errors, easy cases
2022-06-17 19:01:40 +02:00
greatroar f92ecf13c9 all: Move away from pkg/errors, easy cases
github.com/pkg/errors is no longer getting updates, because Go 1.13
went with the more flexible errors.{As,Is} function. Use those instead:
errors from pkg/errors already support the Unwrap interface used by 1.13
error handling. Also:

* check for io.EOF with a straight ==. That value should not be wrapped,
  and the chunker (whose error is checked in the cases changed) does not
  wrap it.
* Give custom Error methods pointer receivers, so there's no ambiguity
  when type-switching since the value type will no longer implement error.
* Make restic.ErrAlreadyLocked private, and rename it to
  alreadyLockedError to match the stdlib convention that error type
  names end in Error.
* Same with rest.ErrIsNotExist => rest.notExistError.
* Make s3.Backend.IsAccessDenied a private function.
2022-06-14 08:36:38 +02:00
MichaelEischer 0c0e7b6957
Merge pull request #3776 from wjiec/bugfix/maxkeys-in-search
Limit number of key files tested while opening a repository
2022-06-12 15:51:10 +02:00
Michael Eischer c9ef873192 tweak password test count changelog 2022-06-12 15:39:06 +02:00
Michael Eischer d08549f0ae fix flaky key test 2022-06-12 14:19:06 +02:00
Jayson Wang f144920ed5 fix handling of maxKeys in SearchKey 2022-06-12 14:19:06 +02:00
Alexander Neumann 1dd4b9b60e
Merge pull request #3788 from greatroar/sftp-posix-rename
backend/sftp: Support atomic rename
2022-06-06 19:39:48 +02:00
Alexander Neumann 07114ccb21
Merge pull request #3789 from greatroar/fix-loadblob
internal/repository: Fix LoadBlob + fuzz test
2022-06-06 19:33:42 +02:00
greatroar c9557b2822 internal/repository: Fix LoadBlob + fuzz test
When given a buf that is big enough for a compressed blob but not its
decompressed contents, the copy at the end of LoadBlob would skip the
last part of the contents.

Fixes #3783.
2022-06-06 17:02:28 +02:00
greatroar fa8f02292e backend/sftp: Support atomic rename
... if the server has posix-rename@openssh.com.
OpenSSH introduced this extension in 2008:
7c29661471
2022-06-06 13:40:42 +02:00
Alexander Neumann 7d64aa7f57
Merge pull request #3784 from MichaelEischer/cleanup-migrate-check
migrate: Cleanup option to request repository check
2022-06-05 17:37:45 +02:00
Michael Eischer 853ceb3bec get rid of tomb package 2022-06-05 15:54:57 +02:00
Michael Eischer e002b09d57 archiver: free workers once finished 2022-06-05 15:48:10 +02:00
Michael Eischer 408ac1a0c2 archiver: remove tomb usage 2022-06-05 15:47:52 +02:00
Michael Eischer 5eba1217e7 migrate: Cleanup option to request repository check 2022-06-04 23:45:00 +02:00
MichaelEischer 0cb6b3d80a
Merge pull request #3778 from greatroar/ellipsis
cmd/restic: Remove trailing "..." from progress messages
2022-06-04 19:16:00 +02:00
MichaelEischer 60ca6b1418
Merge pull request #3774 from greatroar/archiver-pool
archiver: Remove cleanup goroutine from BufferPool
2022-06-04 18:50:24 +02:00
Jerome Küttner 9adaa6e240 Delete existing path before restoring a symlink 2022-06-01 17:26:25 +02:00
greatroar b7c990871f cmd/restic: Remove trailing "..." from progress messages
These were added after message since the last refactor of the progress
printing code. Also skips an allocation in the common case.
2022-05-31 19:06:26 +02:00
greatroar 0db1d11b2e archiver: Remove cleanup goroutine from BufferPool
This isn't doing anything. Channels should get cleaned up by the GC when
the last reference to them disappears, just like all other data
structures. Also inlined BufferPool.Put in Buffer.Release, its only
caller.
2022-05-29 17:09:16 +02:00
Alexander Neumann 74f7fe2b98
Merge pull request #3767 from MichaelEischer/fix-prune-empty-snapshot
prune: Fix crash on snapshot loading error
2022-05-29 16:52:45 +02:00
Alexander Neumann d2c5843c68
Merge pull request #3704 from MichaelEischer/compression-migrations
Support migration to repository format with compression
2022-05-29 15:52:21 +02:00
Alexander Neumann 78a21bbccf
Merge pull request #3752 from MichaelEischer/fix-dir-sync-errors
local: Ignore additional errors for directory syncing
2022-05-29 12:54:51 +02:00
MichaelEischer 2ce8587598
Merge pull request #3771 from greatroar/id-marshaljson
internal/restic: Custom ID.MarshalJSON
2022-05-28 16:24:51 +02:00
Michael Eischer a73fc31b50 Fix linter check 2022-05-28 16:13:46 +02:00
greatroar dde8e9e296 internal/restic: Custom ID.MarshalJSON
This skips an allocation. internal/archiver benchmarks, Linux/amd64:

name                     old time/op    new time/op    delta
ArchiverSaveFileSmall-8    3.94ms ± 6%    3.91ms ± 6%    ~     (p=0.947 n=20+20)
ArchiverSaveFileLarge-8     304ms ± 3%     301ms ± 4%    ~     (p=0.265 n=18+18)

name                     old speed      new speed      delta
ArchiverSaveFileSmall-8  1.04MB/s ± 6%  1.05MB/s ± 6%    ~     (p=0.803 n=20+20)
ArchiverSaveFileLarge-8   142MB/s ± 3%   143MB/s ± 4%    ~     (p=0.421 n=18+19)

name                     old alloc/op   new alloc/op   delta
ArchiverSaveFileSmall-8    17.9MB ± 0%    17.9MB ± 0%  -0.01%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     382MB ± 2%     382MB ± 1%    ~     (p=0.687 n=20+19)

name                     old allocs/op  new allocs/op  delta
ArchiverSaveFileSmall-8       540 ± 1%       528 ± 0%  -2.19%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     1.93k ± 3%     1.79k ± 4%  -7.06%  (p=0.000 n=20+20)
2022-05-27 12:26:37 +02:00
Alexander Neumann bc27c370e7 Update gopkg.in/yaml
This fixes a panic in invalid input, but I think we aren't affected.
2022-05-26 14:23:49 +02:00
Alexander Neumann 9e30152f3c
Merge pull request #3770 from lbausch/update-minio-go
Update github.com/minio/minio-go/v7 to v7.0.27
2022-05-26 14:21:19 +02:00
Lorenz Bausch cc3f8d3732
Update github.com/minio/minio-go/v7 to v7.0.27
This version adds support for Cloudflare R2, as discussed in #3757
2022-05-26 13:05:13 +02:00
Michael Eischer c8e1ac4049 prune: Don't print stack trace if snapshot can't be loaded 2022-05-23 22:38:45 +02:00
Michael Eischer 173695104c prune: Fix crash on empty snapshot 2022-05-23 22:32:59 +02:00
MichaelEischer ded783dd61
Merge pull request #3762 from marigbede/marigbede-gs-documentation-fix
Update 030_preparing_a_new_repo.rst
2022-05-23 22:10:38 +02:00
Arigbede Moses 2aa75d6272
Update 030_preparing_a_new_repo.rst 2022-05-18 22:03:59 +01:00
MichaelEischer 88a8701fb5
Merge pull request #3734 from lbausch/validate-patterns
Validate exclude patterns
2022-05-14 16:20:15 +02:00
MichaelEischer b2a2e5f727
Merge pull request #3753 from greatroar/indexmap-alloc
repository: Re-tune indexmap allocation strategy
2022-05-14 15:44:08 +02:00
MichaelEischer b52c631bd3
Merge pull request #3754 from greatroar/simplify-hashing
hashing: Fix up comments
2022-05-14 15:33:51 +02:00
Lorenz Bausch e7df66cc91
Add changelog entry for validating exclude patterns 2022-05-11 22:41:05 +02:00
Lorenz Bausch 36bd464e8c
Add tests for validating exclude patterns 2022-05-11 22:41:00 +02:00
greatroar 39a335e690 hashing: Fix up comments 2022-05-11 21:36:10 +02:00
greatroar 5141228e0c repository: Re-tune indexmap allocation strategy
fd05037e1a changed the allocation batch
size from 256 to 128 under the assumption that an indexEntry is 60 bytes
on amd64, but it's 64: structs are padded out to a multiple of 8 for
alignment reasons. That means we'd waste no space in malloc even without
the batch allocation, at least on 64-bit machines. While that strategy
cuts the overallocation down dramatically for many small indexes, it also
seems to slow allocation down (Go 1.18, Linux, amd64, -benchtime=2s):

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    DecodeIndexParallel-8     4.67s ± 3%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    IndexHasUnknown-8        37.8ns ± 8%    36.5ns ±14%      ~     (p=0.841 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    37.7ns ±10%      ~     (p=0.968 n=5+5)
    IndexAlloc-8              615ms ±18%     607ms ± 1%      ~     (p=1.000 n=10+5)
    IndexAllocParallel-8      245ms ±11%     285ms ± 6%   +16.40%  (p=0.001 n=10+5)
    MasterIndexAlloc-8        286ms ± 9%     275ms ± 2%      ~     (p=1.000 n=10+5)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 1%      ~     (p=0.690 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.8ms ± 2%    +1.48%  (p=0.016 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%    -0.00%  (p=0.000 n=8+4)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%    -0.00%  (p=0.008 n=8+5)
    MasterIndexAlloc-8        213MB ± 0%     159MB ± 0%   -25.47%  (p=0.000 n=10+5)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     2632k ± 0%  +188.19%  (p=0.008 n=5+5)
    IndexAllocParallel-8       913k ± 0%     2632k ± 0%  +188.21%  (p=0.008 n=5+5)
    MasterIndexAlloc-8         318k ± 0%     1172k ± 0%  +267.86%  (p=0.008 n=5+5)

Instead, this patch sets a batch size of 4, which means no space is
wasted by malloc on 64-bit and very little on 32-bit. It still gets very
close to the savings from not allocating in batches, without requiring
special code for bits.UintSize==64. Benchmark results, again for
Linux/amd64:

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.83s ± 9%     ~     (p=0.315 n=10+10)
    DecodeIndexParallel-8     4.67s ± 3%     4.68s ± 4%     ~     (p=0.315 n=10+10)
    IndexHasUnknown-8        37.8ns ± 8%    44.5ns ±19%     ~     (p=0.095 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    36.9ns ± 8%     ~     (p=0.690 n=5+5)
    IndexAlloc-8              615ms ±18%     628ms ±18%     ~     (p=0.218 n=10+10)
    IndexAllocParallel-8      245ms ±11%     262ms ± 9%   +7.02%  (p=0.043 n=10+10)
    MasterIndexAlloc-8        286ms ± 9%     287ms ±13%     ~     (p=1.000 n=10+10)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 0%     ~     (p=1.000 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.5ms ± 0%     ~     (p=0.056 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%     ~     (p=1.000 n=8+10)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%   -0.00%  (p=0.000 n=8+8)
    MasterIndexAlloc-8        213MB ± 0%     160MB ± 0%  -25.02%  (p=0.000 n=10+9)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+10)
    IndexAllocParallel-8       913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+8)
    MasterIndexAlloc-8         318k ± 0%      525k ± 0%  +64.99%  (p=0.000 n=10+10)

The allocation method indexmap.newEntry has also been rewritten in a
form that is a few instructions shorter.
2022-05-11 21:22:14 +02:00
Michael Eischer 48a0d83143 local: Ignore additional errors for directory syncing
Apparently SMB/CIFS on Linux/macOS returns somewhat random errnos when
trying to sync a windows share which does not support calling fsync for
a directory.
2022-05-11 20:37:59 +02:00
MichaelEischer ac36fda155
Merge pull request #3749 from greatroar/simplify-hashing
hashing: Remove io.WriterTo implementation
2022-05-11 20:03:43 +02:00
MichaelEischer df554e5f69
Merge pull request #3748 from greatroar/runworkers
repository: Remove RunWorkers, report ctx.Err()
2022-05-11 19:38:46 +02:00
greatroar 54b8337813 hashing: Remove io.WriterTo implementation
This functionality has gone unused since
4b3dc415ef changed hashing.Reader's only
client to use ioutil.ReadAll on a bufio.Reader wrapping the hashing
Reader.

Reverts bcb852a8d0.
2022-05-10 23:41:18 +02:00
greatroar 2e0f1f5113 repository: Remove RunWorkers, report ctx.Err()
This removes RunWorkers, which had become mere overhead by successive
refactors. It also ensures that each former user of that function
returns any context error that occurs, so failure to complete an
operation is always reported as an error.
2022-05-10 22:26:00 +02:00
MichaelEischer 47c56dea5c
Merge pull request #3746 from greatroar/cache-lstat
cache: Don't Lstat before creating CACHEDIR.TAG
2022-05-10 20:31:15 +02:00
Alexander Neumann c270ab1e08
Merge pull request #3744 from MichaelEischer/fix-windows-temp-file
Fix error on temp file deletion on windows
2022-05-10 19:53:25 +02:00
greatroar 2da377c582 cache: Don't Lstat before creating the tag file
The tag file is opened with O_CREATE|O_EXCL and ErrExist is handled, so
we don't need to check for existence first.
2022-05-10 18:52:39 +02:00
Michael Eischer ae7e51382a Fix error on temp file deletion on windows
Apparently it can take a moment between closing a tempfile marked as
DELETE_ON_CLOSE and it actually being deleted. During that time the file
is inaccessible. Thus just skip deleting the temp file on windows.
2022-05-09 22:43:26 +02:00
Michael Eischer 5c6db534d4 Add compression migration support to changelog 2022-05-09 22:39:02 +02:00
Michael Eischer c1bbbcd0dc migrate: Allow migrations to request a check run
This is currently only used by upgrade_repo_v2.
2022-05-09 22:31:30 +02:00
Michael Eischer 59eb132dcd check: Better differentiate between warnings and errors 2022-05-09 22:31:30 +02:00
Michael Eischer 5815f727ee checker: convert error type to use pointer-receivers 2022-05-09 22:31:30 +02:00
Michael Eischer 4faff0debe doc: Describe repository upgrade process 2022-05-09 22:31:30 +02:00
Michael Eischer e36a40db10 upgrade_repo_v2: Use atomic replace for supported backends 2022-05-09 22:31:30 +02:00
Michael Eischer 7559d2f105 Document repository version and minimum restic version 2022-05-09 22:31:30 +02:00
Michael Eischer 381bd94c6c prune: Add option to repack uncompressed data 2022-05-09 22:31:30 +02:00
Michael Eischer 5406743102 prune: Automatically repack uncompressed trees for repo v2
Tree packs are cached locally at clients and thus benefit a lot from
being compressed. Ensure this be having prune always repack pack files
containing uncompressed trees.
2022-05-09 22:31:30 +02:00
Alexander Neumann c8c0d659ec Add migration to compress all data 2022-05-09 22:31:30 +02:00
Alexander Neumann 8c244214bf Add tests for upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann a5f1d318ac Try to make repo upgrade migration more failsafe 2022-05-09 22:31:30 +02:00
Alexander Neumann 82ed5a3a15 Add repo upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann 3af6c180e4 Improve migrate command 2022-05-09 22:31:30 +02:00
Michael Eischer 92816fa966 init: Enable compression support by default 2022-05-09 22:31:30 +02:00
MichaelEischer ab49c14621
Merge pull request #3740 from MichaelEischer/fix-restore-size-stats
Fix restore size calculation for multiple snapshots
2022-05-09 21:39:45 +02:00
Michael Eischer 2c07f7fff3 stats: hardlinks only reduce restore within a snapshot
The `stats` command checks inodes to not count hardlinked files multiple
times into the restore size. This check applies across all snapshots and
not only within snapshots. As a result the result size was far too low
when calculating it for multiple snapshots and it would vary depending
on the order in which snapshots were listed.
2022-05-09 21:26:24 +02:00
Lorenz Bausch 9fb81c4246
Validate exclude patterns 2022-05-07 21:12:47 +02:00
Lorenz Bausch e7fd200237
Keep original pattern for later use 2022-05-07 21:08:09 +02:00
MichaelEischer cc8a03b1d0
Merge pull request #3735 from HenrikBengtsson/master
DOCS: Incorrectly used SI units when IEC units were meant (fix #3669)
2022-05-07 11:15:27 +02:00
Henrik Bengtsson 9bb532672a DOCS: Incorrectly used SI units when IEC units were meant (fix #3669) 2022-05-02 21:21:39 -07:00
MichaelEischer 26c333325c
Merge pull request #3715 from ema/master
doc: specify AWS Region via AWS_DEFAULT_REGION
2022-05-02 21:46:56 +02:00
Emanuele Rocca 38c0531b52 doc: specify AWS Region via AWS_DEFAULT_REGION
If no specific Region is mentioned in RESTIC_REPOSITORY, AWS defaults to
us-east-1. For this reason, users that follow the tutorial and create
their S3 bucket in any other region get the following error:

"Fatal: create repository at [...] client.BucketExists"

Explicitly specifying the AWS region name fixes the issue.
2022-05-02 21:26:58 +02:00
Alexander Neumann fb5b9370f3
Merge pull request #3728 from MichaelEischer/debug-examine-upload
Add support to reupload blobs in `debug examine`
2022-04-30 20:39:19 +02:00
Alexander Neumann ffbd48c0c6
Merge pull request #3481 from MichaelEischer/recover-enospace
Recover from no free space errors
2022-04-30 20:22:26 +02:00
Michael Eischer 95bcc9ea31 debug: Support pack ID prefixes in `debug examine` 2022-04-30 20:20:31 +02:00
Michael Eischer 2d6a943911 debug: Add switch to upload blobs extracted by debug examine
This simplifies salvaging a damaged pack file. Reuploading tree blobs
was previously not possible.
2022-04-30 20:20:31 +02:00
Alexander Neumann 9af499d8a4 Tidy go.sum 2022-04-30 20:16:09 +02:00
Alexander Neumann 2e3d23c1d7 Update github.com/klauspost/compress 2022-04-30 20:03:21 +02:00
Michael Eischer dbbeac7174 prune: Add unsafe option to recover from no free space
The new option allows prune to operate with nearly no scratch space by only removing
no longer necessary pack files and first deleting the index before
rebuilding it. By first deleting the index it becomes safe to just
delete no longer necessary pack files. However, as a downside there's
now the risk that the repository becomes inaccessible if prune fails.

To recover from that problem a user might have to manually delete the
repository index and then run (a full) `rebuild-index` again.
2022-04-30 19:21:07 +02:00
Michael Eischer cf5cb673fb repository: Use existing method to collect pack ids 2022-04-30 19:14:21 +02:00
Michael Eischer b335cb6285 repository: Refactor index IDs collection 2022-04-30 19:14:21 +02:00
MichaelEischer 9c047f170a
Merge pull request #3419 from DanielG/relax-file-modes
Enable admin to decide group access to repository files
2022-04-30 16:39:37 +02:00
Daniel Gröber f31b4f29c1 Use config file modes to derive new dir/file modes
Fixes #2351
2022-04-30 15:59:51 +02:00
Alexander Neumann 71c653f9e0
Merge pull request #3727 from MichaelEischer/changelog-3475
Add changelog for local/sftp connection limit
2022-04-30 14:05:21 +02:00
Michael Eischer 29a8f92967 Add changelog for local/sftp connection limit 2022-04-30 13:36:47 +02:00
MichaelEischer ac9324aeaf
Merge pull request #3666 from MichaelEischer/compression
Implement compression support
2022-04-30 11:49:05 +02:00
Alexander Neumann dc5adef255 Add documentation for --repository-version 2022-04-30 11:34:10 +02:00
Michael Eischer 4b01b06f2f repository: Test compressed blobs in StreamPack 2022-04-30 11:34:10 +02:00
Michael Eischer bcab548617 pack: slightly expand testing of compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer ec2b25565a repository: test uncompressedLength field and index example 2022-04-30 11:34:10 +02:00
Michael Eischer 9ffb8920f1 repository: run blackbox tests using old and new repo version 2022-04-30 11:34:10 +02:00
Michael Eischer abe5935693 repository: unify repository version-specific initialization
Mark the master index as compressed also when initializing a new
repository. This is only relevant for testing.
2022-04-30 11:34:10 +02:00
Alexander Neumann 8776031f96 Leave allocating slices to the decompress code 2022-04-30 11:34:10 +02:00
Alexander Neumann 5eb05a0afe Configure zstd encoder/decoder 2022-04-30 11:34:10 +02:00
Michael Eischer 2f36e044db Cleanup pack header check 2022-04-30 11:34:10 +02:00
Alexander Neumann 94dc9a0fa7 Amend changelog 2022-04-30 11:34:10 +02:00
Alexander Neumann 8b11b86383 Add option global --compression 2022-04-30 11:34:10 +02:00
Michael Eischer f38f457a64 Add basic changelog for compression support 2022-04-30 11:34:10 +02:00
Michael Eischer ba27d29d58 Print repository version when opening a repo 2022-04-30 11:34:10 +02:00
Michael Eischer 7132df529e repository: Increase index size for repo version 2
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer 2535524132 debug: Add support for compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer fda7bb0f09 debug: Reduce code duplication 2022-04-30 11:34:10 +02:00
Michael Eischer 66f9048bce repository: Alloc zstd encoder/decoder on demand 2022-04-30 11:34:10 +02:00
Michael Eischer fd05037e1a repository: recalibrate index batch allocation size 2022-04-30 11:34:10 +02:00
Michael Eischer 6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Michael Eischer 362ab06023 init: Add flag to specify created repository version 2022-04-30 10:07:42 +02:00
Michael Eischer 4b957e7373 repository: Implement index/snapshot/lock compression
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.

As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer 0957b74887 Misc design.rst cleanups 2022-04-30 10:07:42 +02:00
Alexander Neumann 270ed00d1f doc: Add repository compression support documentation
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2022-04-30 10:07:42 +02:00
Alexander Neumann 4e1ef7804a
Merge pull request #3717 from MichaelEischer/fix-stuck-repack
Fix stuck repack step
2022-04-30 09:50:43 +02:00
Alexander Neumann e4780d3956
Merge pull request #3718 from MichaelEischer/sftp-docs-fix
doc: sftp with password actually works
2022-04-23 20:27:27 +02:00
Alexander Neumann c183e35b5a
Merge pull request #3719 from MichaelEischer/read-write-order
Describe repository read/write order required by repository format
2022-04-23 15:23:00 +02:00
Michael Eischer 6f9e20a1bb doc: Describe repository read/write order 2022-04-23 15:21:02 +02:00
Michael Eischer f9219e8608 doc: sftp with password actually works 2022-04-23 11:57:36 +02:00
Michael Eischer 3b630d9998 add missing streamPacks changelog 2022-04-23 11:50:19 +02:00
Michael Eischer 566ac11c65 fix changelog name 2022-04-23 11:37:00 +02:00
Michael Eischer f5609d1d3c prune: Fail early if too few backend connections 2022-04-23 11:32:52 +02:00
Michael Eischer e597b99b55 repository: Reduce repack workers to prevent deadlock
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Michael Eischer ee627cd832 backend/mem: Actually enforce connection limit
This will allow tests to detect deadlocks related to the connections
limit.
2022-04-23 11:22:00 +02:00
Michael Eischer 4f97492d28 Backend: Expose connections parameter 2022-04-23 11:13:08 +02:00
rawtaz 07a565e6f7
Merge pull request #3716 from MichaelEischer/password-error-on-stderr
Print password error message on stderr
2022-04-21 01:40:54 +02:00
Michael Eischer bf7da7ff10 Print password error message on stderr
The password prompt itself is already printed on stderr.
2022-04-20 22:22:09 +02:00
Alexander Neumann dba47d29d5
Merge pull request #3711 from restic/doc-grouping
doc: Clarify and make grouping in forget more noticeable
2022-04-16 10:08:11 +02:00
Leo R. Lundgren 8ac7519fd5 doc: Clarify and make grouping in forget more noticeable 2022-04-16 01:13:13 +02:00
Alexander Neumann edc1a24a90
Merge pull request #3707 from duracell/patch-1
doc: fix missing "init" in rest-server example
2022-04-12 20:39:25 +02:00
Michael 9563e2f75c
doc: fix missing "init" in rest-server example 2022-04-12 18:21:53 +02:00
Alexander Neumann 7f133a28b2 Update VERSION file 2022-04-11 20:34:14 +02:00
Alexander Neumann 4f3b1f19cb Set development version for 0.13.1 2022-04-10 20:41:00 +02:00
Alexander Neumann 89ee1cf9ee
Merge pull request #3610 from MichaelEischer/windows-temp-files
Improve handling of temporary files on windows
2022-04-10 20:29:04 +02:00
Alexander Neumann a059ef90f8
Merge pull request #3702 from MichaelEischer/extend-config-error
Print used key name if config fails to load
2022-04-10 20:25:24 +02:00
Michael Eischer 4077a81b34 Add simple test for fs.TempFile on windows 2022-04-09 23:37:58 +02:00
Michael Eischer 9a3f1a9703 Simplify and comment TempFile implementation for windows 2022-04-09 23:37:58 +02:00
Michael Eischer c2aabb2686 Print used key name if config fails to load 2022-04-09 22:38:18 +02:00
MichaelEischer c60a5f00c9
Merge pull request #3675 from ItsMattL/update
Refactor file handing for self-update.
2022-04-09 21:55:56 +02:00
Matt LaPlante 0ba9d4ced7 Refactor file handing for self-update.
* Write new file payload to a temp file before touching the original
binary. Minimizes the possibility of failing mid-write and corrupting
the binary.
* On Windows, move the original binary out to a temp file rather than
removing it as the running binary is locked. Fixes issue #2248.
2022-04-09 21:40:33 +02:00
Alexander Neumann 04e054465a
Merge pull request #3475 from MichaelEischer/local-sftp-conn-limit
Limit concurrent operations for local / sftp backend
2022-04-09 21:33:00 +02:00
Alexander Neumann 1519e9f911
Merge pull request #3570 from MichaelEischer/list-snapshots-before-index
List snapshots before index
2022-04-09 21:15:49 +02:00
Michael Eischer ebab35581c Check in integration test that snapshots are listed before the index
As an exception prune is still allowed to load the index before
snapshots, as it uses exclusive locks. In case of problems with locking
it is also better to load snapshots created after loading the index, as
this will lead to a prune sanity check failure instead of a broken snapshot.
2022-04-09 12:27:27 +02:00
Michael Eischer 7b9ae91e04 copy: Load snapshots before indexes 2022-04-09 12:27:25 +02:00
Michael Eischer 47243176fa diff: list snapshots only once 2022-04-09 12:26:31 +02:00
Michael Eischer 5af828e3e6 add changelogs 2022-04-09 12:26:31 +02:00
Michael Eischer 4636c20397 test that TestFindListOnce calls List only once 2022-04-09 12:26:31 +02:00
Michael Eischer 9e12159230 Fix O(n) backend list calls in FindFilteredSnapshots
When resolving snapshotIDs in FindFilteredSnapshots either
FindLatestSnapshot or FindSnapshot is called. Both operations issue a
list operation to the backend. When for example passing a long list of
snapshot ids to `forget` this could lead to a large number of list
operations.
2022-04-09 12:26:31 +02:00
Michael Eischer 3d29083e60 copy/find/ls/recover/stats: Memorize snapshot listing before index
These commands filter the snapshots according to some criteria which
essentially requires loading the index before filtering the snapshots.
Thus create a copy of the snapshots list beforehand and use it later on.
2022-04-09 12:26:30 +02:00
Michael Eischer 2ec0f3303a backup/diff/dump/restore/stats: List snapshots before index
During a backup the index is written before the corresponding snapshots.
To ensure that a concurrent/later restic run can read a snapshot's data,
restic thus must first load the snapshots and only afterwards the index.
Otherwise it is not possible to ensure that the loaded index is recent
enough to cover all of the snapshot's data.
2022-04-09 12:24:09 +02:00
Michael Eischer ece06f125e sftp: Limit concurrent backend operations 2022-04-09 12:21:38 +02:00
Michael Eischer cd783358d3 local: Limit concurrent backend operations
Use a limit of 2 similar to the filereader concurrency in the archiver.
2022-04-09 12:21:38 +02:00
Michael Eischer 0b258cc054 backends: clean reader closing 2022-04-09 12:21:38 +02:00
Alex Duchesne 9e34c791c9 Better temp file cleanup on Windows. 2022-04-09 12:00:22 +02:00
Alexander Neumann 7d55b4f95e
Merge pull request #3701 from restic/rawtaz-doc-exclude-cachedir
doc: Link to CACHEDIR.TAG specification
2022-04-08 09:35:08 +02:00
rawtaz de4e3117eb
doc: Link to CACHEDIR.TAG specification 2022-04-08 03:07:27 +02:00
MichaelEischer 500079d265
Merge pull request #3689 from brightdroid/patch-1
added documentation for zsh autocompletion
2022-04-03 21:49:47 +02:00
Alexander Neumann 192288bc9c
Merge pull request #3696 from cqjjjzr/fix-win-procgrp
Fix rclone (scoop shim) and sftp issue due to detached console on Windows
2022-04-03 13:46:28 +02:00
Charlie Jiang d9c9415cfd Fix rclone (scoop shim) and sftp issue due to detached console on Windows 2022-04-03 17:53:17 +08:00
Alexander Neumann 59370b6062
Merge pull request #3695 from jernej-9/typo-fix
Fix a typo in the docs
2022-04-01 20:10:40 +02:00
Jernej Debevc 6e5731bf2f
Fix a typo in the docs 2022-04-01 17:16:55 +02:00
Alexander Neumann 305cd1e730
Merge pull request #3693 from greatroar/cast-btrfs-super-magic
Cast unix.Statfs_t.Type to int64 when checking for btrfs
2022-04-01 08:09:46 +02:00
greatroar c23c0f7c14 Cast unix.Statfs_t.Type to int64 when checking for btrfs
Fixes #3687. Uses the cast suggested by @MichaelEischer, except that the
contant isn't cast along, because it's untyped and will be converted by
the compiler as necessary.
2022-03-31 22:30:45 +02:00
Alexander Neumann 774c2e75ca
Merge pull request #3680 from restic/update-deps
Update all dependencies (except fuse), require Go 1.15
2022-03-30 21:36:08 +02:00
Alexander Neumann 66d50b72e3 Require Go 1.15 or later
The library github.com/golang-jwt/jwt/v4 requires the FillByte() method
of *big.Int, so we're raising the minimum Go version to 1.15.
2022-03-30 21:11:17 +02:00
Alexander Neumann 89d86a7933 Update all dependencies (except fuse) 2022-03-30 21:11:13 +02:00
Christoph Roeder 7510bdc247
added documentation for zsh autocompletion 2022-03-30 10:45:52 +02:00
Alexander Neumann f190d2e60e
Merge pull request #3686 from restic/fix-diff
Fix diff
2022-03-30 07:43:07 +02:00
Alexander Neumann 206550a246 Fix diff
Nodes in trees were always printed with a `+` in diff, regardless of
whether or not a dir was added or removed. Let's use the mode we were
passed in printDir().

Closes #3685
2022-03-29 21:05:11 +02:00
Alexander Neumann db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Alexander Neumann 724ace0e99
Merge pull request #3682 from MichaelEischer/refactor-code
Pack size calculation cleanup and misc other changes
2022-03-29 11:06:04 +02:00
Michael Eischer af31266b7d golangci-lint: replace deprecated golint with revive 2022-03-28 22:33:17 +02:00
Michael Eischer 2f81af6afa bloblru: Fix comment for New function 2022-03-28 22:25:25 +02:00
Michael Eischer 61e179ee78 switch to golang.org/x/term 2022-03-28 22:24:15 +02:00
Michael Eischer c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer fefe9f5c0e pack: Hide more implementation details 2022-03-28 22:12:16 +02:00
Michael Eischer a773cb6527 pack: cleanup header size calculation 2022-03-28 22:09:49 +02:00
Michael Eischer 6408686973 repository: Simplify Blob equality check 2022-03-28 22:09:49 +02:00
Michael Eischer 243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer d6db5a1fc2 archiver: Fix test
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Michael Eischer f78bd14e28 repository: Remove pack implementation details from MasterIndex 2022-03-28 22:09:49 +02:00
Michael Eischer dc3d77dacc repository: make saveAndEncrypt private 2022-03-28 22:09:49 +02:00
Michael Eischer 6877e7edbb repository: Rename LoadAndDecrypt to LoadUnpacked
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Michael Eischer 2e1613d4c6 errors: Ensure that errors.IsFatal(errors.Fatal("err")) == true
This fixes a few cases where restic output "Fatal: Fatal: [...]"
2022-03-28 22:09:49 +02:00
Alexander Neumann a08b95c497
Merge pull request #3513 from MichaelEischer/fast-copy
Speed-up copy command
2022-03-28 20:18:43 +02:00
Michael Eischer 537b4c310a copy: Implement by reusing repack
The repack operation copies all selected blobs from a set of pack files
into new pack files. For prune the source and destination repositories
are identical. To implement copy, just use a different source and
destination repository.
2022-03-26 20:47:15 +01:00
Alexander Neumann 4d5db61bd0
Merge pull request #3484 from MichaelEischer/stream-check-repack
Stream packs in `check --read-data` and during repacking
2022-03-26 20:46:17 +01:00
Alexander Neumann 4ab12f59a5 Set development version for 0.13.0 2022-03-26 20:10:07 +01:00
Alexander Neumann 40791fff64 Add version for 0.13.0 2022-03-26 20:09:59 +01:00
Alexander Neumann a53a4a23fd Update manpages and auto-completion 2022-03-26 20:09:59 +01:00
Alexander Neumann b567c08e80 Generate CHANGELOG.md for 0.13.0 2022-03-26 20:09:40 +01:00
Alexander Neumann 0ca89b6fec Prepare changelog for 0.13.0 2022-03-26 20:09:39 +01:00
Alexander Neumann d7e46c187a
Merge pull request #3678 from restic/changelogs
Review, reword and polish unreleased changelog entries
2022-03-26 19:44:30 +01:00
Alexander Neumann 6aefe3e111
Merge pull request #3677 from restic/3490-polish
check: Adjust help and documentation for check --read-data-subset
2022-03-26 19:34:51 +01:00
Leo R. Lundgren 03137a34db Review, reword and polish unreleased changelog entries 2022-03-26 13:01:53 +01:00
Leo R. Lundgren c7d637ec39 check: Adjust help and documentation for check --read-data-subset 2022-03-26 00:11:04 +01:00
Alexander Neumann 6087c4ad75
Merge pull request #3656 from lgommans/forget-security
forget: Update docs for readability and append-only considerations
2022-03-24 21:36:19 +01:00
Leo R. Lundgren cdf478c8f4 doc: More updates to forget documentation and security considerations 2022-03-23 23:12:19 +01:00
Luc Gommans 80969a6347 Update docs according to comments from MichaelEischer in PR #3656 2022-03-23 23:12:19 +01:00
Leo R. Lundgren 676d5d498c doc: Update forget security considerations and thread model 2022-03-23 23:12:19 +01:00
Luc Gommans 9c1d49e312 Document "forget" security considerations and add references
Removing data based on a policy when the attacker had the opportunity to
add data to your repository comes with some considerations. This is
added to the 060_forget.rst documentation.

That document is also updated to reflect that restic now considers
the current system time while running "forget".

References to the security considerations section are added:
- In `restic forget --help`
- In the threat model (design.rst)
- In the (030) setup section where an append-only setup is referenced

A reference is also to be added to the `rest-server` readme's
append-only paragraph (see my fork).

This commit also resolves a typo (amount->number for countable noun),
changes a password length recommendation into the metric that
actually matters when creating passwords (entropy) since I was editing
these doc files anyway, and updates the outdated copyright year in
`conf.py`.

Some wording in 060_forget (line 21..22) was changed to clarify what
"forget" and "prune" do, to try and avoid the apparent misconception
that "forget" does not remove any data.
2022-03-23 23:12:19 +01:00
Alexander Neumann e682f7c0d6 Add tests for StreamPack 2022-03-21 21:15:03 +01:00
Alexander Neumann ca1e2316cf
Merge pull request #3665 from MichaelEischer/sane-list-locks
list: Never lock the repository when listing lock files
2022-03-21 11:14:44 +01:00
Alexander Neumann 0b8b524f12
Merge pull request #3512 from MichaelEischer/cleaner-lock-refresh
Prevent lock refresh from leaving behind lots of stale locks
2022-03-21 11:10:37 +01:00
Alexander Neumann a350625554
Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann 32e61f2620
Update changelog/unreleased/issue-1106
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-03-21 11:04:04 +01:00
Alexander Neumann 8388f66c4c
Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann 0937008648
Merge pull request #3654 from MichaelEischer/limit-huge-tree-streams
Limit number of large tree blobs loaded in parallel by StreamTrees
2022-03-21 11:01:04 +01:00
Alexander Neumann 3a285f91bc
Merge pull request #2311 from vincentbernat/feature/negative-pattern
filter: ability to use negative patterns
2022-03-20 14:02:30 +01:00
Alexander Neumann 29a5778626 Improve wording 2022-03-20 13:46:16 +01:00
Michael Eischer 53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat 2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer 12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer 5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat 13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann 13aae82635
Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann 47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann 6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann 2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann 18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann 9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann 17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz 2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer daea461f15
Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer e0ab689ccd
Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer 7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber 49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz 44d543ede3
Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer 5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer 254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer ad4f4dbc7a
Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar 63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer 4a2d5a146d
Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer 1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer 8df246d0f3
Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
Michael Eischer 27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
gum3ng dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer fb4c5af5c4
Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer 18ec49ddfa
Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer 5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
Michael Eischer bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer 47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer 4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer 930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer 34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer 153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
MichaelEischer cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
MichaelEischer d8f58fb7bf
Merge pull request #3592 from vgt/jsondiff
Add json output for diff command
2022-02-07 21:33:30 +01:00
duritong a4786dda5a
Update doc/020_installation.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-02-06 21:31:18 +01:00
Michael Eischer aaa7f94139 Add changelog for O_NOATIME 2022-02-06 15:00:37 +01:00
Michael Eischer 6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00
greatroar 7080fed7ae Set O_NOATIME flag on Linux
Citing Kerrisk, The Linux Programming Interface:

    The O_NOATIME flag is intended for use by indexing and backup
    programs. Its use can significantly reduce the amount of disk
    activity, because repeated disk seeks back and forth across the
    disk are not required to read the contents of a file and to update
    the last access time in the file’s i-node[.]

restic used to do this, but the functionality was removed along with the
fadvise call in #670.
2022-02-06 15:00:34 +01:00
Michael Eischer 74f29ad09b diff: add basic test for json output format 2022-02-06 11:44:15 +01:00
Michael Eischer 5f34ad523f diff: fix test failure and add remark on quiet to changelog 2022-02-05 23:02:07 +01:00
MichaelEischer 58236ead12
Merge pull request #3619 from aneeshusa/avoid-time-travel-paradoxes-when-finding-parents
Avoid choosing parent snapshot newer than time of current snapshot
2022-02-05 22:51:24 +01:00
Michael Eischer 8ae4d86a84 rename snapshot timestamp filter variable 2022-02-05 22:42:38 +01:00
mh 3f0184ba2a add a note about installation via epel 2022-02-05 22:18:33 +01:00
rawtaz 90473ea9ff
Merge pull request #3638 from leogott/patch-1
Documentation: Update msys2 wiki url after move
2022-01-27 17:07:45 +01:00
Leona 'leo' Gottfried 4e84e8ab3f
Update msys2 wiki url after move
https://github.com/msys2/msys2/wiki/Porting was permanently moved and redirects to https://www.msys2.org/
I substituted the new location of the /wiki/Porting page in the docs
2022-01-27 11:09:50 +01:00
Greg 2e9180638e Fix test failures on Solaris
Add exceptions for symlinks, sticky bits, and device nodes in the same places where the BSDSs and/or Darwin have them.
2022-01-25 18:05:56 -08:00
Aneesh Agrawal 058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
Aneesh Agrawal 502fc3281c Add CONTRIBUTING.md docs to not edit man pages
Document this code review feedback I got for other contributors.
2022-01-23 23:54:24 -05:00
David Vogt 77c850148a Add json output for diff command 2022-01-23 23:22:26 +01:00
MichaelEischer df89aa0087
Merge pull request #3623 from invine/issue-3464
Skip lock file creation on forget with --no-lock and --dry-run
2022-01-23 18:12:34 +01:00
Pavel Frolov 792523b28b [issue 3464] skip lock creation in case of dry-run 2022-01-23 06:44:41 +03:00
MichaelEischer f0a8182493
Merge pull request #3626 from fkusche/issue-3620-doc-unreferenced-packs
Update documentation regarding unreferenced packs
2022-01-22 15:15:25 +01:00
Florian Kusche 6183d0be53 Update output of restic check 2022-01-16 12:34:20 +01:00
rawtaz 7f6fc78f95
Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
rawtaz abfbacf3d3
Merge pull request #3591 from MichaelEischer/prune-fix-max-repack
prune: Handle --max-repack-size=0 as expected
2022-01-13 03:53:20 +01:00
Florian Kusche b0c1d0f9cd Update documentation regarding unreferenced packs
Also removes an unnecessary space at the end of the last line.
2022-01-09 11:45:03 +01:00
rawtaz 8b6fe845d4
Merge pull request #3618 from mattxtaz/master
Add missing colon in prune stats output and realign the fields
2022-01-06 23:07:56 +01:00
mattxtaz 6ff32ee4d3 Add missing colon in prune stats output and change padding to 14 chars to align the fields 2022-01-06 21:15:15 +00:00
rawtaz 2ff3b7d69c
Merge pull request #3615 from gum3ng/issue_3558
doc: Add a FAQ section for invalid Windows filenames
2022-01-02 18:06:02 +01:00
gum3ng 9589de16db [issue 3558]: Add a FAQ section for invalid Windows filenames 2022-01-02 22:24:00 +05:30
MichaelEischer 2c3e5d943d
Merge pull request #3593 from DarkKirb/parallelize-restic-copy
Parallelize blob upload/download for restic copy
2021-12-29 22:31:54 +01:00
Charlotte 🦝 Delenk e2bb384a60
Parallelize blob upload/download for restic copy
Currently restic copy will copy each blob from every snapshot serially,
which has performance implications on high-latency backends such as b2.

This commit introduces 8x parallelism for blob downloads/uploads which
can improve restic copy operations up to 8x for repositories with many
small blobs on b2.

This commit also addresses the TODO comment in the copyTree function.

Related work:

A more thorough improvement of the restic copy performance can be found
in PR #3513
2021-12-29 18:59:09 +01:00
MichaelEischer e5985e0d63
Merge pull request #3602 from cqjjjzr/fix-rclone-sigint
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever (#3601)
2021-12-29 16:56:37 +01:00
MichaelEischer 8832837a8a
Merge pull request #3607 from jtagcat/quieter_cleanup
logging: quiet 'removing n old cache dirs'
2021-12-29 16:06:35 +01:00
jtagcat f92130d878 logging: quiet 'removing n old cache dirs'
Closes #3595

Choosing to include `stdoutIsTerminal()` as:
 - all other instances with `!opts.JSON` do so
 - this likely will not affect anything, especially when autorun
 - this seems to not be a meaningful enough summary
     to include in auto-backup reports

JSON is still likely not guaranteed to work and this is a suboptimal
  solution to this. Ideally, #1804 should refactor all print statements,
  and define+document(+handle) when stdoutIsTerminal() should be used.
  Else, it may end up more inconsistent and bulky
  (duplicate lines, longer files).
2021-12-29 01:08:29 +02:00
Charlie Jiang a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
rawtaz e6e51b84ac
Merge pull request #3605 from gum3ng/issue_3463
Improve clarity of error message in restic ls command
2021-12-27 21:45:23 +01:00
MichaelEischer c5c3dfe10f
Merge pull request #3590 from metalsp0rk/documentation-enhance
Document Safe Passwords and Clarify B2 App Key information
2021-12-27 20:21:43 +01:00
Kyle Brennan 19ec4d8f17
Document safe passwords. Fix #2238 2021-12-27 10:43:18 -08:00
Kyle Brennan 47ecd950b8
Enhance details about user application keys. Fix #2672 2021-12-27 10:43:15 -08:00
MichaelEischer 051cc7ce71
Merge pull request #3589 from metalsp0rk/copy-no-lock
Make Copy respect no lock
2021-12-27 19:11:02 +01:00
Michael Eischer 64e733f3d6 tweak copy --no-lock changelog 2021-12-27 18:22:25 +01:00
Gautam Menghani 017614c41a [#issue 3463] Improve clarity of error message in restic ls command 2021-12-27 22:42:27 +05:30
Michael Eischer 0cfdb82ea4 prune: Handle --max-repack-size=0 as expected
Previously the flag was ignored and `--max-repack-size=1` had to be
used.
2021-12-27 15:48:56 +01:00
Kyle Brennan d5ed5da85c
update changelog for copy --no-lock 2021-12-03 12:16:40 -08:00
Kyle Brennan 8eb83029a8
Make copy honor --no-lock 2021-12-03 09:50:28 -08:00
MichaelEischer 882d58abce
Merge pull request #3163 from palbr/patch-1
Add PGP fingerprint to 020_installation.rst
2021-11-19 23:58:50 +01:00
Peter Albrecht 8de4401bb5
Changed URL for key-file
The keyfile provided by restic's own webserver (https://restic.net) should be
more stable than relying on public keyservers. So I changed the URL to the
GPG keyfile, as recommended by MichaelEischer.
2021-11-19 15:47:59 +01:00
MichaelEischer f7a9b90eb9
Merge pull request #3573 from magandrez/feat/ls-report-symbolic-notation
Provide mode in symbolic notation for `restic ls --json`
2021-11-18 21:29:04 +01:00
MichaelEischer aa214f99b4
Merge pull request #3565 from mathstuf/doc-missing-metadata
doc: mention metadata that is not backed up currently
2021-11-18 21:28:55 +01:00
MichaelEischer 4a25bbaed3
Merge pull request #3578 from adsultana/patch-1
docs: Fix link to "help wanted" issues
2021-11-18 21:20:30 +01:00
Michael Eischer 583edc39b8 doc: reorder backup metadata exceptions 2021-11-18 21:17:38 +01:00
Manuel González 212b2f651f
Add file mode in symbolic notation to `ls --json`
This aligns `restic ls --json` with `restic find --json`, utilizing the same
naming.
2021-11-16 19:45:50 +02:00
Andrew Sultana 15ab96ecd6
docs: Fix link to "help wanted" issues 2021-11-16 00:03:34 +00:00
Ben Boeckel d71afb3d32 doc: mention metadata that is not backed up currently
See: #3497
See: #1622
See: #2075
2021-11-13 18:40:26 -05:00
MichaelEischer 4bf05d91a1
Merge pull request #3571 from garrmcnu/blazer-unknown-authority
Update Backblaze B2 blazer module
2021-11-13 22:46:49 +01:00
MichaelEischer de3afc1005
Merge pull request #3574 from gurjeet/rename_aws_s3_to_amazon_s3
Use S3's proper product name, Amazon S3
2021-11-13 22:30:32 +01:00
Michael Eischer 2ea998f70e Add PR link to changelog 2021-11-13 22:25:54 +01:00
Garry McNulty e8fa3855e7 Update blazer
If a request fails with "x509: certificate signed by unknown authority",
the B2 backend now returns the error without retrying the request.

Closes #3556
Closes #2355
2021-11-13 22:25:54 +01:00
Gurjeet Singh 34a6a24544 Use S3's proper product name, Amazon S3
Per Amazon's product page [1], S3 is officially called "Amazon S3". The
restic project uses the phrase "AWS S3" in some places. This patch
corrects the product name.

[1]:https://aws.amazon.com/s3/
2021-11-13 22:21:06 +01:00
MichaelEischer 1d8a0b06cb
Merge pull request #3575 from MichaelEischer/adjust-http2-canary-test
rest: Adjust http2-missing-eof test to golang >= 1.17.3
2021-11-13 22:13:24 +01:00
Michael Eischer 50053a85d3 rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer f1cfb97237
Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer cb81ee9396
Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer b0e64deb27 rclone: Fix timeout calculation 2021-11-07 17:49:33 +01:00
phcreery 43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
rawtaz 1b152a2c4d
Merge pull request #3568 from MichaelEischer/fix-local-fd-leak
local: Fix fd leak when encountering files directly inside data/
2021-11-07 01:36:48 +01:00
MichaelEischer 15cc3c0e23
Merge pull request #3566 from mathstuf/check-progress-output-stomping
check: wait for progress bar output
2021-11-06 23:34:49 +01:00
kitone 5904f80cfa restic cache should display the name of the cache without shortening it in the case of the restic check 2021-11-06 20:18:51 +01:00
Ben Boeckel 4d579c4387 check: wait for progress bar output
Further code will also output to the terminal and the bar's cursor
positioning causes its output to overlap with the remaining output in a
racy way.

Fixes: #3344
2021-11-06 15:05:09 -04:00
Michael Eischer 15d7313387 local: Fix fd leak when encountering files directly inside data/ 2021-11-06 19:44:57 +01:00
MichaelEischer 6c84ea1412
Merge pull request #3548 from gum3ng/issue_3490
Support for specifying amount of data in read-data-subset
2021-11-05 23:28:11 +01:00
MichaelEischer 78c7dd53ef
Merge pull request #3526 from greatroar/dump-refactor
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
MichaelEischer a34bfa8269
Merge pull request #3562 from restic/rawtaz-mount-usage
mount: Improve usage information when mounted
2021-11-05 22:10:32 +01:00
kitone 0425a30420 add changelog issue-3382 2021-11-04 15:13:48 +01:00
kitone 1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
Gautam Menghani 836fbb9133 [#issue 3490] Support for specifying file size in read-data-subset 2021-11-02 15:25:46 +05:30
greatroar c71729dfc4 Refactor internal/dump + concurrent load/write
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Leo R. Lundgren 711ceb0109 mount: Improve usage information when mounted 2021-11-01 20:59:20 +01:00
MichaelEischer 829c0a67af
Merge pull request #3520 from nxt-engineering/Docker
Docker Multistage build
2021-10-24 17:37:54 +02:00
Alexander Neumann fb5d9345a7
Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone 95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer 257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer 46d08d9404
Merge pull request #3535 from jtagcat/writingverbose
Change: selfupdate: 'writing restic to' as verbose
2021-10-09 22:41:51 +02:00
jtagcat a7853057ab Change: selfupdate: 'writing restic to' as verbose
Running restic self-update --quiet no longer
prints "writing restic to /usr/local/bin/restic".

The only output printed with -q is failures or
"successfully updated restic to version 0.12.1"

https://github.com/restic/restic/pull/3535

fix test fail: changelog title can't end with `.`

shorten changelog title
2021-10-09 23:24:18 +03:00
MichaelEischer eb282532dc
Merge pull request #3534 from jtagcat/patch-1
docs/example: useradd: restic now system account
2021-10-09 20:42:43 +02:00
MichaelEischer f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
MichaelEischer 58e8b34633
Merge pull request #3532 from garrmcnu/s3-credentials-config
s3: Add warning if key ID or secret is empty
2021-10-09 19:32:32 +02:00
MichaelEischer a02cea6e83
Merge pull request #3539 from jnoxon/fix-ec2-metadata-v2
get iam roles working with ec2 metadata v2
2021-10-07 23:00:27 +02:00
Garry McNulty 708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
Jeff Noxon 6f4b5ab8d1 get iam roles working with ec2 metadata v2 2021-10-03 20:06:55 -05:00
greatroar 634a9c162d Check cap instead of len in bloblru
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
jtagcat 632ca2ef52 docs/example: useradd: restic now system account
and: use /sbin/nologin
and: use long flags in the useradd command
and: download v0.12.1 instead of v0.9.6
2021-10-03 00:51:06 +03:00
MichaelEischer 24088f8307
Merge pull request #3528 from greatroar/tree-insert
internal/restic: Don't allocate in Tree.Insert
2021-09-27 22:22:03 +02:00
greatroar c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer 78dac2fd48
Merge pull request #3523 from greatroar/swift-v2
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer 5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer a5e103a212
Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
Uli Martens 1ebcb1d097 Add changelog entry for PR #3508 2021-09-24 15:45:09 +02:00
greatroar fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens 718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar 4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
MichaelEischer cc110c42e6
Merge pull request #2657 from mansam/add-skip-tls-verification-flag
Add --insecure-tls flag to disable SSL cert verification
2021-09-22 21:40:01 +02:00
Sam Lucidi 897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
cimnine 4a95af5290 GitHub action for Docker build 2021-09-21 08:23:33 +02:00
cimnine f28c8bc1c2 Multistage Docker build 2021-09-21 08:23:33 +02:00
MichaelEischer 1827b16ade
Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar 8b758c78a3 Require Go 1.14 to build 2021-09-19 16:18:19 +02:00
greatroar 8d2996eaaa Replace siphash by hash/maphash
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer 58efe21eca
Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
MichaelEischer 71fcf48533
Merge pull request #2594 from greatroar/concurrent-restore-verify
Concurrent restore --verify
2021-09-19 14:52:31 +02:00
Michael Eischer 921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer e62d4f622f Reword changelog 2021-09-19 14:41:07 +02:00
Michael Eischer 2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar 2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar 556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar 92ae951ffa Report timing from restic restore --verify 2021-09-19 12:53:09 +02:00
greatroar 973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
MichaelEischer ef5672a902
Merge pull request #3509 from ajaspers/patch-1
Update instructions for reproducing build
2021-09-12 21:40:37 +02:00
ajaspers c0eddc9969 Update instructions for reproducing build
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.

Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
2021-09-12 09:02:57 -07:00
Michael Eischer fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer 503d4c3e2f Add changelog 2021-09-12 16:00:49 +02:00
Michael Eischer cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer a144c986f2 backup: Reenable JSON status updates with redirected output
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.

This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
Michael Eischer d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00
Michael Eischer 77b129ec74 ui: remove unused Summary field 2021-09-12 15:25:58 +02:00
Michael Eischer 3024239e40 ui/json: restore summary output 2021-09-12 15:25:58 +02:00
Michael Eischer 5ccf583b8a ui: restore a few comments 2021-09-12 15:25:58 +02:00
Michael Eischer 80cbaf6d38 ui: Simplify ReportTotal methods 2021-09-12 15:25:58 +02:00
Dan Willoughby 448419990c Refactor backup progress
Move the shared logic into the progress

Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
Andy 7baa9a570d
document AWS_PROFILE support
Since restic (or whatever library it is using) seems to respect/use AWS_PROFILE, it's worth documenting this.
2021-09-10 14:06:46 +01:00
Alexander Neumann bf9c8771a4
Merge pull request #3482 from MichaelEischer/changelog-3429
Add changelog for #3246 and #3429
2021-09-08 09:15:40 +02:00
Michael Eischer 5e84f38f31 Add changelog for #3246 2021-09-07 21:18:11 +02:00
Alexander Neumann 8fe122d675
Merge pull request #3488 from MichaelEischer/rebuild-broken-index
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
Alexander Neumann 74c47f1f12
Merge pull request #3502 from restic/rawtaz-issue-519
doc: Add note about scheduling to backup section
2021-09-07 16:58:17 +02:00
MichaelEischer fa5ca8af81
Merge pull request #2856 from aawsome/remove-readahead
Simplify cache logic
2021-09-04 20:25:49 +02:00
MichaelEischer b45d88e124
Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
MichaelEischer bc4cbd775b
Merge pull request #2880 from aawsome/enhance-recover
Improve recover command
2021-09-03 23:40:43 +02:00
MichaelEischer a29777f467
Merge pull request #3501 from greatroar/printprogress
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
Alexander Weiss bce87922c0 Improve recover
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
Alexander Weiss 81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
greatroar 7f0aa49f45 cmd/restic: Streamline progress printing
* PrintProgress no longer does unnecessary Sprintf calls, and performs
  fewer allocations in general
* newProgressMax's callback checks whether the terminal supports
  line updates once instead of once per call
* the callback looks up the terminal width once per call instead of
  twice (on Windows)
* the status shortening now uses the Unicode-aware version from
  internal/ui/termstatus (future-proofing)
2021-09-03 11:48:22 +02:00
greatroar 5aaa3e93c1 internal/ui/termstatus: Optimize and publish Truncate
name               old time/op  new time/op  delta
TruncateASCII-8     347ns ± 1%    69ns ± 1%  -80.02%  (p=0.000 n=9+10)
TruncateUnicode-8   447ns ± 3%   348ns ± 1%  -22.04%  (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
MichaelEischer ec2e3b260e
Merge pull request #3499 from greatroar/wrappedconn-pointer
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
rawtaz 26914abe62
doc: Add note about scheduling to backup section
Explains that restic doesn't have built-in scheduling
and mentions a few keywords one can search for.
2021-08-29 22:03:22 +02:00
greatroar 950b818274 rclone: Return one fewer value from run 2021-08-26 18:12:08 +02:00
Juergen Hoetzel defe19fdf6 Quote snapshot prefix in error string 2021-08-25 16:11:28 +02:00
Juergen Hoetzel 409e4936af Improve snapshot filter warning message
Include the root-cause why the snapshot prefix is ignored.
2021-08-25 15:46:21 +02:00
MichaelEischer 10b39d7591
Merge pull request #3487 from andreaso/rest-proto-readme-link
Fix REST protocol link in README file
2021-08-22 18:47:32 +02:00
Michael Eischer 194ed19557 Add changelog 2021-08-22 18:29:58 +02:00
Michael Eischer 877fc9f352 rebuild-index: test that invalid indexes are skipped and removed 2021-08-22 18:24:19 +02:00
Michael Eischer 64258a2c2a rebuild-index: Ignore broken index files
Previously, an e.g. truncated index could prevent rebuild-index from
working.
2021-08-22 18:23:47 +02:00
Andreas Olsson c520672982
Fix REST protocol link in README file
1. All other document links goes to the _Read the Docs_ site
2. The GitHub reStructuredText renderer doesn't appear to do includes,
   making for a rather empty read.
2021-08-22 13:16:51 +02:00
Alexander Neumann 9374c3ce81
Merge pull request #3485 from lbausch/fix_typo
Fix typo in changelog
2021-08-21 13:16:20 +02:00
Lorenz Bausch 4d56b34096
Fix typo in changelog 2021-08-21 12:16:04 +02:00
Alexander Neumann 66382b2861 Update Go to 1.17 2021-08-17 21:38:34 +02:00
Michael Eischer 1fab5892b5 Add changelog for #3429 2021-08-16 17:42:16 +02:00
MichaelEischer c898f7a6bf
Merge pull request #3479 from BUFU1610/patch-1
fixed --keep-within-X options in example
2021-08-15 20:06:26 +02:00
BUFU 7659790923 fixed --keep-within-X options in example
The order of the words was wrong in the example, i.e. fixed to --keep-within-hourly from --keep-hourly-within.
2021-08-15 19:56:22 +02:00
MichaelEischer ecf34783ef
Merge pull request #3480 from MichaelEischer/fix-rest-tests
Fix rest backend tests
2021-08-15 18:59:02 +02:00
Michael Eischer 68370feeee backends: Remove TestSaveFilenames test
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
Michael Eischer 574c83e47f rest: Fix test to use paths which are the sha256 sum of the data 2021-08-15 18:19:43 +02:00
Michael Eischer e6a5801155 rest: Fix test backend url
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
Alexander Neumann d90efd7704 Fix test 2021-08-09 10:30:10 +02:00
Alexander Neumann 9fe5a87785
Merge pull request #3429 from MichaelEischer/safe-key-change
key: Check that new key works before deleting the old one
2021-08-09 10:07:15 +02:00
Alexander Neumann 7f1608dc77
Merge pull request #3246 from restic/content-hash-for-upload
Calculate content hashes during upload
2021-08-08 17:24:08 +02:00
Michael Eischer f4c5dec05d backend: test that a wrong hash fails an upload 2021-08-04 22:17:46 +02:00
Michael Eischer 7c1903e1ee panic if hash returns an error
Add a sanity check that the interface contract is honoured.
2021-08-04 22:17:46 +02:00
Michael Eischer 51b7e3119b mem: calculate md5 content hash for uploads
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
2021-08-04 22:17:46 +02:00
Michael Eischer a009b39e4c gs/swift: calculate md5 content hash for upload 2021-08-04 22:17:46 +02:00
Michael Eischer 1d3e99f475 azure: check upload using md5 content hash
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
2021-08-04 22:17:46 +02:00
Michael Eischer 9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
Michael Eischer ee2f14eaf0 s3: enable content hash calculation for uploads 2021-08-04 22:12:12 +02:00
MichaelEischer 553ea36ca6
Merge pull request #2838 from greatroar/cache-conflicts
Make cache concurrency-safe
2021-08-04 22:11:50 +02:00
greatroar 6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
MichaelEischer 1b1a2115fa
Merge pull request #3436 from greatroar/local-save-tmp
Save files under temporary name in local backend
2021-08-04 22:01:53 +02:00
Michael Eischer 65908647e3 Add changelog 2021-08-04 21:51:53 +02:00
greatroar 81e2499d19 Sync directory to get durable writes in local backend 2021-08-04 21:51:53 +02:00
greatroar 195a5cf996 Save files under temporary name in local backend
Fixes #3435.
2021-08-04 21:51:53 +02:00
MichaelEischer bc97a3d1f9
Merge pull request #3300 from aawsome/backup-dryrun
backup: add --dry-run/-n flag
2021-08-04 21:50:38 +02:00
Michael Eischer 702cff636f Add use case to changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss 780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
erin 4126435663 resolve rawtaz's review comments
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
Alexander Weiss d107a2cfdf Separate dry run tests 2021-08-04 21:19:29 +02:00
Alexander Weiss 38a8a48a25 Simplify dry run backend 2021-08-04 21:19:29 +02:00
Ryan Hitchman 77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
MichaelEischer 533ac4fd95
Merge pull request #3467 from greatroar/wrappedconn-pointer
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
Alexander Neumann 7049f1cbfc Set development version for 0.12.1 2021-08-03 11:45:39 +02:00
Alexander Neumann dc7a8aab24 Add version for 0.12.1 2021-08-03 11:45:36 +02:00
Alexander Neumann 94983a1f36 Update manpages and auto-completion 2021-08-03 11:45:36 +02:00
Alexander Neumann a92faca10e Generate CHANGELOG.md for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann b19cd8c50f Prepare changelog for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann b862732318
Merge pull request #3468 from restic/rework-changelog
Reword changelog entries
2021-08-03 10:06:56 +02:00
Leo R. Lundgren cb844e7136 Polish changelog entries 2021-08-03 00:01:09 +02:00
Alexander Neumann b7fe1fe6b4 Reword changelog entries 2021-08-03 00:01:09 +02:00
MichaelEischer c98bbdcdbe
Merge pull request #3457 from systemmonkey42/feature/untagged
Feature to match untagged snapshots only when listing or forgetting
2021-08-02 23:19:39 +02:00
David le Blanc 326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
greatroar fa3eed1998 Use rclone.wrappedConn by pointer
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
MichaelEischer 5571c3f7fd
Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann d8ea10db8c rest: Rework handling HTTP2 zero-length replies bug
Add comment that the check is based on the stdlib HTTP2 client. Refactor
the checks into a function. Return an error if the value in the
Content-Length header cannot be parsed.
2021-07-31 17:12:24 +02:00
Alexander Neumann 77551597b2
Merge pull request #3416 from torfason/keep-hourly-within
Keep hourly within
2021-07-30 10:36:41 +02:00
Alexander Neumann 92f293cd0b
Merge pull request #3427 from MichaelEischer/find-packs-from-index
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
Magnus Thor Torfason 2081bd12fb forget: Ensure future snapshots do not affect --keep-within-*
Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)
2021-07-24 16:14:43 +00:00
Magnus Thor Torfason 74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Alexander Neumann c707d71b72
Merge pull request #3401 from MichaelEischer/goroutine-shutdown-cleanups
Goroutine shutdown cleanups
2021-07-11 16:32:28 +02:00
Alexander Neumann 691866ce43
Merge pull request #3452 from MichaelEischer/add-s390x-releases
Add release binaries for linux/s390x
2021-07-11 16:30:25 +02:00
Alexander Neumann efd918c59e
Merge pull request #3454 from MichaelEischer/update-dependencies
Update dependencies
2021-07-11 16:29:31 +02:00
Michael Eischer 7d28006e2e Add changelog 2021-07-10 22:39:01 +02:00
Michael Eischer 0880afe67b Use our generate command instead of cobra's complete command 2021-07-10 19:44:18 +02:00
Michael Eischer 100baf74c0 Update cobra
The most noteworthy change seems to be

https://github.com/spf13/cobra/pull/1192
Have Cobra create 'completion' command automatically
2021-07-10 19:43:14 +02:00
Michael Eischer c733ae6b16 Readd and update indirect dependencies
The azure-sdk-for-go is the only remaining module without a go.mod file.
Thus we only need indirect dependencies for it. Remove all other
indirect dependencies.
2021-07-10 19:21:43 +02:00
Michael Eischer 989b398fee Misc library updates
The most noteworthy change is the xattr update which includes
https://github.com/pkg/xattr/pull/54 that adds xattr support for solaris
and illumos
2021-07-10 19:04:47 +02:00
Michael Eischer bbc8146934 Update minio/sha256-simd
Apparently the standard Go sha256 implementation is now faster than the
assembly implementation. The library now only adds support for SHA
extensions available in some processors.

See https://github.com/minio/sha256-simd/pull/57 for more details.
2021-07-10 18:34:16 +02:00
Michael Eischer aa22ebac69 Update backend dependencies
Possibly interesting changes:

https://github.com/Azure/azure-sdk-for-go/pull/14521
Retry on HTTP client errors

https://github.com/minio/minio-go/pull/1452
fix: make sure getObject returns error on truncated responses

f5854403a9
http2: close Transport connection on write errors
2021-07-10 18:27:13 +02:00
Michael Eischer 097ed659b2 rest: test that zero-length replies over HTTP2 work correctly
The first test function ensures that the workaround works as expected.
And the second test function is intended to fail as soon as the issue
has been fixed in golang to allow us to eventually remove the
workaround.
2021-07-10 17:22:42 +02:00
Michael Eischer 185a55026b rest: workaround for HTTP2 zero-length replies bug
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer 495831d53c add release binaries for linux/s390x 2021-07-10 00:10:24 +02:00
Michael Eischer 3442dc87fb find: Address review comments 2021-07-06 21:04:34 +02:00
Michael Eischer a81f34ae47 Add changelog 2021-07-06 21:04:34 +02:00
Michael Eischer 95b44490a0 find: search blob ids for pack in index if pack is missing
If a pack file is missing try to determine the contained pack ids based
on the repository index. This helps with assessing the damage to a
repository before running `rebuild-index`.
2021-07-06 21:04:34 +02:00
Michael Eischer 3caab3c7ac find: Print not found pack files 2021-07-06 21:04:34 +02:00
Michael Eischer 40745b4f82 find: stop file listing after resolving all pack files 2021-07-06 21:04:34 +02:00
Michael Eischer 6c01078f3d find: support resolving multiple pack ids to blobs
Just passing the list of blobs to packsToBlobs would also work in most
cases, however, it could cause unexpected results when multiple pack
files have the same prefix. Forget found prefixes to prevent this.
2021-07-06 21:04:34 +02:00
Leo R. Lundgren 790294dc26 contributing: Change freenode to libera 2021-07-04 11:05:36 +02:00
Alexander Neumann 30d968b0e4
Merge pull request #3449 from MichaelEischer/fix-restore-retries
restore: Correctly handle partial pack download errors
2021-07-04 10:55:09 +02:00
Michael Eischer 43b82d69b4 Add changelog 2021-06-29 21:27:00 +02:00
Michael Eischer bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
rawtaz 58be5172ff
Merge pull request #3437 from MichaelEischer/fix-prune-output
prune: Add missing newlines in error descriptions
2021-06-28 00:56:51 +02:00
MichaelEischer cb6fd281a0
Merge pull request #3442 from restic/rawtaz-doc-verbose
doc: Correct position of --verbose in backup docs
2021-06-27 18:19:37 +02:00
rawtaz eb61de7b3a
Merge pull request #3287 from tjrana/clean-up-pr-template
Clean up PR template
2021-06-25 21:06:50 +02:00
rawtaz 98a88b483d
doc: Correct position of --verbose in backup docs 2021-06-25 19:26:46 +02:00
MichaelEischer 6a4c1ed50d
Merge pull request #3438 from MichaelEischer/fast-fuse-changelog
Add changelog for #3426
2021-06-20 15:05:19 +02:00
Michael Eischer 409306db2b Add changelog for #3426 2021-06-20 14:46:37 +02:00
Michael Eischer aad8864835 prune: Add missing newlines in error descriptions 2021-06-20 14:25:40 +02:00
MichaelEischer c1eb7ac1a1
Merge pull request #3420 from greatroar/local-errs
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar e5f0f67ba0 Modernize error handling in local backend
* Stop prepending the operation name: it's already part of os.PathError,
  leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
  permitted".

* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
Alexander Neumann 45eb30388f
Merge pull request #3426 from MichaelEischer/fast-fuse
mount: enable fuse readahead
2021-06-14 10:27:41 +02:00
Michael Eischer 454b6d608e key: Add test that failed key saves don't break the repository 2021-06-13 13:46:48 +02:00
TJ Rana a61a0255a8
Modify pull request template 2021-06-12 19:34:46 -04:00
Michael Eischer 6add186867 key: Check that a new key file actually works 2021-06-12 23:09:19 +02:00
MichaelEischer a476752962
Merge pull request #3421 from greatroar/s3-fileinfo
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer e8d20ea32c
Merge pull request #3409 from greatroar/lchown-mknod
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
Michael Eischer fe43f53528 mount: enable fuse readahead
Apparently readahead was disabled by default. Enable readahead with the
Linux default size of 128kB. Larger values seem to have no effect.
This can speed up reading from the fuse mount by at least factor 5.

Speedup for a 1G random file stored in a local repository:
(Only one result shown, but times were quite stable, restarted restic
after each command)
$ dd if=/dev/urandom bs=1M count=1024 of=rand
$ shasum -a 256 tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  tmp/rand

before: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m18.294s
user    0m4.522s
sys     0m3.305s

before: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m14.924s
user    0m0.000s
sys     0m4.625s

after:  $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m6.106s
user    0m3.115s
sys     0m0.182s

after:  $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m3.096s
user    0m0.017s
sys     0m0.241s
2021-06-12 17:07:30 +02:00
greatroar 0d4f16b6ba Return s3.fileInfos by pointer
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar 0666c4d244 Make restic.{lchown,mknod} regular functions
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes
Various small code cleanups
2021-05-24 11:30:31 +02:00
rawtaz 2daf033156
Merge pull request #3403 from MichaelEischer/fast-cat
cat: only load index if really necessary
2021-05-24 11:02:10 +02:00
MichaelEischer 5dad45f005
Merge pull request #3312 from soraxas/master
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
MichaelEischer 7eb6372123
Merge pull request #3257 from greatroar/ls-json-empty
ls: print "size":0 for empty files in JSON
2021-05-23 13:49:56 +02:00
Michael Eischer 61b368ddea cat: only load index if really necessary 2021-05-23 13:11:55 +02:00
Michael Eischer fd8bce8184 backup: cleanly shutdown goroutines on error 2021-05-23 13:02:44 +02:00
greatroar d7322a5f36 restic ls --json: print "size":0 for empty files 2021-05-21 21:06:00 +02:00
Tin Lai 9cc1ecdd45
bump cobra and add completions for fish
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
Alexander Neumann af3de702c7
Merge pull request #3332 from restic/debug-1999
Merge `debug examine` to salvage damaged pack files
2021-05-18 09:38:40 +02:00
Alexander Neumann 226cd8d4d1
Merge pull request #3310 from MichaelEischer/copy-unstable-tree
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
Alexander Neumann 4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer 5767c65c62 delete: properly close fileChan if context is canceled 2021-05-17 21:05:54 +02:00
Michael Eischer 75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer 5a87a0ba0a find: use Str() to format short ids 2021-05-17 20:56:49 +02:00
Alexander Neumann 38ccddc84f
Merge pull request #3399 from oli-h/oli-h-patch-060_forget.rst
Sudden find: fix within first "forget" example
2021-05-17 11:39:36 +02:00
Alexander Neumann d7b5061aa5
Merge pull request #3394 from MichaelEischer/apple-m1-build
Build darwin/arm64 binaries for Apple M1
2021-05-17 09:56:44 +02:00
rawtaz 27141ae87f
Merge pull request #2999 from restic/issue-2109
backup: Improve wording for --one-file-system description
2021-05-15 00:23:36 +02:00
Leo R. Lundgren 90d75651e6 backup: Improve wording for --one-file-system description 2021-05-15 00:06:27 +02:00
rawtaz 2a915069a8
Merge pull request #3393 from MichaelEischer/fix-filter-crash
filter: Fix crash for '**' pattern
2021-05-14 23:58:44 +02:00
Michael Eischer 55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
Michael Eischer af6f6fba15 Build darwin/arm64 binaries for Apple M1 2021-05-14 23:04:45 +02:00
MichaelEischer 6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer 7349c6d338
Merge pull request #3167 from renard/limit-snapshots-list
Add option to limit snapshots list
2021-05-13 20:26:33 +02:00
Sébastien Gross 2a92b68e65 cmd/snapshots: Add option to limit snapshots list
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`

Output example:

    $ restic snapshots --latest 2
    repository 0d3eb989 opened successfully, password is correct
    ID        Time                 Host        Tags        Paths
    ------------------------------------------------------------
    5a33bdcc  2020-12-14 12:30:00  local                   /home
    73887d8e  2020-12-15 12:30:00  local                   /home
    ------------------------------------------------------------
    2 snapshots

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
MichaelEischer 64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
oli-h 23f9cb838d
Sudden find: fix within first "forget" example
I think this was a typo or copy/paste problem when created the doc.
Feel free to merge. Cheers
2021-05-10 08:55:42 +02:00
rawtaz 01261770bb
Merge pull request #3386 from adrian5/doc-change
doc: Add direct link to GitHub issue
2021-05-09 00:12:21 +02:00
adrian5 a0f1c74000
doc: Add direct link to GitHub issue 2021-05-08 21:13:11 +00:00
Alexander Neumann be6fc02c04
Merge pull request #3376 from restic/rawtaz-doc-exclude
doc: Polish exclude file documentation
2021-04-27 19:25:00 +02:00
Alexander Neumann 3ce5544796
Merge pull request #3321 from restic/doc-files-from
doc: Improve docs for --files-from et al
2021-04-27 19:15:31 +02:00
rawtaz 556caa326f
doc: Polish exclude file documentation 2021-04-25 20:36:11 +02:00
greatroar ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
rawtaz f7316cea07
Merge pull request #3371 from cyounkins/max_unused_default
doc: default for --max-unused
2021-04-23 03:18:05 +02:00
Craig Younkins 32a84ab3e4 doc: default for --max-unused 2021-04-22 21:12:37 -04:00
rawtaz 6c7eabf08c
Merge pull request #3363 from davegallant/design-doc-grammar-fix
doc: Fix grammar in design.rst
2021-04-15 11:01:38 +02:00
Dave Gallant 4aaf10c356
doc: Fix grammar in design.rst 2021-04-14 17:25:53 -04:00
Michael Eischer f65e1211d9 Add changelog entry 2021-04-11 20:02:09 +02:00
Michael Eischer 7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer 80564a9bc9 Properly detect mintty output redirection
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer 5e6af77b7a Unify interactive terminal detection code
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar 23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Alexander Neumann b922fc851b
Merge pull request #3356 from restic/rawtaz-doc-dollar-sign
doc: Clarify dollar sign expansion in exclude files
2021-04-09 12:30:54 +02:00
rawtaz ccfd5f1d4a
Merge pull request #3298 from jniggemann/patch-1
doc: Add note about bash completion path
2021-04-09 00:17:17 +02:00
Jan Niggemann 9a1f685179 doc: Add note about bash completion path 2021-04-08 23:53:38 +02:00
rawtaz b5e40b370c doc: Clarify dollar sign expansion in exclude files 2021-04-08 23:46:53 +02:00
rawtaz 74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
Leo R. Lundgren 5861bb031c doc: Improve docs for --files-from et al 2021-04-07 18:31:46 +02:00
rawtaz c2569ff923
Merge pull request #3354 from greatroar/filechange-docs
docs: on Windows, the filesize must match for "unchanged"
2021-04-07 15:29:46 +02:00
greatroar ecbe7f3a99 Backup docs: on Windows, the filesize must match for "unchanged" 2021-04-04 17:01:48 +02:00
Michael Eischer 88731d8c28 Add changelog 2021-03-27 17:08:35 +01:00
greatroar dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
MichaelEischer efb10b3c40
Merge pull request #3347 from jazcap53/fix_ambiguous_warning
Change ambiguous Warning message
2021-03-27 12:54:07 +01:00
MichaelEischer d456437ad1
Merge pull request #3343 from hluup/3334
Display 'created new cache in ' message only if output is terminal
2021-03-27 12:43:37 +01:00
MichaelEischer 4c61825249
Merge pull request #3335 from cyounkins/patch-1
prune --max-unused does not limit metadata repacking
2021-03-27 12:40:35 +01:00
Andrew Jarcho 9f44129c2f Change ambiguous Warning message
Fixes #3346

On branch fix_ambiguous_warning
Changes to be committed:
    modified:   cmd_backup.go
2021-03-24 10:44:47 -04:00
Hendrik Luup 5592c17e4a Display 'created new cache in ' message only if output is terminal 2021-03-22 11:31:08 +02:00
Craig Younkins aa0a7b78a8 doc: prune --max-unused unlimited will still repack metadata
Adding minor clarification to documentation of `prune --max-unused unlimited` to indicate metadata will still be repacked. The referenced PR indicates `max-repack-size` CAN limit metadata repacking.

Ref: https://forum.restic.net/t/max-unused-unlimited-still-repacks/3661
Ref: https://github.com/restic/restic/pull/3228
2021-03-21 13:19:09 -04:00
Alexander Neumann 88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
MichaelEischer e678acafcf
Merge pull request #3331 from greatroar/upgrade-sftp
Upgrade pkg/sftp to 1.13
2021-03-10 22:27:32 +01:00
Michael Eischer 54d58edacc debug: fix usage for examine command 2021-03-10 22:22:33 +01:00
Michael Eischer 5975ed61f3 debug: fix linter warning 2021-03-10 21:22:53 +01:00
Michael Eischer dc62ec5933 debug: use Printf/Warnf for output 2021-03-10 21:20:21 +01:00
Michael Eischer 547d9b384d debug: cleanup repair code 2021-03-10 21:15:38 +01:00
greatroar 187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer fa7b9d5dfe debug: Cleanup pack size checks 2021-03-10 20:57:14 +01:00
Michael Eischer 6774fc6454 debug: check arguments and cleanup error handling 2021-03-10 20:43:22 +01:00
Michael Eischer 096f15db5c debug: extract examinePack function 2021-03-10 20:21:05 +01:00
Michael Eischer 84491ff40b debug: store repaired and correct blobs 2021-03-10 20:03:44 +01:00
Alexander Neumann b3c3121622 debug: Save raw decrypt (disregarding signature) 2021-03-10 20:03:44 +01:00
Alexander Neumann ce4b6d0874 examine: add byte repair mode 2021-03-10 20:03:44 +01:00
Alexander Neumann 52061e817c debug: fix percentage 2021-03-10 20:03:44 +01:00
Alexander Neumann 133ac42a0b debug: check file content hash 2021-03-10 20:03:44 +01:00
Alexander Neumann 90f975fa1c debug: make output less verbose 2021-03-10 20:03:44 +01:00
Alexander Neumann 086993bae1 debug: check packs not in index, implement repair 2021-03-10 20:03:44 +01:00
Alexander Neumann d6f78163d4 Add 'debug examine' command to debug #1999 2021-03-10 20:03:44 +01:00
MichaelEischer c9b4fadd91
Merge pull request #3294 from Achilleshiel/fix-copy-repofile
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
Michael Eischer da458a55db Cleanup comments in secondary repo test 2021-03-08 22:41:13 +01:00
Wouter Horlings 9ccdba9df6 Add repositoryFile2 option
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.

This also fixes the conflict with the `--repository-file` and `--repo2`
flag.

Tests are added for the initSecondaryGlobalOpts function.

This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.

With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.

In total the tests checks wether the following are passed correct:
   - Password
   - PasswordFile
   - Repo
   - RepositoryFile

The following situation must return an error to pass the test:
   - no Repo or RepositoryFile defined
   - Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
MichaelEischer a0f9d73d44
Merge pull request #3305 from juergenhoetzel/terminal-race-condition
backup: In case of error also Wait() for terminal goroutine
2021-03-08 22:20:18 +01:00
Michael Eischer 427781928f copy: test that trees with unstable json encoding are properly copied 2021-03-08 22:16:48 +01:00
Michael Eischer 2fc7abac35 copy: copy raw bytes of tree blobs
This avoids problems when for some reason the JSON encoding changes.
This also ensures forward compatibility with future restic versions
which might e.g. add new fields to the tree metadata.
2021-03-08 22:16:48 +01:00
Michael Eischer f9c581f219 Tweak changelog 2021-03-08 22:11:34 +01:00
Juergen Hoetzel 18fccb5995 backup: In case of error also Wait() for terminal goroutine
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
Michael Eischer 2a9f0f19b6 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
rawtaz d686fa25de
Merge pull request #3323 from restic/doc-setcap
doc: Clarify setcap applying only to current binary
2021-03-08 19:32:11 +01:00
rawtaz f000f41c91 doc: Clarify setcap applying only to current binary 2021-03-08 19:21:06 +01:00
MichaelEischer 64fe733fa0
Merge pull request #3320 from MichaelEischer/fix-ci
Update to Go 1.16 and pin go version used for golangcil-lint
2021-03-08 19:18:10 +01:00
Michael Eischer 781378a65e Bump go version to 1.16 2021-03-04 23:24:26 +01:00
Michael Eischer 64a9272c9a go mod tidy for go 1.16 2021-03-04 23:18:53 +01:00
Michael Eischer ee89e33f12 CI: Pin go version used for golint to 1.15.x
This prevents unexpected lint failures when a new go version is
released.
2021-03-04 23:12:48 +01:00
Michael Eischer f0e9068ef2 go mod tidy for go 1.16 2021-03-04 22:56:08 +01:00
Michael Eischer f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
MichaelEischer 814a399e4c
Merge pull request #3283 from dennypage/master
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
Michael Eischer 13730e3844 Add link to pr to changelog 2021-02-28 00:40:52 +01:00
Denny Page 0f41e99ea7 Treat an empty password as a fatal error for repository init. 2021-02-28 00:40:00 +01:00
MichaelEischer 6712c6de73
Merge pull request #3309 from MichaelEischer/fix-random-check-crash
check: Fix crash of --read-data-subset=x% on empty repository
2021-02-27 16:24:56 +01:00
Michael Eischer 9e852af5be check: Fix crash of --read-data-subset=x% on empty repository
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
MichaelEischer 4baebdc6f5
Merge pull request #3308 from MichaelEischer/fix-not-a-directory
local: Ignore files in intermediate folders
2021-02-27 15:34:36 +01:00
Michael Eischer a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
TJ Rana 91e8f0e486
Clean up writing style 2021-02-16 22:19:11 -05:00
TJ Rana d7d562b287
Clarify checklist items 2021-02-16 22:11:22 -05:00
TJ Rana 76e1ba6fd0
Clean up pull request template 2021-02-16 22:03:33 -05:00
rawtaz 8eb6a5805b
Merge pull request #3286 from MichaelEischer/fix-quiet-backup
backup: Correctly handle --quiet flag
2021-02-15 22:35:04 +01:00
Michael Eischer 65bd2a9a49 backup: Correctly handle --quiet flag
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00
Alexander Neumann 12f0ccc237 helpers: Also push versioned image to Docker 2021-02-15 20:08:40 +01:00
Alexander Neumann bb53fcfc0d Set development version for 0.12.0 2021-02-14 11:44:24 +01:00
Alexander Neumann 27f241334e Add version for 0.12.0 2021-02-14 11:44:20 +01:00
Alexander Neumann 4e99a3d650 Update manpages and auto-completion 2021-02-14 11:44:20 +01:00
Alexander Neumann 1cb1cd6f44 Generate CHANGELOG.md for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann 1a34260cf0 Prepare changelog for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann 13d52c88fb
Merge pull request #3282 from restic/changelog-unreleased
changelog: Correct and slightly polish some unreleased changelog files
2021-02-14 10:49:33 +01:00
Leo R. Lundgren 4b5ca1e914 changelog: Correct and slightly polish some unreleased changelog files 2021-02-14 00:57:54 +01:00
MichaelEischer 917f5b910a
Merge pull request #3270 from restic/fix-ls-json
ls: Check for non-nil error before calling panic()
2021-02-09 22:05:04 +01:00
Alexander Neumann c0f2c1d871 ls: Check for non-nil error before calling panic() 2021-02-07 21:12:54 +01:00
Alexander Neumann 9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann 2dd592a06c
Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Lorenz Brun 362338dd60 Add changelog entry 2021-02-01 20:20:34 +01:00
MichaelEischer 6ac032be64
Merge pull request #3256 from aawsome/rebuild-index-bar-done
rebuild-index: Add missing bar.Done()
2021-01-31 18:48:14 +01:00
MichaelEischer 0ce05d5725
Merge pull request #3236 from Wouter0100/patch-1
doc: Note only path-style URL support for S3
2021-01-31 18:44:58 +01:00
Michael Eischer 0aed8d47d7 docs: Fix typo in S3 URL support 2021-01-31 18:33:15 +01:00
Alexander Weiss 39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Michael Eischer 47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
MichaelEischer b3dc127af5
Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
Wouter van Os 8442c43209
doc: Update wording for S3 path-only requirement 2021-01-31 15:10:03 +01:00
Alexander Weiss 6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss 5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss 573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
MichaelEischer b8550a21f2
Merge pull request #3253 from restic/improve-prune-error
prune: Improve error message for missing files
2021-01-31 12:03:20 +01:00
Alexander Neumann 027a51529d prune: Improve error message for missing files
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
Lorenz Brun 5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Alexander Neumann f647614e24
Merge pull request #3248 from MichaelEischer/backend-cleanups
Backend code and test cleanups
2021-01-30 21:37:33 +01:00
Michael Eischer e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann 0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
MichaelEischer 99228be623
Merge pull request #3250 from restic/add-golangci-lint-config
Add golangci lint config, add many error checks
2021-01-30 21:09:38 +01:00
Alexander Neumann 04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann a00e27adf6 Add entry to changelog 2021-01-30 20:22:25 +01:00
Alexander Neumann 0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann 200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann cbd88c457a backup: Improve error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann 1a0eb05bfa errcheck: Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann 3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann 16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann 75f53955ee errcheck: Add error checks
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann 1632a84e7b Add a section explaining golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann b3d5bf7c99 Update golangci-lint version 2021-01-30 20:02:37 +01:00
Alexander Neumann 57627a307f Add config for golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann 6ab7d49a03
Merge pull request #3251 from MichaelEischer/rest-dropped-error
rest: handle dropped error in save operation
2021-01-30 19:44:50 +01:00
Michael Eischer a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Alexander Neumann dd94efb307
Merge pull request #3249 from MichaelEischer/fix-dropped-gs-error
gs: Don't drop error when finishing upload
2021-01-30 16:24:57 +01:00
Michael Eischer 8a486eafed gs: Don't drop error when finishing upload
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
MichaelEischer 4d576c2f79
Merge pull request #3217 from M4a1x/read-data-subset-fix
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
Max Stabel f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
MichaelEischer fb3cf3f885
Merge pull request #3245 from aawsome/prune-fix-statistics-cacheable
prune: Fix statistics for --repack-cacheable-only
2021-01-29 23:14:49 +01:00
Alexander Weiss e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann cdd704920d azure: Pass data length to Azure libray
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann bbdf18c4a2
Merge pull request #3176 from MichaelEischer/backend-content-length
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer 1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer 4526d5d197 swift: explicitly pass upload size to library
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer dca9b6f5db azure: explicitly pass upload size
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann a16ce65295
Merge pull request #3244 from MichaelEischer/better-damage-reports
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
Alexander Neumann 5c41120c70 Add entry to changelog 2021-01-29 11:31:36 +01:00
Alexander Neumann 5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann 81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
rawtaz de7e3a0648
Merge pull request #2823 from greatroar/trust-mtime
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
greatroar 6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer 58b5679f14 prune: reword missing blobs error
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
Michael Eischer 7b8886c052 prune: report missing but unneeded pack files
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
Michael Eischer ff95999246 rebuild-index: report added/removed/reindexed files
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
Michael Eischer b71c52797a find: correctly expand multiple blob ids
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
MichaelEischer 82140967d3
Merge pull request #3228 from aawsome/prune-all-trees
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
MichaelEischer 43cb26010a
Merge pull request #3242 from greatroar/fprintln
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
MichaelEischer 35033d9b79
Merge pull request #3177 from MichaelEischer/fix-2759
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
Alexander Neumann 84822d44d4
Merge pull request #2536 from MatthewVance/threat-model
Update threat model
2021-01-28 14:26:03 +01:00
Matthew Vance 58c7f4694d Update threat model 2021-01-28 14:20:42 +01:00
Alexander Neumann 4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann 44169d0dc4
Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
Alexander Neumann 6aa7e9f9c6
Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Alexander Neumann 72eec8c0c4
Merge pull request #3106 from MichaelEischer/parallel-tree-walk
Parallelize tree walk in prune and copy and add progress bar to check
2021-01-28 12:06:42 +01:00
Michael Eischer 68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer 1e306be000 Add changelog entry 2021-01-28 11:10:50 +01:00
Michael Eischer ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer 313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer 505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer 258ce0c1e5 parallel: report progress for StreamTrees
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer 3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer 0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Michael Eischer f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer 6e03f80ca2 check: Split the parallelized tree loader into a reusable component
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer 1d7bb01a6b check: Cleanup tree loading and switch to use errgroup
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Wouter van Os b5972f184c
doc: Note only path-style URL support for S3
This adds a node to mention that currently only path-style URL's are supported for S3, as per code at:
- aa0faa8c7d/internal/backend/s3/config.go (L42-L45)
- aa0faa8c7d/internal/backend/s3/config.go (L48-L62)
2021-01-23 16:54:08 +01:00
Alexander Weiss d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer 678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer 6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
MichaelEischer aa0faa8c7d
Merge pull request #3208 from restic/add-mips
Add mips* architectures to CI and release
2021-01-03 16:06:10 +01:00
MichaelEischer f7ec263a22
Merge pull request #3109 from aawsome/optimize-filerestorer
restore: Don't save pack content in memory
2021-01-03 14:53:41 +01:00
Alexander Neumann 7d665fa1f4 Add entry to changelog 2021-01-03 14:41:11 +01:00
Michael Eischer 69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Neumann 36db248e30 Split cross compilation targets into two jobs 2021-01-03 10:50:54 +01:00
Alexander Neumann eb72b10f55 Add mips* architectures to CI and release 2021-01-03 10:41:54 +01:00
Alexander Neumann 622f4c7daa Run cloud backends also on pushes 2021-01-03 10:38:10 +01:00
Alexander Neumann f8c50394d6 Revert "Run cloud tests if secrets are available"
This reverts commit aa648bdcac.
2021-01-03 10:28:01 +01:00
Alexander Neumann aa648bdcac Run cloud tests if secrets are available 2021-01-03 10:26:55 +01:00
Alexander Neumann e8abc79ce9 Add hints for keeping the list of architectures in sync 2021-01-01 10:09:04 +01:00
Alexander Weiss 34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss 7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss 07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss 3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Michael Eischer 97388b3504 Properly check that --quiet and --verbose are not combined
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
Alexander Neumann 8b84c96d9d
Merge pull request #3204 from MichaelEischer/archiver-tomb-race
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer e1efc193e1
Merge pull request #3139 from aawsome/prune-healing
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
Alexander Weiss f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss f6df94a50e prune: Add self-healing
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
MichaelEischer 31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
MichaelEischer 7fda2f2ad8
Merge pull request #3134 from greatroar/unlock-warn
Warn when unlock fails instead of returning an error
2020-12-29 18:30:08 +01:00
greatroar dec5008369 Warn when unlock fails instead of returning an error
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
Quentin Lemaire 873505ed3b
Update related changelog 2020-12-29 17:12:46 +01:00
Quentin Lemaire 25ecf9eafb
fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire e88f3fb80c
fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Alexander Neumann b2efa0af39
Merge pull request #3164 from MichaelEischer/improve-context-cancel
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer 25f4acdaa8 Add changelog entry 2020-12-29 16:32:18 +01:00
Michael Eischer cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer 05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
Michael Eischer 92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer 34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer 023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer 684600cf42 ui: update status for the backup command on non-interactive terminals
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer 85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer 969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
Michael Eischer 13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer c2ef049f1b ui/progress: don't print progress on non-interactive terminals
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
2020-12-29 16:03:43 +01:00
MichaelEischer a488d4c847
Merge pull request #2833 from greatroar/aix
AIX port
2020-12-29 12:32:50 +01:00
Alexander Neumann 4133b1ea65 Synchronize OS and architectures for testing 2020-12-29 11:11:50 +01:00
Alexander Neumann 46d2ca5095 Remove old test script 2020-12-29 11:02:48 +01:00
Quentin Lemaire 334d8ce724
feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
Alexander Neumann c661518df9
Merge pull request #2793 from greatroar/rclone-doc
Clarify rclone-over-SSH docs
2020-12-29 10:46:34 +01:00
Michael Eischer 0d81f16343 Add AIX as cross-compile target to CI 2020-12-29 01:35:01 +01:00
greatroar 3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar 18531e3d6f Portability fixes to internal/restic
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer ca07317815 add changelog 2020-12-28 21:06:47 +01:00
Michael Eischer d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer 08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer e483b63c40 retrybackend: Fail operations when context is already canceled
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer 736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
MichaelEischer 9c41e4a343
Merge pull request #3192 from DRON-666/fix-options
Fix formatting of `restic options` command
2020-12-23 22:53:40 +01:00
DRON-666 332b1896d1 Some `options` fixes
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer cb6b0f6255
Merge pull request #3181 from tWido/feature-no-retry-permission
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer 1e73aac610
Merge pull request #3179 from aawsome/check-filesize
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss 2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido 7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
MichaelEischer 8efb874f48
Merge pull request #3148 from aawsome/simplify-index-rebuild
rebuild-index: code simplification
2020-12-22 23:27:53 +01:00
Michael Eischer de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss 68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer 24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Michael Eischer 96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
MichaelEischer 69f9d269eb
Merge pull request #3189 from restic/fix-counter-segfault
ui/progress: Use mutex instead of atomic
2020-12-22 22:14:08 +01:00
rawtaz ec59c73489
Merge pull request #3188 from restic/forget-prune-dry-run
forget: Enable --dry-run together with --prune
2020-12-22 21:09:15 +01:00
Alexander Neumann 6c514adb8a ui/progress: Use mutex instead of atomic
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
Alexander Neumann edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Alexander Neumann f7c7c2f730
Merge pull request #3175 from MichaelEischer/fix-3099-changelog
Fix changelog for #3099
2020-12-20 11:29:57 +01:00
Michael Eischer cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
Michael Eischer 5cd40f8b58 Add changelog 2020-12-19 11:38:38 +01:00
MichaelEischer d32949ee54
Merge pull request #3081 from DRON-666/dump-zip
Add zip support to the `dump` command
2020-12-19 11:33:33 +01:00
DRON-666 83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666 e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666 33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666 da9053b184 Some gramma fixes in documentation 2020-12-19 01:16:15 +03:00
DRON-666 ef1aeb8724 dump: Update docs and changelog 2020-12-19 01:09:47 +03:00
DRON-666 2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666 89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666 0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer bfadc82a20 Fix changelog for #3099 2020-12-18 20:58:24 +01:00
Michael Eischer 34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
MichaelEischer 22260d130d
Merge pull request #3170 from greatroar/dont-retry
Don't retry permanent errors in backends
2020-12-17 23:29:56 +01:00
greatroar 9341a83b05 Changelog entry for #2453 and #3170 2020-12-17 23:15:48 +01:00
greatroar 66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar 746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar f7784bddb3 Don't retry when "no space left on device" in local backend
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann 1cdd38d9e0
Merge pull request #3165 from restic/2747-doc-forget
doc: Clarify calendar boundaries for --keep-* options
2020-12-16 08:38:03 +01:00
Leo R. Lundgren b3c0d2f45b doc: Clarify calendar boundaries for --keep-* options 2020-12-15 12:43:06 +01:00
Peter Albrecht cd25e36811
Add PGP fingerprint to 020_installation.rst
I like the idea of verifying the integrity of applications, I download from the internet. So I was very happy to see that restic does provide SHA256-checksums which are signed with the maintainers PGP key.

The only thing I miss: I could not find a direct way to download the used PGP key and verify the keys fingerprint.

Doing some searches, I found:
* https://github.com/restic/rest-server/issues/121
* https://restic.net/blog/2015-09-16/verifying-code-archive-integrity/

To help other restic users, I think you should add information about your PGP key/fingerprint to this installation doc, too. To save you some precious time, I created a draft, how this doc might be expanded, in this pull-request. You are free to accept it or change the text to your liking.

I copied the key/fingerprint text from: ``restic/restic/master/doc/090_participating.rst``

Thank you for your work in restic!
2020-12-13 17:00:00 +01:00
Alexander Neumann e96677cafb
Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer 1d69341e88 swift: Add support for id based keystone v3 auth parameters
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann 36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann 7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann ba31c6fdaa
Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann b58799d83a
Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann 0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Alexander Weiss d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Michael Eischer b48f579530 termstatus: Fix canUpdateStatus detection for redirected output
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer 401ef92c5f backup: Fix shutdown hang when running in the background
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss 26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss 5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
MichaelEischer 3264eae9f6
Merge pull request #3149 from aawsome/fix-rebuild-index
Bugfix for rebuild-index
2020-12-05 22:18:22 +01:00
Alexander Weiss 83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Neumann 43cf301450
Merge pull request #3141 from MichaelEischer/fix-fuse-dir-owner
fuse: Properly set uid/gid for directories
2020-12-01 16:06:24 +01:00
Michael Eischer d05c88a5d6 fuse: Properly set uid/gid for directories
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
Alexander Neumann 058b102db0
Merge pull request #3138 from MichaelEischer/fix-debug-build 2020-11-30 12:27:51 +01:00
rawtaz fcebc7d250
Merge pull request #3135 from restic/manual
doc: Fix misc missing/incorrect text in manual
2020-11-29 19:20:51 +01:00
MichaelEischer f2959127b6
Merge pull request #3065 from greatroar/local-subdirs
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Leo R. Lundgren 61460dee52 doc: Fix misc missing/incorrect text in manual 2020-11-29 18:59:24 +01:00
Michael Eischer 54a6d98945 Enable debug builds for CI 2020-11-29 18:47:00 +01:00
Michael Eischer f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer 52b98f7f95
Merge pull request #3017 from greatroar/files-from0
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
Alexander Neumann 04d856e601
Merge pull request #3136 from restic/rawtaz-copy-doc
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
Alexander Neumann a7b49c4889
Merge pull request #3119 from restic/keep-mountpoints
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
rawtaz a568211b98
Merge pull request #3128 from vrenaville/tzdata
[FIX] Timezone in docker image
2020-11-28 22:43:18 +01:00
Leo R. Lundgren f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar 55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann 162117c42c Add changelog 2020-11-28 17:00:31 +01:00
Alexander Neumann 82ae942965 backup: Keep mountpoints for --one-file-system
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
Alexander Neumann f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann 037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann 2f9346a5af
Merge pull request #3125 from metalsp0rk/cat-respect-no-lock
Make restic cat respect --no-lock
2020-11-28 12:45:59 +01:00
Alexander Neumann a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan 666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
vrenaville adc7a6555f [FIX] Timezone in docker image 2020-11-27 08:05:19 +01:00
Alexander Neumann 9a97095a4c
Merge pull request #3120 from aawsome/blob-implementation
Blob implementation
2020-11-22 20:59:30 +01:00
Alexander Weiss aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss 92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann c844580e0f
Merge pull request #3101 from aawsome/packsizes
Compute packsizes in MasterIndex
2020-11-22 15:49:19 +01:00
Alexander Weiss 67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss 4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
rawtaz cac481634c
Merge pull request #3115 from johanbove/patch-1
Docs: Update 04_backup.rst
2020-11-20 11:23:03 +01:00
Johan Bové c23b1a4cba
Update 04_backup.rst
Fixed typo - _files_ are included from _folders_, not other _files_.
2020-11-20 07:52:23 +01:00
Alexander Neumann 110a32a08b
Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
greatroar 8e213e82fc backend/local: replace fs.Walk with custom walker
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann 8a150ee91f
Merge pull request #3112 from MichaelEischer/reduce-progress-log-spam
Limit progress bar updates to once per second on non-terminal outputs
2020-11-19 07:54:36 +01:00
Alexander Weiss cb5ec7ea6b Fix statistics in prune for duplicates
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
Michael Eischer 625410f003 Limit progress bar updates to once per second on non-terminal outputs
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
Alexander Neumann 75eff92b56
Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith a24e986b2b do not require gs bucket permissions to init repository
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz 6822ce8479
Merge pull request #3105 from restic/fix-init-repo-file
Allow using --repository-file in init
2020-11-17 21:54:44 +01:00
Alexander Neumann d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
rawtaz 342520b648
Merge pull request #3103 from greatroar/rephrase-pr3102
Rephrase #3095/#3102 changelog entry
2020-11-17 12:52:12 +01:00
greatroar 028f2b8c0e Rephrase #3095/#3102 changelog entry 2020-11-17 12:37:06 +01:00
rawtaz 1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Alexander Neumann 5f3b802ee7
Merge pull request #3099 from MichaelEischer/check-less-memory 2020-11-16 10:11:25 +01:00
Michael Eischer 022dc35be9 Add changelog 2020-11-15 19:02:51 +01:00
Michael Eischer 1f43cac12d check: Only track data blobs when unused blobs should be reported
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer 6da66c15d8 check: Simplify referenced blob tracking
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer 3500f9490c check: Simplify blob status tracking
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
MichaelEischer 45ba456291
Merge pull request #3038 from fgma/check-data-subset-percentage
Check data subset: check random percentage subset
2020-11-15 18:24:13 +01:00
Michael Eischer caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
Michael Eischer 15c537f9db Rename changelog entry to contain issue id 2020-11-15 18:13:50 +01:00
fgma 8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann 3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann 9968220652
Merge pull request #2850 from greatroar/packer-malloc
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
kitone 0649828555 Add changelog 2020-11-15 17:09:30 +01:00
kitone 3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
greatroar ab2b7d7f9a Decrease allocation rate in internal/pack
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar 9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss 9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss 3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss 30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss 187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss 1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss 5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss 43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
MichaelEischer 6feaf6bd1f
Merge pull request #3030 from greatroar/concrete-statt
Replace restic.statT interface by concrete types
2020-11-14 23:32:17 +01:00
greatroar c45f8ee075 Replace restic.statT interface by concrete types
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer 3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
Fix #2688: Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
Michael Eischer fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer 333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire a8ad6b9a4b
fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire b0882b3f3c
fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire e74110a833
docs: Write new entry to changelog/unreleased 2020-11-14 15:48:56 +00:00
Quentin Lemaire ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Neumann 913a34f568
Merge pull request #3094 from restic/fix-ci-tests
Fix ci tests
2020-11-14 11:24:28 +01:00
Alexander Neumann 468612b108 Disable fuse test for macOS 2020-11-14 11:02:49 +01:00
Alexander Weiss 7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss 17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss 80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran 94a154c7ca Remove `--drive-use-trash=false` from rclone param
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann 219d9a62f2 Download assets from repo 2020-11-13 21:56:14 +01:00
rawtaz e8713bc209
Merge pull request #3093 from restic/fix-diff-metadata
diff: Correctly count top-level blobs
2020-11-13 21:50:29 +01:00
Alexander Neumann 04d1983800
Merge pull request #3090 from fgma/vss-fix-386
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
fgma 88208c3db2 vss: improved changelog entry 2020-11-13 21:13:17 +01:00
Alexander Neumann 59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
Alexander Neumann 145830005b Use old versions, otherwise tar doesn't work 2020-11-12 21:35:01 +01:00
Alexander Neumann 8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma 859d89b032 vss: add changelog file for issue 3090 2020-11-12 19:38:22 +01:00
fgma f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
Alexander Neumann 0be906a92f CI: Use netcologne mirror 2020-11-11 20:59:23 +01:00
rawtaz dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann e4e0ce09ad CI: Update links for Windows binaries 2020-11-11 20:51:01 +01:00
Alexander Neumann 0334114865 CI: Hardcode mirror URL 2020-11-11 20:42:47 +01:00
Alexander Neumann b1bbdcb637 Improve changelog 2020-11-11 20:30:30 +01:00
Alexander Neumann 4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Alexander Neumann 3e0456d88b Highlight that s3.list-objects-v1 is a temporary 2020-11-11 20:11:35 +01:00
Alexander Neumann dd94174379
Merge pull request #3086 from greatroar/diff-errors
Improve error reporting from restic diff
2020-11-11 19:48:03 +01:00
greatroar 63e32c44c0 Improve error reporting from restic diff
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma f013662e3f Remove separate section on Ceph, and move s3.list-objects-v1 note to S3 section 2020-11-11 15:11:14 +01:00
Nick Douma 4320ff2bbf Add changelog entry for s3.list-objects-v1 2020-11-11 12:33:22 +01:00
Nick Douma 354b7e89cc Document the extended s3.list-objects-v1 flag in a new Ceph section 2020-11-11 12:32:46 +01:00
Nick Douma 829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma 4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer 407843c5f9
Merge pull request #3034 from hoyho/gh_master
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer 46d31ab86d
Merge pull request #3058 from greatroar/counter
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss 239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho 9df52327cc bugfix: omit ENOTDATA for extended attributes
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar 21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar ddca699cd2 Replace restic.Progress with new progress.Counter
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
rawtaz 605db3b389
Merge pull request #3077 from MichaelEischer/tame-golangci-lint
Switch to the default set of golangci-lint linters
2020-11-08 22:34:27 +01:00
Michael Eischer 8de129e12f Switch to the default set of golangci-lint linters 2020-11-08 22:17:50 +01:00
Alexander Neumann 2072f0a481
Merge pull request #3076 from restic/check-go-mod-tidy
Check that go.mod/go.sum are up to date
2020-11-08 21:37:22 +01:00
Alexander Neumann 5731e391f8
Merge pull request #3075 from restic/warn-prune-no-cache
UI: Add several hints
2020-11-08 20:37:10 +01:00
Alexander Neumann 6a0a1d1f1c Check that go.mod/go.sum are up to date 2020-11-08 20:32:25 +01:00
Alexander Neumann a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann 823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann a5989707ac Move test for golang-ci 2020-11-08 19:58:16 +01:00
Alexander Neumann 3a0cfafeb5 Fix quotes 2020-11-08 19:52:37 +01:00
Alexander Neumann c923bd957d Fix test for pull request 2020-11-08 19:51:11 +01:00
Alexander Neumann 1a3f885d3d Only run golangci-lint for PRs 2020-11-08 17:34:07 +01:00
Alexander Neumann 3bf43d7951
Merge pull request #2475 from restic/use-github-actions
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann 561da92396 Replace badges in README 2020-11-08 17:26:59 +01:00
Alexander Neumann 5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
MichaelEischer 9e4e0077fb
Merge pull request #3069 from greatroar/xattr-eintr
Update pkg/xattr to handle EINTR on Linux
2020-11-08 12:52:23 +01:00
MichaelEischer 1758da855f
Merge pull request #3068 from aawsome/context-mem-backend
Return context error in mem backend
2020-11-08 12:24:12 +01:00
greatroar 15ea90feed Update pkg/xattr to handle EINTR on Linux
Updates #2659. This is a case where the stdlib will not handle EINTR for
us, even with Go 1.16. That xattr calls are directly affected can be
seen in the report for issue #2968.
2020-11-08 09:34:24 +01:00
Alexander Weiss 826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar a2d4209322 Don't recurse in local backend's List if not required
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
2020-11-07 08:54:13 +01:00
Alexander Neumann 275f713211 Remove Travis and AppVeyor 2020-11-06 21:49:52 +01:00
MichaelEischer 4707bdb204
Merge pull request #2842 from aawsome/rebuild-index-inmem
Rebuild index in prune by using in-memory index
2020-11-06 20:51:20 +01:00
Alexander Neumann 47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss 38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann 7f6f31c34b Remove Hound 2020-11-06 11:43:58 +01:00
Alexander Neumann 164b4cb2f6 Add changelog for #3014 2020-11-06 10:05:42 +01:00
Alexander Neumann 4a9b05aff1
Merge pull request #3063 from aawsome/fix-3062
Fix #3062
2020-11-05 19:36:40 +01:00
Alexander Weiss aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Neumann a5592e83f7
Merge pull request #3014 from ivandeex/pr-stream-reset
Fix sporadic stream reset between rclone and restic
2020-11-05 16:11:50 +01:00
Ivan Andreev ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Alexander Neumann 8a2a326189
Merge pull request #2535 from ncw/fix-2528
s3: add bucket-lookup parameter to select path or dns style bucket lookup
2020-11-05 12:39:34 +01:00
Nick Craig-Wood 86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup
This is to enable restic working with Alibaba cloud

Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann 636b2f2e94
Merge pull request #2941 from MichaelEischer/parallel-repack
prune: Parallelize repack step
2020-11-05 11:00:41 +01:00
Alexander Neumann ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann 866a52ad4e Remove unneeded seek
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann 7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann 8a0dbe7c1a helpers: Remove old changelog files for release 2020-11-05 10:26:00 +01:00
Alexander Neumann 4e3ad8b3b1 Remove changelog files from 0.11.0 release 2020-11-05 10:22:39 +01:00
Alexander Neumann 5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
Reimplementation of prune
2020-11-05 10:12:19 +01:00
Alexander Neumann d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
Alexander Neumann c5300a2c56 Update manpages and auto-completion 2020-11-05 09:41:35 +01:00
Alexander Neumann 1bd92896d7 Generate CHANGELOG.md for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann 9dba01021e Prepare changelog for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann 23f6b8c3fd Add .vscode to gitignore 2020-11-05 09:40:56 +01:00
Alexander Neumann 79a50e3b1f helpers: Create changelog subdir for release 2020-11-05 09:39:06 +01:00
Alexander Neumann b8a5ca2d10 Add gitignore to changelog/unreleased 2020-11-05 09:01:32 +01:00
fgma 916b2d303b
vss: fix potential crash (not reachable in restic) (#3045)
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
rawtaz 62345abe4a
Merge pull request #3053 from maikell/patch-1
Update README.md
2020-11-03 19:06:59 +01:00
Maikel van Leeuwen e024fc6d4d
Update README.md
Typo in link. 
Links to  https://forum.restic.net%3E/ should be https://forum.restic.net
2020-11-03 18:51:11 +01:00
Alexander Neumann 1ca60bccfb Refactor condition for MaxRepackBytes
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
Alexander Neumann 7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann c1a3de4a6e Refactor max-unused calculation, add `unlimited` option
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
Alexander Neumann f8c4dd7b1a Split packe rewrite logic into two case branches
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
Alexander Neumann a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann 095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Neumann 1dd9fdce74 Reword changelog slightly 2020-11-03 16:42:21 +01:00
Alexander Weiss b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss 7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss 3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss 581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Neumann 0db9024aad
Merge pull request #3052 from restic/error-on-invalid-size
Return an error for invalid sizes
2020-11-03 12:13:49 +01:00
Alexander Neumann 21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann 2fb1957ca4 Rewrite README as Markdown 2020-11-03 09:20:09 +01:00
Alexander Neumann 9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann f14436953a
Merge pull request #3050 from greatroar/widechars
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar 11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh)
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
2020-11-02 16:41:23 +01:00
greatroar 3ed84ff0c6 Fix string truncation in ui/termstatus
Fixes #3046.
2020-11-02 12:50:49 +01:00
Alexander Neumann 8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann 5f0fa2129e Improve readability
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann 04dfa19c7e Improve changelog 2020-11-02 11:23:09 +01:00
Alexander Neumann 6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann 445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann 3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
MichaelEischer 5d379b5359
Merge pull request #3042 from rubiojr/remove-unused-function
Remove unused function
2020-11-01 21:55:22 +01:00
Sergio Rubio e708628cfd
Remove unused function
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui
internal/ui refactoring
2020-10-25 17:51:39 +01:00
rawtaz e8b4d8d8bc
Merge pull request #2998 from greatroar/env-docs
Systematize documentation of environment variables
2020-10-24 22:49:31 +02:00
Michael Eischer 1aa61e6def Add changelog 2020-10-24 22:49:29 +02:00
Michael Eischer 8d7d6ad2d5 dump: include username in tar 2020-10-24 22:42:01 +02:00
Michael Eischer fe09e6f865 dump: test proper permissions and directory name 2020-10-24 22:42:01 +02:00
Michael Eischer 1e3c9a2c11 dump: Fix file permission to tar mapping
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer e21dcb0eea dump: Additional ACL tests 2020-10-24 22:41:01 +02:00
MichaelEischer 31b8d7a639
Merge pull request #2274 from fgma/master
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma 5695f9ebd2 vss: Implement VSS support for Windows
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar 8091151638 doc: Update restic mount availability and requirements
NetBSD doesn't support restic mount either, so it's easier to list the
positive cases. Also noted that FUSE for macOS is required on the Mac.
2020-10-22 09:57:30 +02:00
greatroar ae179ee63e Systematize documentation of environment variables
Cache locations were documented inconsistently in three places.

The backup docs mentioned PATH being used to find fusermount, which is
never run by restic backup. It now mentions ssh and rclone, which are
used by backends.

The notion of a "system-wide" environment variable makes no sense.
TMPDIR is now mentioned because it allows for optimization and may
have security implications.
2020-10-22 09:57:30 +02:00
MichaelEischer 0590e3e12d
Merge pull request #3035 from greatroar/no-unmount-on-failure
restic mount: don't umount unless we actually tried to mount
2020-10-21 22:10:42 +02:00
greatroar 3807d13bdc restic mount: don't umount unless we actually tried to mount
Previously, failure to open a repository would result in an "unable to
umount" message.
2020-10-21 12:15:05 +02:00
greatroar 63be3704d9 Allow debug logging to pipes and terminals 2020-10-19 22:30:30 +02:00
greatroar 35419de232 Simplify ui.StdioWrapper.Write
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar 863a590a81 Refactor termstatus.Term.{Print,Error} methods 2020-10-17 20:30:46 +02:00
greatroar 7c0b6a82db Remove unused public method ui.linesWriter.Flush 2020-10-17 20:30:24 +02:00
MichaelEischer 96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
MichaelEischer 39fe1e96fe
Merge pull request #3018 from greatroar/background-check
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer 4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex
Refactor index decoding
2020-10-15 23:22:33 +02:00
MichaelEischer ce87fbd7dc
Merge pull request #3022 from greatroar/refactor-runworkers
Defer channel closing outside repository.RunWorkers
2020-10-14 22:44:42 +02:00
greatroar b27375f5ce defer close(ch) outside repository.RunWorkers 2020-10-14 15:50:16 +02:00
greatroar 720e0ee0c7 if cond { return true }; return false => return cond 2020-10-13 20:56:43 +02:00
greatroar 27db3ec262 Refactor index decoding
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar f80b07b2c8 Fix IsProcessBackground on Linux with stdin redirection
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer 6003dada14
Merge pull request #3011 from restic/doc-alpine
doc: Add restic install instructions for Alpine Linux
2020-10-12 23:29:40 +02:00
MichaelEischer 41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
MichaelEischer 8299c5559c
Merge pull request #3015 from greatroar/changelog-template
Improved changelog template
2020-10-12 23:12:29 +02:00
Alexander Neumann 56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group}
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
greatroar efc075df87 Improved changelog template 2020-10-12 12:15:15 +02:00
Leo R. Lundgren f500b0d90e doc: Add restic install instructions for Alpine Linux 2020-10-11 18:31:07 +02:00
Alexander Neumann c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
MichaelEischer 164d8af3dd
Merge pull request #2906 from kitone/fix-inconsistent-timestamps-permissions
Restore inconsistent timestamps permissions
2020-10-10 14:56:43 +02:00
kitone 052564007a Add changelog 2020-10-10 14:37:51 +02:00
kitone 6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions.
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone 9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone 295ddb9e57 Add test case for inconsistent timestamps and permissions restoration
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer e638b46a13 Embed context into ReaderAt
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer 2964d2ad15 Skip unlocking for nil locks
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
2020-10-09 22:39:06 +02:00
Michael Eischer 0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer 37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer 4b0fcaed45 unlock: use proper context for locks cleanup
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
Michael Eischer 645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer dc2e664209 integration_fuse_test: use global context
No need to use the TODO context.
2020-10-09 22:37:56 +02:00
Michael Eischer a449450021 init: pass proper context to master key generation
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
Michael Eischer 603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer 27456f6545 debug: use proper context
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
Michael Eischer c458e114d4 pass context to Find / FindSnapshot
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer 45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer 307a6ba3a3 Upgrade minio sdk to v7
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer 50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann 5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
Alexander Neumann 35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
Alexander Neumann 30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
Alexander Neumann 2f3eeff2e7
Merge pull request #2964 from MichaelEischer/fix-repro-instructions
Fix instructions to reproduce the release binaries
2020-10-09 21:28:24 +02:00
MichaelEischer 356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
rawtaz ab769abeaf
Merge pull request #3002 from MichaelEischer/update-contributing
Update links to help wanted labels in CONTRIBUTING.md and link to forum
2020-10-09 17:24:25 +02:00
Michael Eischer 4036991c91 Update links to help wanted labels in CONTRIBUTING.md and link to forum 2020-10-09 16:15:43 +02:00
Michael Eischer 88c8e903d2 filter: Fix glob matching on absolute path marker on windows
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
greatroar 740758a5fa Optimize filter pattern matching
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-09 15:51:14 +02:00
MichaelEischer 862ee4b2c9
Merge pull request #2970 from labkode/fopenskip
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador 958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer 88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
MichaelEischer 4a424af1d5
Merge pull request #2989 from MichaelEischer/remove-local-chmod
local: mark repository files as read-only and handle chmod errors
2020-10-08 19:04:13 +02:00
Michael Eischer bc65da2baf Add changelog 2020-10-08 14:45:26 +02:00
Michael Eischer b79f18209f Upgrade github.com/cenkalti/backoff module
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
Michael Eischer 8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer 0acc3c5923 filter: special case patterns without globbing characters
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer 54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer bcc3bddcf4 filter: only check whether a child path could match when necessary
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer 17c53efb0d filter: Optimize double wildcard expansion
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer 7959796269 filter: Special case for absolute paths
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer 375c2a56de filter: Parse filter patterns only once
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer b8eacd1364 filter: Reduce redundant path and pattern splitting
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer f4282aa6fd local: mark repository files as read-only
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
2020-10-07 12:29:37 +02:00
Michael Eischer 40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
greatroar f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer 0ae02f3030
Merge pull request #2844 from aawsome/prune-integration-tests
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
rawtaz e401859afb
Merge pull request #2992 from lbausch/repo_wording
Replace repo/repos with repository/repositories in `--no-lock` help text
2020-10-06 23:03:55 +02:00
Lorenz Bausch af4100e07d
Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Neumann ea5bbe0857 Remove dysfunctional link to saythanks 2020-10-06 20:43:44 +02:00
Alexander Weiss 6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
MichaelEischer 09d39e260d
Merge pull request #2980 from greatroar/error-wrapping
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
Michael Eischer 1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer 9ffb698c8d Modernize context import in internal/fuse
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
Michael Eischer efbb850d92 Remove a few redundant type specifiers
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
MichaelEischer cfd57c480a
Merge pull request #2991 from restic/nonsense
cache/check: Clarify error message when given arguments
2020-10-06 00:40:12 +02:00
Leo R. Lundgren e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz 5d8cfff3f2
Merge pull request #2988 from greatroar/snapshots-json
Regression test for #2979
2020-10-05 21:02:52 +02:00
greatroar 673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer 1ab4c710e1
Merge pull request #2987 from greatroar/no-password-for-help
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar feedf0ebce Don't read password for generate, help or self-update
Fixes #2951.
2020-10-05 19:54:07 +02:00
MichaelEischer 1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
Michael Eischer 187518a8a3 docs: Properly reproduce zip archives for windows binaries 2020-10-05 18:24:46 +02:00
Michael Eischer a232c833dc docs: Update binary reproduction instructions and refer to older instructions 2020-10-05 18:24:46 +02:00
MichaelEischer c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
MichaelEischer 8390a8aaf3
Merge pull request #2986 from greatroar/upgrade-xxhash
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:49:08 +02:00
greatroar 1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
MichaelEischer abe9fa261f
Merge pull request #2983 from greatroar/upgrade-siphash
Upgrade siphash to 1.2.2
2020-10-05 17:18:07 +02:00
MichaelEischer 6001b45bf7
Merge pull request #2984 from restic/snapshots-json
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 17:14:23 +02:00
greatroar 6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
greatroar 8252ea8e3d Upgrade siphash to 1.2.2
New version is potentially faster on 32-bit ARM.
2020-10-05 14:56:13 +02:00
Leo R. Lundgren 137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar 7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
rawtaz a23d90d270
Merge pull request #2975 from restic/rawtaz-doc-resume
Add FAQ on whether restic can resume backups
2020-10-03 23:12:34 +02:00
MichaelEischer 2630411530
Merge pull request #2974 from gottwald/swap-gcs-lib
Swap deprecated GCS lib with replacement
2020-10-03 21:58:11 +02:00
rawtaz b9b82d878d Add FAQ on whether restic can resume backups
This new FAQ entry explains that restic can resume interrupted backups.
2020-10-03 21:39:45 +02:00
Ingo Gottwald 8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
rawtaz 7c3c6fa431
Merge pull request #2977 from MichaelEischer/downgrade-cobra
Downgrade cobra to v0.0.5
2020-10-03 18:23:13 +02:00
Michael Eischer 29908906b7 Downgrade cobra to v0.0.5 2020-10-03 18:13:07 +02:00
rawtaz bbeb439f41
Merge pull request #2937 from andreaso/self-update-output-path-fix
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson ce14df303b
Don't require `self-update --output` placeholder file
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer 3c6671b18b
Merge pull request #2973 from restic/rawtaz-repo2
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
MichaelEischer 86ff1f2bf6
Merge pull request #2972 from restic/rawtaz-verbose
Clarify max verbose level in option description
2020-10-02 22:21:45 +02:00
rawtaz 0cce6dc31c
Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz 6253ff0187
Clarify max verbose level in option description
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
rawtaz e9943e864f
Merge pull request #2955 from jtagcat/clarify--compact
help: --compact: clarify description
2020-10-02 15:59:08 +02:00
jtagcat a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
MichaelEischer 4df8861e09
Merge pull request #2967 from gottwald/fix-deprctd-gs-constructor
Replace deprecated method in gs backend
2020-10-01 11:01:49 +02:00
Ingo Gottwald 00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
MichaelEischer f361ed66de
Merge pull request #2910 from mtdcr/repository-file
Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
2020-10-01 01:04:23 +02:00
Michael Eischer 7b50a65492 Update backup help output in documentation 2020-10-01 00:50:27 +02:00
Michael Eischer c18b119a9b Document new option --repository-file 2020-10-01 00:50:27 +02:00
Michael Eischer 61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter 97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
2020-10-01 00:50:26 +02:00
Michael Eischer d44df9d00d backup: Always remove the status lines once a backup ends 2020-09-30 23:13:10 +02:00
Michael Eischer 8d0ba55ecd backup: Fix possible deadlock of scanner goroutine
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.

This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.

This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
rawtaz 34ea960559
Merge pull request #2966 from MichaelEischer/recover-help-typo
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
rawtaz 5ea01b00df
Merge pull request #2965 from MichaelEischer/rework-password-prompt
Clarify verbose password prompt
2020-09-30 18:15:05 +02:00
MichaelEischer 48d0ab5276
Merge pull request #2959 from restic/keep-text
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
Michael Eischer 2a79c1a44d recover: Fix typo in the command help
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
MichaelEischer fd02407863
Merge pull request #2849 from classmarkets/gcs-access-token
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
Michael Eischer aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
Leo R. Lundgren 028c9a5343 backup: Correct keep policy text
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
MichaelEischer 94136132e3
Merge pull request #2957 from plumbeo/patch-3
Don’t print excessively detailed debug messages on object deletion when —verbose is used
2020-09-27 22:08:07 +02:00
plumbeo 009bd907f2
Don’t print too many messages when deleting files
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
MichaelEischer 14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred 206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer 4875f7b659
Merge pull request #2614 from greatroar/simplify-fs
Simplify internal/fs
2020-09-21 22:05:10 +02:00
MichaelEischer 6f2093e491
Merge pull request #2940 from andreaso/appveyor-tar-https
Use https:// to download appveyor tar
2020-09-21 21:50:26 +02:00
MichaelEischer 16f31b2f73
Merge pull request #2939 from J0WI/patch-1
Update Go version to 1.15 in Docker build script
2020-09-21 21:46:33 +02:00
greatroar 0d65b78168 Simplify os.ModeType|os.ModeCharDevice => os.ModeType
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
greatroar 95ebba85ff Remove stray Printf from internal/fs 2020-09-21 14:21:32 +02:00
greatroar 59b343a9bf Remove OS-specific versions of fs.MkdirAll
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
greatroar 1557c58eef Fix and simplify fs.Reader
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
greatroar c504aa505c Remove unused fs.Rename 2020-09-21 14:21:32 +02:00
greatroar 1b20f6beec Remove io.Writer from fs.File
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
rawtaz 10e3340863
Merge pull request #2945 from YoshieraHuang/max-file-size
Fix nil check in rejectBySize
2020-09-21 14:18:45 +02:00
yoshiera 3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Andreas Olsson fd1f7b7268
Use https:// to download appveyor tar
When supported one might as well use https://.
2020-09-20 18:49:46 +02:00
J0WI 090a73f7c5
Update Go version to 1.15 in Docker build script 2020-09-20 15:42:12 +00:00
rawtaz 1cfb01a8a6
Merge pull request #2936 from andreaso/contrib-update-version
Update versions in contribution documentation
2020-09-20 13:27:13 +02:00
Andreas Olsson ce62d3d689
Update versions in contribution documentation
Reflects 3c44598 and 429f97b.
2020-09-20 10:32:27 +02:00
Michael Eischer 8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer 60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer 9333f707fa init: use Create method for rclone backend
This properly issues the initial repository creation command

Fixes #1896
2020-09-19 19:11:43 +02:00
Alexander Neumann 429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann 40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Alexander Neumann c8a94eced7 Update manpages and auto-completion 2020-09-19 17:38:26 +02:00
Alexander Neumann ee6e981b4e Generate CHANGELOG.md for 0.10.0 2020-09-19 17:38:10 +02:00
Alexander Neumann 96fd982f6a Prepare changelog for 0.10.0 2020-09-19 17:37:59 +02:00
Alexander Neumann 6ff0082c02
Merge pull request #2928 from MichaelEischer/init-copy-chunker
init: Add `--copy-chunker-parameters` option
2020-09-19 17:32:13 +02:00
Alexander Neumann 95c1d7d959
Merge pull request #2927 from restic/update-deps
Update dependencies
2020-09-19 17:29:35 +02:00
Michael Eischer 07f4e7d10b Only log HTTP requests when a debug log is configured
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
Michael Eischer f003410402 init: Add `--copy-chunker-params` option
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer 9ad8250a78 Reduce overhead of debug calls if no log is enabled
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
Michael Eischer 655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
MichaelEischer 1823b8195c
Merge pull request #2930 from rawtaz/password-file-text
Improve wording for --password-file and related options
2020-09-19 16:06:15 +02:00
Leo R. Lundgren 311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Alexander Neumann a10b44a265 Add note about fuse 2020-09-19 14:35:55 +02:00
Alexander Neumann baf3a9aa3b
Merge pull request #2929 from rawtaz/password-command-text
Improve wording for --password-command and --password-command2 options
2020-09-19 14:30:39 +02:00
Leo R. Lundgren ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
Alexander Neumann 8ce0ce387f Add changelog 2020-09-19 14:22:31 +02:00
Alexander Neumann 3c44598bf6 Drop Go 1.12, require at least 1.13 2020-09-19 14:20:04 +02:00
Alexander Neumann 3bb55fd6bf
Merge pull request #2914 from YoshieraHuang/max-file-size
Support excluding files by size
2020-09-19 14:12:15 +02:00
Alexander Neumann 36efefa7bd Update dependencies, pin fuse library
This updates all dependencies which causes the following changes:
 * The fuse library is set to the last version supporting macOS
 * The minimal version of Go required to build restic is now 1.13
2020-09-19 14:06:37 +02:00
yoshiera ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
MichaelEischer 4dcd6abf37
Merge pull request #2616 from greatroar/no-cache-interface
Remove practically unused cache interface
2020-09-18 19:50:47 +02:00
greatroar cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar 23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar 6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
greatroar 9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Alexander Neumann b10dce541e
Merge pull request #2598 from MichaelEischer/diff-speedup
Diff speedup
2020-09-13 16:42:05 +02:00
Michael Eischer 4f221c4022 Add changelog entry 2020-09-13 16:28:46 +02:00
Michael Eischer f5c448aa65 diff: Optimize diff calculation for shared subtrees
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
2020-09-13 16:28:46 +02:00
Michael Eischer c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann 0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
Alexander Neumann 97950ab81a options: Fix test for Go >= 1.15 2020-09-12 17:36:44 +02:00
Alexander Neumann 59fca85844 Remove darwin/386 from integration tests 2020-09-12 17:09:33 +02:00
Alexander Neumann e207257714 Fix Appveyor 2020-09-12 16:59:03 +02:00
Alexander Neumann 82e1cbed4f Update Go versions 2020-09-12 16:52:21 +02:00
Alexander Neumann 8903b6c88a helpers: Rework list of architectures
This also removes darwin/386 which is not supported on Go 1.15 any more.
2020-09-12 16:39:41 +02:00
Alexander Neumann 93583c01b1 helpers: Fetch modules for release binaries 2020-09-12 16:25:36 +02:00
MichaelEischer 88664ba222
Merge pull request #2893 from MichaelEischer/restore-preallocate
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
Michael Eischer 121233e1b3 Add preallocate tests 2020-09-07 21:41:47 +02:00
Michael Eischer 8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer 2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
rawtaz d3a286928a
Merge pull request #2905 from MichaelEischer/fix-ls-help
ls: Explicitly enforce that the user specifies a snapshot ID
2020-09-05 10:44:01 +02:00
Michael Eischer c46edcd9d6 error strings should not end with punctuation 2020-09-05 10:07:17 +02:00
Michael Eischer 1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer 4784540f04 repository: Simplify worker group code 2020-09-05 10:07:16 +02:00
Michael Eischer 84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer 7ee0964880 crypto: Remove unused error 2020-09-05 10:07:16 +02:00
Michael Eischer f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer e2dc5034d3 restic/file: Use FileType for all File constants
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
Michael Eischer 9a1b3cb5d9 restorer: Remove unused treeID struct member from test 2020-09-05 10:07:16 +02:00
Michael Eischer b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer 068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer 014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer d9a80e07b9 repository: Simplify index age calculation 2020-09-05 10:07:16 +02:00
Michael Eischer d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer 460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer 49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer 2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer 37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer 337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer 2ddb7ffb7e
Merge pull request #2255 from Kidswiss/tar
Fix dumping issues with / and the first sub level
2020-09-01 21:52:17 +02:00
Michael Eischer 81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer 55071ee367
Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path
Add filter by tag and path to stats command
2020-08-31 22:11:01 +02:00
Michael Eischer dcf9ded977 dump: Use slashes as path separators in tar on windows
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
Michael Eischer bcf44a9c3f dump: Properly handle file paths without / prefix
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
2020-08-30 18:25:42 +02:00
Oliver Buschjost a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
Simon Beck da4193c3ef Add changelog 2020-08-30 16:00:48 +02:00
Simon Beck fe6445e0f4 dump: Added unit tests for tar output 2020-08-30 16:00:48 +02:00
MichaelEischer ea81a0e282
Merge pull request #2606 from middelink/fix-323
Add copy functionality.
2020-08-30 10:18:24 +02:00
Simon Beck 80a11960dd dump: Always dump relative paths into tarballs
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
Simon Beck d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck b98598e55f dump: Fix dumping issues with / and the first sub level
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
2020-08-29 21:14:29 +02:00
Michael Eischer d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
2020-08-29 14:28:53 +02:00
MichaelEischer c34c731698
Merge pull request #2702 from MichaelEischer/fix-backup-dir-count
backup: Fix reporting of directory count in summary
2020-08-29 11:29:21 +02:00
Michael Eischer 412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer bbe8b73f03 Update help text of backup command in docs 2020-08-29 10:48:44 +02:00
Michael Eischer 91e8d998cd Add documentation for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer 9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer 15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer 88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer 591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer ec9a53b7e8 copy: Mark and skip previously copied snapshots
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
2020-08-29 10:48:44 +02:00
MichaelEischer 34a3adfd8d
Merge pull request #2898 from aawsome/add-iexclude-file
backup: Add --iexclude-file option
2020-08-28 21:59:16 +02:00
Alexander Weiss 9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
MichaelEischer efb4a981cf
Merge pull request #2904 from jasonccox/consistent-usage-strings
Update usage strings to put flags before args
2020-08-28 20:39:14 +02:00
Jason Cox 2447f3f110 Update usage strings to put flags before args
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
2020-08-28 02:16:22 +00:00
Michael Eischer b25978a53c backup: Fix reporting of directory count in summary
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Michael Eischer b0a8c4ad6c copy: Only process each tree once
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
2020-08-26 22:17:52 +02:00
Michael Eischer 908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar 4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink 7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
MichaelEischer eb7c00387c
Merge pull request #2899 from MichaelEischer/fix-check-progress-crash
Unify progress bar of check and prune commands
2020-08-26 00:00:50 +02:00
Michael Eischer bc0501d72c Add changelog 2020-08-25 23:33:10 +02:00
Michael Eischer 17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
MichaelEischer e915cedc3d
Merge pull request #2897 from restic/rawtaz-github-issue-feature
github: Add question to feature issue template
2020-08-24 23:52:38 +02:00
MichaelEischer cdcaecd27d
Merge pull request #2896 from restic/rawtaz-key-hint
doc: Add missing environment variables
2020-08-24 23:51:05 +02:00
rawtaz b43ab67a22 doc: Add missing environment variables 2020-08-24 00:49:34 +02:00
rawtaz 7ddfd6cabe
github: Add question to feature issue template 2020-08-24 00:22:44 +02:00
Alexander Neumann b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
rawtaz fa135f72bf
Merge pull request #2439 from htrendev/ignore-sync-on-macos
Ignore not supported error on sync() when using local backend
2020-08-22 01:38:49 +02:00
Hristo Trendev 51c22f4223 local backend: ignore not supported error on sync()
Closes #2395
2020-08-22 01:27:07 +02:00
rawtaz 1a5b66f33b
Merge pull request #2879 from 0xMH/master
Added more explantation for --one-file-system option
2020-08-21 19:21:01 +02:00
NoNE da6a34e044 doc: Add more explanation for --one-file-system option 2020-08-21 19:19:13 +02:00
Bruce Dillahunty fe69b83074
Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer 08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer d8b80e9862 Add changelog 2020-08-16 11:36:14 +02:00
Michael Eischer 1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer 575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer 8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer f4b9544ab2 prune: Add test that repack aborts on wrong blob 2020-08-16 11:34:01 +02:00
Michael Eischer 367449dede prune: Reduce memory allocations while repacking
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
Michael Eischer 7042bafea5 prune: Abort repacking when a pack contains a wrong blob
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
Michael Eischer 744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer 3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome 0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
rawtaz 643bbbe156
doc: Correct wording for repository (#2884)
Replaces "backend" with "repository" to match restic output.
2020-08-11 22:30:17 +02:00
MichaelEischer eca0f0ad24
Merge pull request #2863 from aawsome/index-no-duplicates
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
MichaelEischer 08dee8a52b
Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
Alexander Weiss b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss 5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
MichaelEischer 84b6f1ec53
Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer 06fb4ea3f0
Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
rawtaz 0b21ec44b7
Merge pull request #2869 from josephrocca/patch-1
doc: --verbose 2 to --verbose=2
2020-08-04 01:06:58 +02:00
josephrocca bd36731119
`--verbose 2` to `--verbose=2`
`--verbose 2` seems to be incorrect here (gives an error/warning that "the `2` directory does not exist, skipping")
2020-08-04 08:39:43 +10:00
MichaelEischer 38a2f9c07b
Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz 5af2815627
Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00
MichaelEischer b55de2260d
Merge pull request #2868 from MichaelEischer/fix-getusedblobs
prune: Stop progress bar after searching used blobs
2020-08-03 19:58:59 +02:00
Sébastien Gross 9be4fe3e84 cmd/mount: honour --no-lock flag
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-08-03 19:45:39 +02:00
Michael Eischer 05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer 04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar 8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer 66d089e239
Merge pull request #2841 from aawsome/optimize-getUsedBlobs
Extract get used blobs in prune as separate function
2020-08-01 22:40:33 +02:00
MichaelEischer 49d3efe547
Merge pull request #2840 from aawsome/delete-parallel
Make delete parallel
2020-08-01 22:26:15 +02:00
Alexander Weiss 9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss 0eb8553c87 add changelog for #2840 2020-08-01 20:43:18 +02:00
Alexander Weiss d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss 1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss 2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann 4cf9656f12
Merge pull request #2861 from MichaelEischer/fix-rclone-crash
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
Alexander Neumann 2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer 2d7ab9115f Add changelog entry 2020-08-01 12:29:16 +02:00
Michael Eischer a178e5628e Remove duplicate TreeLoader interface 2020-08-01 12:29:16 +02:00
Michael Eischer af66a62c04 FindUsedBlobs: Test that seen blobs are skipped
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
Michael Eischer 9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
Michael Eischer 184103647a FindUsedBlobs: merge seen into blobs BlobSet
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer c81b122374 rclone: Don't panic after unexpected subprocess exit
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
rawtaz 48f97f3567
Merge pull request #2857 from MichaelEischer/incomplete-backup-error
Don't print a stacktrace if some files could not be read
2020-07-29 00:06:25 +02:00
Michael Eischer 3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
MichaelEischer 248c7c3828
Merge pull request #2587 from aawsome/optimize-fuse
Make `restic mount` faster and consume less memory
2020-07-28 23:19:17 +02:00
Alexander Weiss f8316948d1 Optimize FUSE - make command `restic mount` faster and consume less memory
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
MichaelEischer be54ceff66
Merge pull request #2855 from MichaelEischer/rclone-exit
Fix rclone subprocess handling
2020-07-26 22:55:25 +02:00
Michael Eischer ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer 01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer 3cd927d180 rclone: Give rclone time to finish before closing stdin pipe
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
greatroar bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer 8554332894 rclone: Close rclone side of stdio_conn pipes
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
greatroar 3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
MichaelEischer 573a2fb240
Merge pull request #2789 from aawsome/fix-lookup
make Lookup() return all blobs
2020-07-25 21:32:23 +02:00
Michael Eischer c847aace35 Rename Index interface to MasterIndex
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss 9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer 020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar 07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
MichaelEischer 37c95bf5da
Merge pull request #2835 from J0WI/patch-1
Update Go version to 1.14.6 in Docker build script
2020-07-25 17:23:54 +02:00
rawtaz c86d2f23aa
Merge pull request #2854 from restic/rawtaz-patch-1
Clarify questions and text in PR template
2020-07-25 15:41:27 +02:00
rawtaz 96ec04d74d
Clarify questions and text in PR template 2020-07-25 15:38:00 +02:00
J0WI 9c3414374a Update Go version to 1.14.6 in Docker build script 2020-07-25 14:45:36 +02:00
MichaelEischer 3d530dfc91
Merge pull request #2827 from aawsome/archiver-test-contents
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
MichaelEischer c43f5b2664
Merge pull request #2733 from hibbert/support_irsa
Enable support for EKS IRSA
2020-07-25 12:58:43 +02:00
Andy Hibbert 38087e40d9 Update minio-go version so it supports EKS IRSA 2020-07-25 12:41:18 +02:00
MichaelEischer bbc960f957
Merge pull request #2635 from greatroar/optimize-sortbycached
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
greatroar 309598c237 Simplify sortCachedPacksFirst test in internal/repository
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
greatroar 03d23e6faa Speed up blob sorting in internal/repository
name                    old time/op    new time/op    delta
SortCachedPacksFirst-8     208µs ± 3%     186µs ± 3%  -10.74%  (p=0.000 n=10+8)

name                    old alloc/op   new alloc/op   delta
SortCachedPacksFirst-8     213kB ± 0%     139kB ± 0%  -34.62%  (p=0.000 n=10+10)

name                    old allocs/op  new allocs/op  delta
SortCachedPacksFirst-8     1.03k ± 0%     1.03k ± 0%   -0.19%  (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
greatroar b10acd2af7 Test and benchmark blob sorting in internal/repository 2020-07-25 12:12:58 +02:00
Alexander Weiss 9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
rawtaz 5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
Michael Eischer 7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer 34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
MichaelEischer bcd47ec3a2
Merge pull request #2818 from aawsome/merge-index-chaining
Merge index (based on chaining index implementation)
2020-07-22 22:05:36 +02:00
Alexander Weiss a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Peter Schultz 758b44b9c0 gs: support authentication with access token
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
Alexander Weiss 3b7a3711e6 Add more realistic index benchmarks
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
Michael Eischer 9b0e718852 checker: Test that blob types are not confused 2020-07-20 23:43:47 +02:00
MichaelEischer 82c908871d
Merge pull request #2812 from greatroar/chaining
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
Michael Eischer ddf0b8cd0b checker: Properly distinguish between data and tree blobs
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
Michael Eischer 2d0c138c9b checker: Test that check only decodes trees once
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
Michael Eischer ef325ffc02 checker: Cleanup error handling code
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
Michael Eischer 0f67ae813a Add changelog 2020-07-20 22:51:53 +02:00
Michael Eischer 7a165f32a9 checker: Traverse trees in depth-first order
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation.  In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.

Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
Michael Eischer 36c69e3ca7 checker: Unify blobs, processed trees and referenced blobs map
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
Michael Eischer 35d8413639 checker: Remove dead index map 2020-07-20 22:37:31 +02:00
Michael Eischer c66a0e408c checker: Reduce cost of debug log
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
Michael Eischer 70f4c014ef checker: Decode identical tree nodes only once
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
Michael Eischer f0d8710611 Add benchmark for checker scaling with snapshot count 2020-07-20 22:37:31 +02:00
MichaelEischer bd3e280f6d
Merge pull request #2546 from Summerdave/fix-backup-error-code
Return an error when errors occured during backup
2020-07-20 22:31:46 +02:00
MichaelEischer 2746dcdb5f
Merge pull request #2845 from restic/update-chunker
Update chunker
2020-07-20 22:20:28 +02:00
David Sommer 5729d967f5 backup: Return exit status code 3 when failing to read source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
2020-07-20 22:19:01 +02:00
Alexander Neumann f9f6124558 Update chunker 2020-07-20 22:07:02 +02:00
Alexander Neumann 8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar 7bda28f31f Chaining hash table for repository.Index
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
2020-07-19 13:58:22 +02:00
greatroar 255ba83c4b Parallel index benchmarks + benchmark optimizations
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.

BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)

Results on linux/amd64, -benchtime=3s -count=20:

name                                     old time/op    new time/op       delta
PackerManager-8                             178ms ± 0%        178ms ± 0%           ~     (p=0.165 n=20+20)
DecodeIndex-8                              13.6µs ± 2%  4539886.8µs ± 0%  +33293901.38%  (p=0.000 n=20+18)
IndexHasUnknown-8                          44.4ns ± 7%       44.4ns ± 5%           ~     (p=0.873 n=20+19)
IndexHasKnown-8                            49.2ns ± 3%       48.3ns ± 0%         -1.86%  (p=0.000 n=20+16)
IndexAlloc-8                                802ms ± 1%        758ms ± 1%         -5.51%  (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8              124ns ± 1%        122ns ± 0%         -1.41%  (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8            373ns ± 2%        369ns ± 2%         -1.13%  (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8      67.8ns ± 3%       68.4ns ± 5%           ~     (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8     316ns ± 3%        315ns ± 3%           ~     (p=0.846 n=20+20)
SaveAndEncrypt-8                           30.5ms ± 1%       30.2ms ± 1%         -1.09%  (p=0.000 n=19+19)
LoadTree-8                                  527µs ± 1%        540µs ± 1%         +2.37%  (p=0.000 n=19+20)
LoadBlob-8                                 5.65ms ± 0%       5.64ms ± 0%         -0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                           7.07ms ± 2%       5.93ms ± 0%        -16.15%  (p=0.000 n=19+20)
LoadIndex-8                                32.1ms ± 2%       25.1ms ± 0%        -21.64%  (p=0.000 n=20+18)

name                                     old speed      new speed         delta
PackerManager-8                           296MB/s ± 0%      296MB/s ± 0%           ~     (p=0.159 n=20+20)
SaveAndEncrypt-8                          138MB/s ± 1%      139MB/s ± 1%         +1.10%  (p=0.000 n=19+19)
LoadBlob-8                                177MB/s ± 0%      177MB/s ± 0%         +0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                          141MB/s ± 2%      169MB/s ± 0%        +19.24%  (p=0.000 n=19+20)

name                                     old alloc/op   new alloc/op      delta
PackerManager-8                            91.8kB ± 0%       91.8kB ± 0%           ~     (p=0.826 n=19+12)
IndexAlloc-8                                786MB ± 0%        786MB ± 0%         +0.01%  (p=0.000 n=20+20)
SaveAndEncrypt-8                           21.0MB ± 0%       21.0MB ± 0%         -0.00%  (p=0.012 n=20+19)

name                                     old allocs/op  new allocs/op     delta
PackerManager-8                             1.41k ± 0%        1.41k ± 0%           ~     (all equal)
IndexAlloc-8                                 977k ± 0%         977k ± 0%         +0.01%  (p=0.022 n=20+20)
SaveAndEncrypt-8                             73.0 ± 0%         73.0 ± 0%           ~     (all equal)
2020-07-19 13:58:05 +02:00
MichaelEischer 7dc200c593
Merge pull request #2832 from alrs/fix-fuse-test-err
internal/fuse: fix dropped test error
2020-07-16 22:28:20 +02:00
Lars Lehtonen 9ac90cf5cd
internal/fuse: fix dropped test error 2020-07-12 21:42:31 -07:00
MichaelEischer b84f5177cb
Merge pull request #2790 from greatroar/fix-quadratic-read
Fix quadratic file reading in restic mount
2020-07-12 18:42:14 +02:00
greatroar 4cf1c8e8da Changelog entry for subquadratic reading in mount 2020-07-12 18:27:16 +02:00
greatroar 58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
greatroar d42c169458 Fix quadratic file reading in restic mount 2020-07-12 18:27:16 +02:00
rawtaz 8598bb042b
Merge pull request #2769 from darkdragon-001/patch-1
Update documentation for environment variables
2020-07-12 00:40:21 +02:00
darkdragon-001 c6b74962df Update 040_backup.rst
Changes proposed in #2763:

- Adding `RESTIC_CACHE_DIR` environment variables (introduced in #2425 for Unix and #2607 for Mac, Win).
- Adding used system-wide environment variables with links to the corresponding section.
2020-07-12 00:31:56 +02:00
MichaelEischer 2c72924ffb
Merge pull request #2813 from greatroar/encrypt-benchmark
Fix repository_test.BenchmarkSaveAndEncrypt
2020-07-05 17:52:10 +02:00
greatroar 02bec13ef2 Fix repository_test.BenchmarkSaveAndEncrypt
The benchmark was actually testing the speed of index lookups.

name              old time/op    new time/op        delta
SaveAndEncrypt-8     101ns ± 2%    31505824ns ± 1%    +31311591.31%  (p=0.000 n=10+10)

name              old speed      new speed          delta
SaveAndEncrypt-8  41.7TB/s ± 2%       0.0TB/s ± 1%         -100.00%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op       delta
SaveAndEncrypt-8     1.00B ± 0%  20989508.40B ± 0%  +2098950740.00%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op      delta
SaveAndEncrypt-8      0.00             123.00 ± 0%            +Inf%  (p=0.000 n=10+9)

(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
MichaelEischer 64976b1a4d
Merge pull request #2815 from scelfo/patch-1
--hostname flag is deprecated
2020-06-29 22:04:56 +02:00
Tony Scelfo 6a607d6ded
--hostname flag is deprecated
Update documentation to follow the deprecation warning "Flag --hostname has been deprecated, use --host" that is given on the latest build(s).
2020-06-29 12:53:57 -06:00
rawtaz 6fedf1a7f4
Merge pull request #2809 from lessfoobar/patch-2
Update 080_examples.rst
2020-06-26 00:12:05 +02:00
@lessfoobar df946fd9f8
Update 080_examples.rst 2020-06-26 00:01:30 +02:00
Jakub Wilk 4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
rawtaz 1bc80c3c8d
Merge pull request #2802 from andreaso/cachedir-https-links
Use https:// for linking CACHEDIR.TAG specs
2020-06-21 20:01:42 +02:00
Andreas Olsson 0fcef2ec23
Use https:// for linking CACHEDIR.TAG specs
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
2020-06-21 19:36:26 +02:00
MichaelEischer 212607dc8a
Merge pull request #2760 from greatroar/backend-benchmark
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
greatroar 55c3a90a0d Clarify rclone-over-SSH docs
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
2020-06-17 15:22:20 +02:00
greatroar 190d8e2f51 Flatten backend.LimitedReadCloser structure
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
greatroar f4cd2a7120 Make backend benchmarks fairer by removing checks
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
MichaelEischer aba270df7e
Merge pull request #2786 from aawsome/memory-list-blobs
Loop over index files for 'list blobs'
2020-06-15 22:27:29 +02:00
Alexander Weiss b5543cff5d Loop over index files for 'list blobs'
=> reduces memory consumption a lot!
2020-06-14 17:25:34 +02:00
Alexander Neumann 285b5236c2
Merge pull request #2684 from MichaelEischer/cleanup-output
Cleanup command output code
2020-06-14 11:25:40 +02:00
Michael Eischer bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer 182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
2020-06-14 11:08:11 +02:00
MichaelEischer 74bc7141c1
Merge pull request #2781 from aawsome/reduce-index-memory
Reduce index memory
2020-06-14 10:58:28 +02:00
Alexander Weiss 1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss ce4a2f4ca6 save packIDs and duplicates separately
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
Alexander Weiss cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss 7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz 1d66bb9e62
Merge pull request #2784 from MichaelEischer/minio-sha256-changelog
Add changelog for #2709
2020-06-13 18:33:54 +02:00
Michael Eischer 0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss d3c59d18e5 Fix inconsistency of saving/loading config file
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
MichaelEischer dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer 6896c6449b
Merge pull request #2779 from greatroar/archiver-comment
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
MichaelEischer 735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss 70347e95d5 disable index uploads for prune command
+ modifications of changelog
2020-06-12 09:24:38 +02:00
greatroar 0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss 91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
MichaelEischer fae7f78057
Merge pull request #2755 from greatroar/solaris-workaround
Remove Solaris build workaround in integration tests
2020-06-10 23:30:51 +02:00
MichaelEischer ac9ec4b990
Merge pull request #2778 from MichaelEischer/fix-unlock-hint
Revive hint to the unlock command if a repository is locked
2020-06-10 21:26:38 +02:00
Michael Eischer 087c770161 Revive hint to the unlock command if a repository is locked
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
MichaelEischer 6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex
Change condition for full index
2020-06-10 20:40:19 +02:00
Alexander Weiss 8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz 26704be17f
Merge pull request #2776 from nunoperalta/patch-1
Fix #2774 - make unable to umount message clearer
2020-06-07 17:19:50 +02:00
nunoperalta 2c3360db98
Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer cba6ad8d8e
Merge pull request #2711 from greatroar/update-x-net
Update golang.org/x/net for fewer allocations in http2
2020-05-25 20:43:29 +02:00
greatroar 2a3312ac35 Remove Solaris build workaround in integration tests
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
2020-05-25 10:33:36 +02:00
greatroar c35c4e0cbf Update golang.org/x/net for fewer allocations in http2
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
2020-05-24 16:39:17 +02:00
MichaelEischer 84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
MichaelEischer f12f9ae240
Merge pull request #2741 from tbm/typos
Fix typos
2020-05-21 23:40:12 +02:00
Martin Michlmayr 5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer 32ac5486e9
Merge pull request #2732 from restic/rawtaz-update-contributing
Reword parts of Providing Patches in CONTRIBUTING.md
2020-05-12 21:37:08 +02:00
rawtaz c4336978eb
Reword parts of Providing Patches in CONTRIBUTION.md
Primarily makes the request that contributors create a new branch for their changes stand out more.
2020-05-12 14:03:57 +02:00
greatroar 649cbec6c5 Simplify build tags for restic mount
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
MichaelEischer b17bd7f860
Merge pull request #2719 from greatroar/update-x-crypto
Update x/crypto to stop poly1305 segfaulting on ARM
2020-05-09 00:02:18 +02:00
greatroar 68f1e9c524 Update x/crypto to fix poly1305 on ARM
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
2020-05-05 11:05:35 +02:00
MichaelEischer 1ee2306033
Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt
backup: Avoid race between password prompt and open repository message
2020-05-01 23:58:55 +02:00
Michael Eischer c882a92cd6 backup: Avoid race between password prompt and open repository message
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
2020-05-01 23:42:16 +02:00
MichaelEischer f54db5d796
Merge pull request #2713 from greatroar/unused
Move Index.FindBlob to tests
2020-05-01 20:41:59 +02:00
MichaelEischer 843e7f404e
Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround
Revert "Darwin test fix: allow 1μs timestamp difference"
2020-05-01 18:13:38 +02:00
Michael Eischer d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference"
This reverts commit f3016a9096.
2020-05-01 17:11:30 +02:00
greatroar 9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar f97a680887 Fix repository benchmarks
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
greatroar 42a3db05b0 Use Minio's optimized SHA-256
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
MichaelEischer 070d43e290
Merge pull request #2704 from azak-azkaran/patch-1
Update 030_preparing_a_new_repo.rst
2020-04-23 20:48:34 +02:00
MichaelEischer d4bd32a37e
Merge pull request #2640 from greatroar/simplify-loadblob-usage
Simplify Repository.LoadBlob usage
2020-04-23 20:23:35 +02:00
greatroar e7d7b85d59 Merge Repository.{LoadBlob,loadBlob}
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:

name              old time/op    new time/op    delta
LoadTree-8           478µs ± 1%     481µs ± 2%    ~     (p=0.315 n=9+10)
LoadBlob-8          11.6ms ± 1%    11.6ms ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8    13.3ms ± 3%    13.3ms ± 3%    ~     (p=0.905 n=10+9)
LoadIndex-8         33.6ms ± 2%    33.2ms ± 1%  -1.15%  (p=0.028 n=10+9)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.1kB ± 0%  -0.23%  (p=0.000 n=10+10)
LoadBlob-8          2.28kB ± 0%    2.18kB ± 0%  -4.21%  (p=0.000 n=10+10)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%    ~     (all equal)
LoadIndex-8         5.22MB ± 0%    5.22MB ± 0%    ~     (p=0.631 n=10+10)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       651 ± 0%  -0.15%  (p=0.000 n=10+10)
LoadBlob-8            24.0 ± 0%      23.0 ± 0%  -4.17%  (p=0.000 n=10+10)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%    ~     (all equal)
LoadIndex-8          30.2k ± 0%     30.2k ± 0%    ~     (p=0.610 n=10+10)

name              old speed      new speed      delta
LoadBlob-8        86.4MB/s ± 1%  85.9MB/s ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8  75.4MB/s ± 3%  75.4MB/s ± 3%    ~     (p=0.858 n=10+9)
2020-04-23 10:04:20 +02:00
greatroar be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
azak-azkaran c5100d5632
Update 030_preparing_a_new_repo.rst
fixed markdown for wasabi region link
2020-04-23 09:24:19 +02:00
MichaelEischer 956a1b0f96
Merge pull request #2294 from BenWiederhake/debugsyms
build: Preserve debug symbols in debug build
2020-04-19 17:26:03 +02:00
Ben Wiederhake fab626a3df build: Preserve debug symbols in debug and profile build
Signed-off-by: Ben Wiederhake <BenWiederhake.GitHub@gmx.de>
2020-04-19 14:48:40 +02:00
MichaelEischer 4f00564574
Merge pull request #2621 from greatroar/fixes
Some small fixes
2020-04-18 18:07:08 +02:00
MichaelEischer f77477129f
Merge pull request #2584 from greatroar/mount-cache-uid-gid
Cache uid and gid for top directories in internal/fuse
2020-04-18 17:45:14 +02:00
greatroar 2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
greatroar 8fb2c0d3c1 Typo in crypto test name 2020-04-18 17:39:06 +02:00
greatroar 072cf7b02d Fix debug messages in internal/fuse 2020-04-18 17:39:06 +02:00
greatroar df66daa5c9 Fix context usage in backend tests
Found by go vet. This is also the only complaint is has.
2020-04-18 17:39:06 +02:00
MichaelEischer 9790d8ce1c
Merge pull request #2668 from MichaelEischer/fix-stats-blobs-crash
stats: Fix crash in blobs-per-file mode on missing blob
2020-04-18 17:21:26 +02:00
MichaelEischer 16710454f4
Merge pull request #2628 from MichaelEischer/one-element-pack-lists
cache: Don't sort one element pack lists
2020-04-18 17:09:06 +02:00
MichaelEischer 08ec6c9f17
Merge pull request #2677 from MichaelEischer/complain-about-invalid-indexes
rebuild_index: Report invalid packs that were ignored
2020-04-18 16:57:50 +02:00
MichaelEischer 7910ff4c0e
Merge pull request #2648 from nairb774/iowritestring
termstatus: Use io.WriteString to output messages.
2020-04-18 13:48:42 +02:00
MichaelEischer c4da9d1e90
Merge pull request #2638 from greatroar/no-close-in-packer
Don't Close in Packer.Finalize
2020-04-18 13:07:21 +02:00
rawtaz 3ed61987a2
Merge pull request #2695 from restic/rawtaz-doc-exclude-file-tilde
doc: Add note on tilde expansion in exclude files
2020-04-18 13:01:17 +02:00
rawtaz 9dba7a2577
doc: Add note on tilde expansion in exclude files
Explains to the reader that tilde expansion does not work in exclude files, and that they should instead use the $HOME variable.
2020-04-18 12:47:47 +02:00
MichaelEischer a1352906e2
Merge pull request #2622 from greatroar/optimize-packer-manager
Fix PackerManager benchmark and optimize hashing.Writer
2020-04-18 12:46:34 +02:00
MichaelEischer b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered
Make all signal.Notify channels buffered
2020-04-18 11:31:02 +02:00
MichaelEischer f033850aa0
Merge pull request #2692 from MichaelEischer/fix-archiveraborttest
archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
2020-04-13 18:36:48 +02:00
Michael Eischer bdf7ba20cb archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.

As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.

Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
2020-04-13 18:23:17 +02:00
rawtaz 3ee6b8ec63
Merge pull request #2689 from MichaelEischer/fix-background-hang
Fix shutdown hang when restic is started as background job
2020-04-12 22:39:44 +02:00
Michael Eischer 4a400f94bb Add changelog 2020-04-12 22:27:09 +02:00
Michael Eischer 1a1c572bac Fix shutdown hang when restic is started as background job
restic uses a cleanup hook to ensure that it restores the terminal
configuration to a sane state, when restic is interrupted while reading
a password from the terminal. However, this causes a problem, when
restic runs in a background job, as reconfiguring a terminal will cause
a SIGTTOU to be sent to restic pausing it. Therefore, restic seems to
hang on shutdown when it was running in the background.

This commit changes the behavior to only restore the terminal
configuration if restic was interrupted while reading a password from
the terminal. As reading a password from the terminal requires that
restic is in the foreground, this should avoid restic getting stopped.

Fixes #2298
Issue introduced in #402
2020-04-12 22:27:09 +02:00
rawtaz 5a7c27ddb6
Merge pull request #2681 from MichaelEischer/optimize-debug
Reduce memory usage and startup time of debug command
2020-04-04 00:08:14 +02:00
Michael Eischer fb842759fc debug: don't load the repository index
The existing commands don't need a loaded repository index which can
take several minutes to load on larger repositories.
2020-04-04 00:01:01 +02:00
Michael Eischer 7aa2f8a61e debug: get stdout/stderr from gopts/globalOptions 2020-04-04 00:01:01 +02:00
Michael Eischer 08bf3bae79 debug: explicitly pass stdout to dump functions 2020-04-03 23:32:44 +02:00
rawtaz e7b741b2d7
Merge pull request #2682 from MichaelEischer/cleanup-cli-paramter-names
Cleanup CLI parameter names for backup / global flags
2020-04-03 21:35:34 +02:00
Michael Eischer 6bee62e346 Update doc excerpts for `--help`
This adds some previously missing changes and the new paramters names
from the previous commit.
2020-04-03 19:49:06 +02:00
Michael Eischer bc74cd3ae5 backup/global: Use proper name for command line argument parameters
Several paramters printed a generic "string" or "stringArray" name.
2020-04-03 19:49:04 +02:00
Michael Eischer 90243ed1c4 rebuild_index: Report invalid packs that were ignored 2020-04-02 22:38:31 +02:00
Michael Eischer 0ce81d88b6 stats: Fix crash in blobs-per-file mode on missing blob
In a damaged repository with a missing blob, the error message tried to
dereference the subtreeID field of the current node, which is a file
however. Said field is set to nil for a file thus causing a segfault
when dereferenced.

Fix this by using the actual parentTreeID.
2020-03-27 22:17:54 +01:00
rawtaz c03bc88b29
Merge pull request #2669 from MichaelEischer/doc-cifs-linux-bug
doc: Warn about compatibility issues with CIFS and restic
2020-03-26 23:01:06 +01:00
Brian Atkinson b8da7b1f4d termstatus: Use io.WriteString to output messages.
The previous implementation was repeating the implementation that is
found inside of io.WriteString. Simplify by making use of the stdlib's
implementation.
2020-03-26 14:55:00 -07:00
Michael Eischer f1b4d97945 doc: Warn about compatibility issues with CIFS and restic
On Linux CIFS (SMB) seems to be incompatible with the async preemption
implementation of Go 1.14. CIFS seems not to restart syscalls (open,
read, chmod, readdir, ...) as expected by Go, which sets SA_RESTART for
its signal handler to have syscalls restarted automatically. This leads
to Go passing up lots of EINTR return codes to restic.

See https://github.com/restic/restic/issues/2659 for a detailed explanation.
2020-03-26 21:52:37 +01:00
Peter Schultz 90fc639a67 Allow specifying user and host when adding keys
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
2020-03-23 13:11:10 +01:00
rawtaz 2b5a6d255a
Merge pull request #2660 from restic/rawtaz-delete-old-issue-template
Delete ISSUE_TEMPLATE.md (not used anymore)
2020-03-21 20:48:39 +01:00
rawtaz f004dbe605
Delete ISSUE_TEMPLATE.md (not used anymore)
Nowadays the ISSUE_TEMPLATE/ directory and its files are used for this feature.
2020-03-21 20:30:19 +01:00
rawtaz 9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close
internal/restic: close os.File after checking for error
2020-03-18 22:23:20 +01:00
rawtaz 7d9300efca
Merge pull request #2637 from greatroar/unused
Remove Go 1.5 compatibility code
2020-03-18 22:22:15 +01:00
rawtaz c38aaaa768
Merge pull request #2652 from greatroar/upgrade-fuse
Upgrade bazil.org/fuse and remove restic mount --allow-root
2020-03-18 22:21:50 +01:00
greatroar 8941041355 Upgrade bazil.org/fuse to version that fixes FreeBSD opBmap bug 2020-03-18 10:00:39 +01:00
greatroar 18fee4806f Remove broken --allow-root from restic mount 2020-03-17 23:35:06 +01:00
greatroar 47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar 74a64c47e4 Move testing logic to test file in internal/pack 2020-03-09 14:32:28 +01:00
greatroar a23e9c86ba Remove closing logic from Packer.Finalize
The method only ever receives *hashing.Writers, which don't implement
io.Closer. These come from packerManager.findPacker and have their
actual writers closed in Repository.savePacker. Moving the closing logic
to hashing.Writer results in "file already closed" errors.
2020-03-09 14:31:45 +01:00
greatroar 4de12bf593 Remove restic.RandReader
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
2020-03-09 10:00:28 +01:00
rawtaz c542a509f0
Merge pull request #2633 from greatroar/fix-ssh-commandline
Revert "Put host last in SSH command line"
2020-03-08 17:29:41 +01:00
greatroar 8cf3bb8737 Revert "Put host last in SSH command line"
This reverts commit e1969d1e33.
2020-03-08 16:45:33 +01:00
Michael Eischer b46cc6d57e repository: Don't sort one element pack lists
When loading a blob, restic first looks up pack files containing the
blob. To avoid unnecessary work an already cached pack file is preferred.
However, if there is only a single pack file to choose from (which is
the normal case) sorting the one-element list won't change anything.
Therefore avoid the unnecessary cache check in that case.
2020-03-07 10:26:06 +01:00
Lars Lehtonen 4a2156d3f0
internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
greatroar 41fee11f66 Micro-optimization for hashing.Writer/PackerManager
name             old time/op    new time/op    delta
PackerManager-8     247ms ± 1%     246ms ± 1%  -0.43%  (p=0.001 n=18+18)

name             old speed      new speed      delta
PackerManager-8   213MB/s ± 1%   214MB/s ± 1%  +0.43%  (p=0.001 n=18+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8    92.2kB ± 0%    91.5kB ± 0%  -0.82%  (p=0.000 n=19+20)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.43k ± 0%     1.41k ± 0%  -1.67%  (p=0.000 n=20+20)
2020-03-05 22:30:04 +01:00
greatroar b592614061 Improve PackerManager benchmark
The previous benchmark spent much of its time allocating RNGs and
generating too many random numbers. It now spends 90% of its time
hashing and half of the rest writing to files.

name             old time/op    new time/op    delta
PackerManager-8     319ms ± 1%     247ms ± 1%  -22.48%  (p=0.000 n=20+18)

name             old speed      new speed      delta
PackerManager-8   143MB/s ± 1%   213MB/s ± 1%  +48.63%  (p=0.000 n=10+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8     635kB ± 0%      92kB ± 0%  -85.48%  (p=0.000 n=10+19)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.64k ± 0%     1.43k ± 0%  -12.76%  (p=0.000 n=10+20)
2020-03-05 22:30:03 +01:00
greatroar b7c3039eb2 Remove Go 1.5 compatibility code from PackerManager benchmark
This alone is enough to speed up the benchmark by ~10%.
2020-03-05 22:29:06 +01:00
Alexander Neumann a307797c11
Merge pull request #2612 from restic/rawtaz-maintainer-pr
Add notes about maintainer edit access in PRs
2020-03-01 21:24:34 +01:00
rawtaz a03f107144
Add note about allowing maintainers edits in PRs. 2020-03-01 21:12:11 +01:00
rawtaz a141ab1bda
Add maintaner edit checkbox to PR template 2020-03-01 21:05:52 +01:00
Alexander Neumann 52abec967f
Merge pull request #2605 from middelink/fix-2604
Fix running tests on a SELinux enabled system
2020-03-01 20:25:28 +01:00
Pauline Middelink 2828a9c2b0 Fix running tests on a SELinux enabled system
Archivers TestMetadataChanged incorrectly clears the Extended Attributes
from the expected metadata of the temporary file. This is incorrect as on
SELinux enabled filesystem, as the kernel will automaticly add a SElinux
label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we
still need to clear them if there are no attributes found.
2020-03-01 20:23:22 +01:00
rawtaz cec7d581f3
Merge pull request #2610 from restic/remove-vendor
Remove vendored dependencies
2020-03-01 20:22:45 +01:00
Alexander Neumann 12aa1e61da CI: Enable Go Module Proxy 2020-03-01 19:58:20 +01:00
Alexander Neumann 95da6c1c1d
Merge pull request #2589 from greatroar/no-stable-sort
Replace sort.Stable by sort.Strings
2020-03-01 19:40:28 +01:00
Alexander Neumann 0c03a80fc4
Merge pull request #2592 from greatroar/sftp-ipv6
Support IPv6 in SFTP backend
2020-03-01 19:38:44 +01:00
Alexander Neumann b1c77172c2 Add entry to changelog 2020-03-01 19:32:35 +01:00
Alexander Neumann c0373cd307 Remove -mod=vendor from all documentation and code 2020-03-01 19:32:35 +01:00
Alexander Neumann 28121090c2 Remove vendor from CI tests 2020-03-01 19:32:35 +01:00
Alexander Neumann 44a57d66c3 Remove vendor from build scripts 2020-03-01 11:30:02 +01:00
Alexander Neumann 266f9dbe16 Remove vendor dir 2020-03-01 11:20:42 +01:00
greatroar 60e4a88f17 Changelog entry for SFTP w/ IPv6 addresses 2020-03-01 11:04:06 +01:00
Alexander Neumann c50f91b7f9
Merge pull request #2602 from greatroar/no-bufpool
Remove sync.Pool from internal/repository
2020-03-01 10:50:57 +01:00
Alexander Neumann e851d29565
Merge pull request #2608 from greatroar/simplify-termstatus
Simplify termstatus
2020-03-01 10:48:29 +01:00
Alexander Neumann b67b7ebfe6
Merge pull request #2583 from greatroar/unused
Remove some unused or duplicated code
2020-03-01 10:46:17 +01:00
greatroar 751eba0e68 Remove unnecessary pipe checks in termstatus
canUpdateStatus has already determined that the file descriptor is not a
pipe.
2020-02-29 18:03:49 +01:00
greatroar 7447c44484 Use golang.org/x/sys/windows in termstatus
Some functionality is missing, but at least the types are all defined.
Replaced short, word, dword by their Go names to match the x/sys
convention.
2020-02-29 18:03:49 +01:00
greatroar c8a672fa29 Remove code copy-pasted from x/crypto/ssh/terminal 2020-02-29 18:03:49 +01:00
greatroar 863ba76494 Drop mattn/go-isatty in favor of crypto/ssh/terminal 2020-02-29 18:03:47 +01:00
greatroar 8526cc6647 Remove sync.Pool from internal/repository
The pool was used improperly, causing more allocations to be
performed than without it.

name              old time/op    new time/op    delta
SaveAndEncrypt-8    36.8ms ± 2%    36.9ms ± 2%    ~     (p=0.218 n=10+10)

name              old speed      new speed      delta
SaveAndEncrypt-8   114MB/s ± 2%   114MB/s ± 2%    ~     (p=0.218 n=10+10)

name              old alloc/op   new alloc/op   delta
SaveAndEncrypt-8    21.1MB ± 0%    21.0MB ± 0%  -0.44%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
SaveAndEncrypt-8      79.0 ± 0%      77.0 ± 0%  -2.53%  (p=0.000 n=10+10)
2020-02-29 17:54:46 +01:00
rawtaz 694b7a17e7
Merge pull request #2607 from greatroar/cachedir
Honor RESTIC_CACHE_DIR on Mac and Windows
2020-02-28 20:41:32 +01:00
greatroar 5cd0bce452 Honor RESTIC_CACHE_DIR on Mac and Windows 2020-02-28 15:44:32 +01:00
rawtaz 58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames
Allow multiple hostnames tags
2020-02-27 08:35:23 +01:00
rawtaz 65d3fb6b33
Merge pull request #2603 from greatroar/restorer-waitgroup
Fix unsafe sync.WaitGroup usage in restorer.fileRestorer
2020-02-27 00:30:59 +01:00
rawtaz f165048172
Merge pull request #2582 from greatroar/no-go1.9
Remove Go 1.9 compatibility code
2020-02-27 00:15:06 +01:00
greatroar de5516a90e Fix sync.WaitGroup usage in restorer.fileRestorer 2020-02-27 00:07:49 +01:00
greatroar 4f6fd9fb98 Remove remnant of Go 1.9 compatibility code from tests 2020-02-26 22:23:38 +01:00
Alexander Weiss 9a9101d144 Support specifying multiple host flags for various commands
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
2020-02-26 22:17:59 +01:00
rawtaz 616f9499ae
Merge pull request #2600 from restic/update-go
Update Go version to >= 1.11, add Go 1.14
2020-02-26 21:29:30 +01:00
Alexander Neumann a40ac37550 Add changelog file 2020-02-26 20:55:12 +01:00
Alexander Neumann 99fd80a585 Remove all workarounds for Go < 1.11 2020-02-26 20:35:13 +01:00
Alexander Neumann 2464f7c4d1 Update Go versions, drop Go 1.10 2020-02-26 20:35:13 +01:00
rawtaz b5c7778428
Merge pull request #2195 from ifedorenko/out-of-order-restore-no-progress
restorer: allow writing target file blobs out of order
2020-02-26 20:07:51 +01:00
Igor Fedorenko c52198d12c restorer: go mod vendor; go mod tidy
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:15:04 +01:00
Igor Fedorenko f17ffa0283 restorer: Allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00
greatroar 5e2afd91e7 Assert that archiver.Tree implements fmt.Stringer 2020-02-26 11:05:38 +01:00
greatroar 79b882e901 Merge duplicated readdir functionality
internal/archiver.readdir and internal/fs.ReadDir were unused.

internal/fs.ReadDirNames and internal/archiver.readdirnames were doing
nearly the same thing, except one sorted its output and opened with
fs.O_NOFOLLOW. Both were only used in internal/archiver.
2020-02-26 11:05:38 +01:00
greatroar 1b502fa9ef Cache uid and gid for top directories in internal/fuse 2020-02-24 10:46:09 +01:00
greatroar e1969d1e33 Put host last in SSH command line
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar 6ac6bca7a1 Support IPv6 in SFTP backend
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
greatroar 3a6feb0596 Replace sort.Stable by sort.Strings
Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings
for a slice of unique strings.
2020-02-18 19:41:06 +01:00
greatroar 2f8aa2ce30 Remove unused fs.FS from archiver.FileSaver 2020-02-18 10:39:14 +01:00
rawtaz f2bf06a419
Merge pull request #2540 from dp-github/issue-2531
Count hard-linked files correctly in stats command
2020-02-15 21:06:46 +01:00
David Potter 71900248ab Correct #2531 (`stats` restore-size calculates hard links incorrectly) 2020-02-15 19:34:22 +01:00
David Potter 23055aaadf Correct #2537 (cmd_stats file counting issue) 2020-02-15 19:34:16 +01:00
rawtaz 8bf6a3af97
Merge pull request #2285 from curiousleo/bugfix/2281-json-output
Fix JSON Printf issues with format verbs
2020-02-15 02:00:57 +01:00
curiousleo 27d6e5e186 Add changelog file. 2020-02-15 01:32:43 +01:00
curiousleo 4214b1746e Ensure Print{,f,ln} use global stdout 2020-02-15 01:32:43 +01:00
curiousleo f6f240573a Don't Printf already formatted output
Fixes #2281.
2020-02-15 01:32:43 +01:00
rawtaz ecdf49679e
Merge pull request #2579 from MichaelEischer/fix-flaky-archiver-test
Fix flaky TestArchiverAbortEarlyOnError
2020-02-15 01:29:37 +01:00
Michael Eischer e1f722d266 archiver: Fix flaky TestArchiverAbortEarlyOnError
Each of the random test files was split into the same five blobs. The
test fails once the fifth blob is passed on to `SaveBlob`. That is for
certain interleavings of goroutine execution it would be possible for
the test to trigger the testErr just after storing the first file.

The fixed test uses a different file content for each of the nine files
and fails after writing the fourth blob. The file content is also small
enough to ensure that for each file only a single blob is saved. This
guarantees that the test cannot fail before reading the first four
files. FileReadConcurrency = 2 allows up to two files queued for
processing. Therefore the test can at most open the sixth file before it
has to save the fourth file / blob which triggers the testErr.
2020-02-14 23:16:13 +01:00
rawtaz 2d47381b1d
Merge pull request #2545 from MichaelEischer/fix-racy-backup-json
Fix racy json output for backup command
2020-02-13 22:33:17 +01:00
Michael Eischer 294ca55967 Add changelog entry 2020-02-13 21:14:20 +01:00
Michael Eischer 78c518ccac Print backup summary after status output is shutdown 2020-02-13 21:14:20 +01:00
Michael Eischer 42a3292bcf Better name for jsonstatus package
internal/ui/jsonstatus and termstatus sound similar but are not related
in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status`
are the counterparts. Rename the latter to `internal/ui/json/backup` to
make this clear.
2020-02-13 21:14:20 +01:00
Michael Eischer ef70a2fcb3 Fix mangled JSON output by backup command
jsonstatus wrote the JSON output without synchronization to the
stdio_wrapper which caused mangling between different status lines.

Use the Print and Error methods of termstatus instead which use a
central goroutine to synchronize output.
2020-02-13 21:14:20 +01:00
rawtaz af20015725
Merge pull request #2287 from benchti/patch-1
doc: Correct parameter order in dump example
2020-02-13 01:54:10 +01:00
rawtaz 680a14afa1
Merge pull request #2530 from restic/fix-sftp-mkdirall
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
rawtaz bf199e5ca7
Minor speling corection. 2020-02-13 00:47:09 +01:00
rawtaz ae127a412d
Merge pull request #2337 from rfjakob/favicon
docs: switch to gopher favicon
2020-02-13 00:14:33 +01:00
Jakob Unterwurzacher 5ae7e6f945 docs: switch to gopher favicon
The favicon on restic.readthedocs.io still contained the old
superman-style logo.

This changes it to the 32x32 gopher icon also used on https://forum.restic.net/ ,
just converted to .ico using Gimp.

Tested by building the documentation and opening index.html in Chrome.
The new favicon looks fine.
2020-02-13 00:09:37 +01:00
rawtaz d8da9c4401
Merge pull request #2577 from alrs/fix-internal-errs
internal: Fix code and test dropped errors
2020-02-12 23:41:54 +01:00
rawtaz daf3bfc882
Merge pull request #2461 from rigtorp/man-exit-status
Add documentation on exit status codes to man pages
2020-02-12 23:17:49 +01:00
Erik Rigtorp 94f4f13388 Add documentation on exit status codes to man pages
This is step one to start defining useful exit codes for all the commands.
2020-02-12 23:09:26 +01:00
rawtaz 4615bdfd70
Merge pull request #2576 from restic/update-chunker
Update the chunker
2020-02-12 22:41:36 +01:00
rawtaz 299f5971f2
Merge pull request #2560 from brualan/master
Two small improvements to code quality
2020-02-12 22:40:20 +01:00
rawtaz 7a1352ae87
Merge pull request #2570 from MichaelEischer/close-file-on-type-change
Close file if file type has changed after initial stat
2020-02-12 22:39:22 +01:00
Lars Lehtonen 72734d59b5
internal/archiver: fix dropped error 2020-02-12 13:37:37 -08:00
rawtaz 3679669aae
Merge pull request #2574 from alrs/fix-dropped-cmd-err
cmd/restic: fix dropped error
2020-02-12 22:36:45 +01:00
Lars Lehtonen 3ed54e762e
internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Lars Lehtonen fea835b4e2
internal/repository: fix dropped test error 2020-02-12 13:33:54 -08:00
Lars Lehtonen 16b321b140
internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Alexander Neumann b58dfda212 Add entry to changelog 2020-02-12 21:29:26 +01:00
Alexander Neumann b229aa5cf1 Update the chunker 2020-02-12 21:29:26 +01:00
Alexander Neumann 81c0b031f9 Add link to the forum in 'new issue' dialog 2020-02-12 21:25:48 +01:00
Michael Eischer 760863e7f9 archiver: Fix TestRacyFileSwap on windows 2020-02-11 21:09:47 +01:00
Michael Eischer a135699397 Close file if file type has changed after initial stat 2020-02-11 21:09:47 +01:00
Lars Lehtonen 94a4d45dfb
cmd/restic: fix a dropped error 2020-02-11 09:19:03 -08:00
rawtaz 4de384087d
Merge pull request #2575 from alrs/fix-travis
restic/internal/ui: fix gofmt nit that popped up in Go 1.13
2020-02-11 13:49:13 +01:00
Lars Lehtonen 57815d9cd6
restic/internal/ui: fix gofmt nit that popped up in Go 1.13 2020-02-10 11:06:47 -08:00
rawtaz 644673bcf2
Merge pull request #2573 from dhoffend/fix-processed-bytes
fix backup --json total_bytes_processed output
2020-02-10 13:35:55 +01:00
Daniel Hoffend e7cdf2acbb fix backup --json total_bytes_processed output
Closes #2429
2020-02-09 02:02:45 +01:00
rawtaz 0f22f008ed
Merge pull request #2552 from iambryancs/update-to-go1.13
Update Go version to 1.13.6 in Docker build script
2020-02-07 15:24:37 +01:00
rawtaz c184da21d0
Merge pull request #2568 from flofeld/patch-1
corrected option s3.storage-class in documentation
2020-02-06 23:34:36 +01:00
Florian Feldmann ef5efc6a82
corrected option s3.storage-class
Took me some time to figure it out
2020-02-06 22:53:47 +01:00
Alexander Bruyako 688014487b avoid "index out of range" error
in case of len(format) == 0, we will get an
"index out of range" error
Usage of strings.HasSuffix is allowing to avoid that
2020-01-27 19:24:42 +03:00
Alexander Bruyako 38ddfbc4d3 simpler error return 2020-01-27 18:41:46 +03:00
Alexander Bruyako da48b925ff remove unnecessary error return
I was running "golangci-lint" and found this two warnings

internal/checker/checker.go:135:18: (*Checker).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^
internal/repository/repository.go:457:18: (*Repository).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^

It turns out that these functions are used only in "RunWorkers(...)",
which is used only two times in whole project right after this "final"
functions.
And because these "final" functions always return "nil", I've
descided, that it would be better to remove requriments for "final" func
to return error to avoid magick "return nil" at their end.
2020-01-27 18:28:21 +03:00
iambryancs 4bcd56fbae Update Go version to 1.13.6 in Docker build script 2020-01-17 02:13:11 +08:00
Alexander Neumann 6e9778ae0e Add changelog file 2020-01-06 21:46:22 +01:00
Alexander Neumann 63c67be908 Add hint abouth absolute sftp paths for Synology NAS 2020-01-06 21:37:57 +01:00
rawtaz d70a4a9350
Merge pull request #2373 from vrusinov/issue-2372
Ignore username difference in TestMetadataChanged
2020-01-05 21:07:34 +01:00
Alexander Neumann 2cd9c7ef16 sftp: Use MkdirAll provided by the client
Closes #2518
2020-01-01 17:26:38 +01:00
Alexander Neumann 6ec5dc8016 Fix template for new version of calens
The `wrap` function has been renamed to `wrapIndent`
2020-01-01 11:58:21 +01:00
rawtaz fe430a680a
Merge pull request #2524 from cure/patch-1
Fix typos in documentation of FixTime
2019-12-22 16:36:23 +01:00
Ward Vandewege 69a0d0ee90
fix typos 2019-12-21 21:00:28 -05:00
rawtaz 4557881066
Merge pull request #2423 from streambinder/master
sftp: support user@domain parsing as user
2019-12-19 22:39:36 +01:00
streambinder 9b5d069ade doc: 030: sfpt: hint for domain-confined users 2019-12-19 13:15:37 +01:00
streambinder c56cbfe95b changelog: add pull-2423 2019-12-19 13:15:37 +01:00
streambinder 97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
rawtaz 72bd467e23
Merge pull request #2271 from lorenzbausch/feature/find_timestamp
Display snapshot date when using `restic find`
2019-12-17 10:35:59 +01:00
Lorenz Bausch d818b7618b
Display respective snapshot date when using `restic find` 2019-12-17 07:51:12 +01:00
Matt Holt 1c3812a6f6
Merge pull request #2342 from rpsene/master
Add support for ppc64le
2019-12-16 11:50:52 -07:00
Rafael Peria de Sene ec91b80f09 Add support for ppc64le
This commit adds support for cross-compiling Restic for ppc64le
as part of the release process.

Add a new file:   changelog/unreleased/issue-2277
Modifies:   helpers/build-release-binaries/main.go
Modifies:   run_integration_tests.go
2019-12-15 22:52:13 -03:00
rawtaz 18a336c154
Merge pull request #2510 from rawtaz/kastenhq-changelog-2390
Add changelog for #2390.
2019-12-10 23:31:20 +01:00
Eric Hamilton d21a13f1ee Create issue-2390 2019-12-10 23:20:18 +01:00
rawtaz e0eac30ee5
Merge pull request #2509 from restic/improve-check-docs
doc: Improve check documentation
2019-12-10 23:07:23 +01:00
rawtaz fccb579471
doc: Improve check documentation
Make it clearer what the difference is between regular check and check with --read-data.
2019-12-10 21:15:59 +01:00
Alexander Neumann 6c700f95b5 Merge pull request #2358 from filippobottega/master 2019-11-29 20:27:30 +01:00
Filippo Bottega 95d070c147 FAQ: Add hint for antivirus setup
Add a tip to configure antivirus to exclude restic process.
2019-11-29 20:26:20 +01:00
rawtaz da4473aba6
Merge pull request #2391 from kastenhq/refresh-lock-time
Update Lock.Time in lock.Refresh()
2019-11-26 21:44:29 +01:00
rawtaz 6e85a58045
Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed
Fix test metadata changed
2019-11-26 21:42:42 +01:00
rawtaz 476e2e8762
Merge pull request #2354 from HugoReeves/master
Doc Addition: Wasabi repo setup instructions
2019-11-25 13:56:49 +01:00
Alexander Neumann 246bb46e82 Correct CHANEGLOG.md 2019-11-24 15:18:25 +01:00
Alexander Neumann fe0361ffcb Move changelog entry for #2179
For background on how this happened see
https://forum.restic.net/t/psa-attention-anyone-not-using-a-recent-master-ie-0-9-5-version-restic-will-not-detect-changed-files-if-mtime-is-reset-by-the-application-eg-ms-excel-and-xls-xlsx-files-on-macos/1772/8?u=fd0
2019-11-24 15:15:19 +01:00
rawtaz 952469473f
Merge pull request #2486 from rawtaz/changelog-gitkeep
changelog: Add .gitkeep to persist unreleased/ folder
2019-11-24 14:59:39 +01:00
rawtaz 02108f202e
Merge pull request #2487 from rawtaz/2469-fix-diff-stats
diff: Fix wrong bytes reported in diff stats (#2469)
2019-11-23 12:15:28 +01:00
Alexander Neumann b02357f542 Fix message 2019-11-23 10:31:40 +01:00
Alexander Neumann f2aeaef8f1 Improve questions 2019-11-23 10:19:08 +01:00
Alexander Neumann 547702d285 Remove coverage badge 2019-11-23 10:19:08 +01:00
rawtaz 599831f874
Merge pull request #2489 from restic/improve-forget-docs
Improve example for forget --keep-daily
2019-11-22 21:20:35 +01:00
Alexander Neumann 14c90d9e85 Improve example for forget --keep-daily
Following up on https://github.com/restic/restic/pull/2406
2019-11-22 20:44:50 +01:00
Alexander Neumann c41bbb3761 Re-formulate the question in the issue template
Closes #2410
2019-11-22 20:39:47 +01:00
rawtaz 8a54a0f5ac
Merge pull request #2488 from rawtaz/changelog-template
changelog: Fix typo in TEMPLATE
2019-11-22 18:23:02 +01:00
Leo R. Lundgren c0a5530950 changelog: Fix typo in TEMPLATE. 2019-11-22 17:32:56 +01:00
Leo R. Lundgren 77ef92b95c diff: Fix wrong bytes reported in diff stats (#2469) 2019-11-22 17:26:01 +01:00
Leo R. Lundgren 69f75bbf70 changelog: Add .gitkeep to persist the unreleased/ folder. 2019-11-22 17:14:17 +01:00
Alexander Neumann 30519f01ff Set development version for 0.9.6 2019-11-22 16:19:03 +01:00
Alexander Neumann b723ca3de5 Add version for 0.9.6 2019-11-22 16:18:59 +01:00
Alexander Neumann f5084d70d7 Generate CHANGELOG.md for 0.9.6 2019-11-22 16:18:55 +01:00
Alexander Neumann 29b7b17491 Prepare changelog files for 0.9.6 2019-11-22 16:18:33 +01:00
rawtaz e14c4b1737
Merge pull request #2484 from restic/add-s3-region
s3: Allow specifying region
2019-11-22 15:51:17 +01:00
Alexander Neumann 745d79fe5f Merge pull request #2483 from restic/update-minio-go
Update minio-go
2019-11-22 15:31:12 +01:00
Alexander Neumann fb95426f64 Rename environment variable to AWS_DEFAULT_REGION
This seems to be the correct name, at least the AWS cli uses it:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
2019-11-22 15:24:42 +01:00
Alexander Neumann 4cadc89ad3 Add documentation and changelog 2019-11-22 15:16:59 +01:00
Alexander Neumann 409909a7f5 Add option description for Region 2019-11-22 15:09:09 +01:00
mdauphin df500a372d Add AWS_REGION env var to specify s3 region 2019-11-22 15:04:48 +01:00
Alexander Neumann a444731dc0 Update vendored dependencies 2019-11-22 14:57:56 +01:00
Alexander Neumann a6e8af7e0f Update minio-go 2019-11-22 14:50:46 +01:00
Alexander Neumann aa5af8af0e Merge pull request #2478 from restic/update-ci
Update Go version for Appveyor/Travis
2019-11-22 14:36:38 +01:00
Alexander Neumann 4e3353109d Reset changes by 'go get' 2019-11-22 13:17:52 +01:00
Alexander Neumann 02c8d38095 Fix error check for findGoFiles 2019-11-22 12:20:27 +01:00
Alexander Neumann fd6211653c More debug for findImports 2019-11-22 12:16:46 +01:00
rawtaz 3d4f2dd6b4
Merge pull request #2368 from rlue/doc/backblaze
Clarify docs for Backblaze B2 support
2019-11-22 11:10:22 +01:00
Ryan Lue c1ddc0c18b Clarify docs for Backblaze B2 support 2019-11-22 18:07:29 +08:00
rawtaz c95f032a9c
Merge pull request #2310 from vincentbernat/fix/doc-contributing
Update documentation for contributing
2019-11-21 20:30:49 +01:00
Vincent Bernat 3087776135 Update documentation for contributing
- No need to checkout into `GOPATH` anymore
 - `CHANGELOG.md` shouldn't be updated directly
2019-11-21 20:27:48 +01:00
rawtaz b6f01ffbe6
Merge pull request #2394 from bugvillage/doc-backup-passwd
add description env var RESTIC_PASSWORD_COMMAND
2019-11-20 23:26:55 +01:00
rawtaz 41fe9318b1
Merge pull request #2425 from thiell/restic_cache_dir_env
Add support for $RESTIC_CACHE_DIR
2019-11-20 21:18:57 +01:00
Alexander Neumann 8387d18d4d Update Go version used for CI tests 2019-11-20 21:13:56 +01:00
rawtaz 929d2b8df3
Merge pull request #2479 from rawtaz/sing-pulitzer-group
Accept both singular and plural for --group-by (#2330)
2019-11-20 21:03:05 +01:00
Leo R. Lundgren 4f0682d730 Correct name of changelog file for #2321 2019-11-20 21:01:10 +01:00
Alexander Neumann 967d1bbf0c Run 'go mod tidy' 2019-11-20 20:54:43 +01:00
Alexander Neumann 2f80b37b93 Update go-autorest so restic builds with Go 1.13 2019-11-20 20:53:57 +01:00
Leo R. Lundgren 4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
Alexander Neumann 6b1e5d4e18 Update Go version for Appveyor/Travis 2019-11-20 18:32:58 +01:00
rawtaz 26d1f9f4ba
Merge pull request #2307 from gary-kim/fix-2306-password-retries
Allow multiple retries for interactive password input
2019-11-20 18:30:20 +01:00
rawtaz 6a89c0f0ef
Merge pull request #2476 from rawtaz/update-doc
doc: Simplify wording for check's --read-data-subset parameter.
2019-11-19 03:38:37 +01:00
rawtaz b87230b93d
Merge pull request #2456 from jenting/installation-opensuse
Docs: add installation guide for openSUSE
2019-11-19 02:15:19 +01:00
JenTing Hsiao 6f2b8d622a
Add installation guide for openSUSE
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
2019-11-19 09:08:18 +08:00
Leo R. Lundgren 90440212f2 doc: Simplify wording for check's --read-data-subset parameter. 2019-11-19 00:43:41 +01:00
rawtaz 3a5c9aadad
Merge pull request #2444 from chanibal/patch-2
Docs: fixed escaping of argument in documentation
2019-11-18 21:45:00 +01:00
rawtaz a78142c1bb
Merge pull request #2450 from hallamjeff/patch-1
Update 080_examples.rst
2019-11-18 21:43:19 +01:00
rawtaz 07045c7e23
Merge pull request #2442 from lbausch/patch-1
Docs: Highlight option correctly
2019-11-18 21:39:07 +01:00
rawtaz 0a5d42db3f
Merge pull request #2463 from whs-dot-hk/fix-ci-rclone
Update rclone repo
2019-11-18 21:37:02 +01:00
Jeff Hallam 67d99b8cfb
Update 080_examples.rst
added backticks around the -o and --option text.
2019-11-18 16:36:20 -04:00
Alexander Neumann 1a0c0dc277 Remove codecov.io
While it was a nice idea, some tests like the backend integration tests
required credentials which were not available to third-party PRs. This
lead to a lot of uncovered code which confused people. So let's disable
codecov.io for now.
2019-11-18 21:21:40 +01:00
rawtaz e86d9307d0
Merge pull request #2472 from rawtaz/update-backup-doc
doc: Improve exclude/include patterns info
2019-11-18 21:04:18 +01:00
rawtaz 923e681af3
Merge pull request #2471 from 8176135/patch-1
Small fix to the forget --tag command documentation
2019-11-18 20:40:24 +01:00
Leo R. Lundgren 37770b1d82 doc: Improve include patterns info
Clarifies and adds information on how to specify include patterns.
2019-11-18 02:23:01 +01:00
rawtaz 02fea4f76a doc: Improve exclude patterns info
Improves grammar, clarifies and adds information on how to specify exclude patterns.
2019-11-18 01:59:50 +01:00
whs 7cacba0394 Assume WithAtime default to false 2019-11-06 16:38:46 +08:00
whs e6db3596f1 Update rclone repo 2019-11-06 11:59:48 +08:00
8176135 3acc7af310
Small fix to the forget --tag command
As described in #2460
2019-11-03 11:48:56 +13:00
Jeff Hallam 5c4653f427
Update 080_examples.rst
Added an explanation of how to specify the AWS S3 storage classes and what the default value is.
2019-10-21 19:24:50 -03:00
Krzysztof Bociurko f7317a9287
Docs: fixed escaping of argument in documentation
Issue was visible as `foo`` in https://restic.readthedocs.io/en/latest/040_backup.html
2019-10-16 19:03:40 +02:00
lbausch 30db8057e4
Docs: Highlight option correctly 2019-10-13 21:19:03 +02:00
Stephane Thiell 0e897ef7b8 Add support for $RESTIC_CACHE_DIR
Add support for restic-specific $RESTIC_CACHE_DIR environment variable
to override the cache directory like --cache-dir would have.
2019-09-26 15:59:56 -07:00
bugvillage b3e727f40d fixing typo/align around RESTIC_PASSWORD_COMMAND 2019-09-07 15:30:23 +02:00
bugvillage 17feccd998 add description env var RESTIC_PASSWORD_COMMAND
The environment variable RESTIC_PASSWORD_COMMAND works but has
not been documented yet. e.g. it could contain a command that
would fetch the password from a local user keyring

enhances: https://github.com/restic/restic/pull/2094
2019-09-07 14:00:53 +02:00
Eric Hamilton 1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Vladimir Rusinov db20c0b8d0 Ignore username difference in TestMetadataChanged.
In some (rare) cases "fake" UID 51234 may exist in a system running a
test. When this is the case, `cmp.Equal(want, node3)` will fail based on
difference between empty string and an actual username present in a
system.

Fixes github issue #2372
2019-08-13 22:25:00 +01:00
Alexander Neumann 604b18aa74 build: Fix building in ~/go
Before, build.go only unset GO111MODULE and GOPATH, so the Go compiler
did not see either and worked in Module mode. But if the code is checked
out below ~/go (the default GOPATH), it will detect that the source is
within GOPATH and switch to non-Module mode. Now we're setting
GO111MODULE to "on" explicitly.
2019-07-29 09:22:22 +02:00
Alexander Neumann 01c51b3449 Merge pull request #2321 from restic/fix-index-error-handling
Check errors returned by LoadIndex()
2019-07-27 09:37:18 +02:00
Alexander Neumann de8cf5e345 Merge pull request #2324 from jkahrs/fix/change-cleanup-cache-message
updade message that is shown when a cache cleanup is advised
2019-07-27 09:37:10 +02:00
Alexander Neumann cfa2ac69e0 Merge pull request #2322 from brualan/master
minor refactoring
2019-07-27 09:37:06 +02:00
Hugo Reeves 3ca306d69a
Doc Addition: Added Wasabi repo setup 2019-07-26 22:00:48 +12:00
Alexander Neumann 1e9eefa066
Debian stable has restic 0.9.4 (#2333)
Debian stable has restic 0.9.4
2019-07-11 08:17:40 +02:00
Michael M. Chang e9af012229
Debian has updated restic
Debian now has restic 0.9.4+ds-2+b1 in stable (buster)
2019-07-10 11:53:17 -04:00
Jean Kahrs 8066e93f47 updade message that is shown when a cache cleanup is advised 2019-07-02 10:49:49 +02:00
Alexandr Bruyako e19622e4b1 start using fileMode const 2019-07-01 00:26:00 +03:00
Alexandr Bruyako 38ea7ed4f6 remove unused code 2019-07-01 00:24:45 +03:00
Alexandr Bruyako 76d1866444 avoiding unnecessary type conversions 2019-06-30 23:58:00 +03:00
Alexandr Bruyako 8b22fe29cf improved slice copying 2019-06-30 23:56:36 +03:00
Alexandr Bruyako 02014be76c simplified prefix removal, removed unnecessary if-else statements 2019-06-30 23:34:47 +03:00
Alexandr Bruyako 16eeed2ad5 simplified string sorting by using a more suitable function 2019-06-30 23:20:32 +03:00
Alexander Neumann 3f94f63967 Add entry to changelog 2019-06-30 21:38:10 +02:00
Alexander Neumann 88716794e3 Check errors returned by LoadIndex()
Bug was reported in the forum here: https://forum.restic.net/t/check-rebuild-index-prune/1848/13
2019-06-30 21:34:53 +02:00
Fred Akalin 3ca424050f Add upper bound for t in --read-data-subset=n/t (#2304)
* Add upper bound for t in --read-data-subset=n/t

* Add changelog entry
2019-06-29 14:34:53 -04:00
Gary Kim fea2464d4d
Allow multiple retries for interactive password input
Restic used to quit if the repository password was typed incorrectly once.
Restic will now ask the user again for the repository password if typed incorrectly.
The user will now get three tries to input the correct password before restic quits.
2019-06-13 20:11:02 +08:00
Ben 5d272e5c08
Dump : parameter order in sample
Usage:
  restic dump [flags] snapshotID file
2019-05-28 18:20:52 +02:00
Alexander Neumann 5bd5db4294 Merge pull request #2206 from garrmcnu/fs-stdin-directory
fs: Handle absolute pathname for --stdin-filename
2019-05-08 16:05:37 +02:00
Garry McNulty 4429a66b5f backup: Convert relative pathname for --stdin-filename to absolute (#2063) 2019-05-08 15:21:27 +02:00
Garry McNulty 8066195e6e fs: Handle absolute pathname for --stdin-filename
Return valid directory info from Lstat() for parent directories of the
specified filename. Previously only "/" and "." were valid directories.

Also set directory mode as this is checked by archiver.

Closes #2063
2019-05-08 15:21:27 +02:00
Garry McNulty f7f14cf8c9 fs: Add file info base name check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Garry McNulty 5096f3b491 fs: Update directory check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Alexander Neumann cf3fc2a5b1 Merge pull request #2266 from restic/fix-windows-tests
Fix tests on Windows
2019-05-05 21:34:04 +02:00
Alexander Neumann 920d458a4a archiver: Use untyped constants for testing FileInfo 2019-05-05 14:57:38 +02:00
Alexander Neumann b016dc2ff0 archiver/Windows: Skip test new-content-same-filestamp 2019-05-05 14:02:11 +02:00
Alexander Neumann 355db0bc29 windows: Use LastWriteTime for ctime and mtime
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
2019-05-05 14:02:11 +02:00
Alexander Neumann 6e2fe73189 archiver: Move tests back into the same file
Move all Archiver tests back into `archiver_test.go` and add some tiny
helpers to mock what `lstat` returns (for Windows and Unix separately).
2019-05-05 14:02:11 +02:00
Alexander Neumann 303a5dab6a archiver: Clarify value in test struct
Since I could not remember what the value for `Check` means this commit
renames it to `SameFile`: when set to true, the test should make sure
that `FileChanged` should return false (=file is unmodified).
2019-05-05 12:57:00 +02:00
Alexander Neumann 7dcd2968b6 Merge pull request #2261 from lorenzbausch/patch-1
Fix typo
2019-05-04 10:23:49 +02:00
Lorenz Bausch 298f490195
Fix typo 2019-05-02 11:51:35 +02:00
Alexander Neumann 37cb82b28b Merge pull request #2257 from gliptak/windows1
Don't run TestMetadataChanged test on Windows
2019-04-28 08:39:00 +02:00
Gábor Lipták bce6438d22
Don't run TestMetadataChanged test on Windows
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2019-04-27 21:23:47 -04:00
Alexander Neumann 919dd2ac84 Merge pull request #2252 from restic/fix-2249
Read fresh metadata for unmodified files
2019-04-25 09:15:50 +02:00
Alexander Neumann 870bc5108e Merge pull request #2253 from restic/fix-2174
Make sure timestamps are valid
2019-04-25 09:14:43 +02:00
Alexander Neumann 418296c5c9 Add hint for "closes" wording to PR template 2019-04-25 09:00:15 +02:00
Alexander Neumann a6481b3707 Merge pull request #2212 from cbane/check-ctime
Examine file ctime when checking if files have changed.
2019-04-25 08:58:27 +02:00
Courtney Bane 00b527fb09 Update changelog text, and add pull request link. 2019-04-24 20:54:15 -05:00
Courtney Bane 0ebfc55ee3 Use existing setTimestamp function for ctime test and improve error checking. 2019-04-24 20:53:08 -05:00
Courtney Bane 35b7607802 Don't check ctime when ignoring inode. 2019-04-24 20:53:08 -05:00
Alexander Neumann fad9f65c65 Merge pull request #2251 from restic/update-deps
Update dependencies
2019-04-24 16:42:57 +02:00
Alexander Neumann 939f3e972c node: Make sure year of all timestamps is valid
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
2019-04-24 16:39:36 +02:00
Alexander Neumann ca8c3b4fd5 Update dependencies 2019-04-24 15:17:48 +02:00
Alexander Neumann 4f45b14f25 Add changelog file 2019-04-24 15:17:25 +02:00
Alexander Neumann 389067fb8b Only use list of blobs for old node
Closes #2249
2019-04-24 15:07:26 +02:00
Alexander Neumann 4b0ca9ddab Add test for #2249 2019-04-24 15:07:23 +02:00
Courtney Bane b8c2544dcb Examine file ctime when checking if files have changed. 2019-04-23 21:54:35 -05:00
Alexander Neumann c7762453cf Set development version for 0.9.5 2019-04-23 13:19:43 +02:00
2347 changed files with 72563 additions and 539361 deletions

2
.gitattributes vendored Normal file
View File

@ -0,0 +1,2 @@
# Workaround for https://github.com/golang/go/issues/52268.
**/testdata/fuzz/*/* eol=lf

View File

@ -1,27 +0,0 @@
<!--
Welcome! If you have a question or are unsure if you should open an issue,
please use the forum instead!
https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
Describe the issue
------------------

View File

@ -32,23 +32,30 @@ Output of `restic version`
--------------------------
How did you run restic exactly?
-------------------------------
What backend/service did you use to store the repository?
---------------------------------------------------------
Problem description / Steps to reproduce
----------------------------------------
<!--
This section should include at least:
* A description of the problem you are having with restic.
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
* The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
Expected behavior
-----------------
@ -65,26 +72,16 @@ In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
Do you have any idea what may have caused this?
-----------------------------------------------
<!--
Did something noteworthy happen on your system, Internet connection, backend services, etc?
-->
Do you have an idea how to solve the issue?
-------------------------------------------
Did restic help you or made you happy in any way?
-------------------------------------------------
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!

View File

@ -39,16 +39,16 @@ Please describe the feature you'd like us to add here.
-->
What are you trying to do?
--------------------------
What are you trying to do? What problem would this solve?
---------------------------------------------------------
<!--
This section should contain a brief description what you're trying to do, which
would be possible after implementing the new feature.
-->
Did restic help you or made you happy in any way?
-------------------------------------------------
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!

4
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@ -0,0 +1,4 @@
contact_links:
- name: restic forum
url: https://forum.restic.net
about: Please ask questions about using restic here, do not open an issue for questions.

View File

@ -1,36 +1,40 @@
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
changes.
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
-->
What is the purpose of this change? What does it change?
--------------------------------------------------------
What does this PR change? What problem does it solve?
-----------------------------------------------------
<!--
Describe the changes here, as detailed as needed.
Describe the changes and their purpose here, as detailed as needed.
-->
Was the change discussed in an issue or in the forum before?
------------------------------------------------------------
Was the change previously discussed in an issue or on the forum?
----------------------------------------------------------------
<!--
Link issues and relevant forum posts here.
If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
is closed automatically when this PR is merged.
-->
Checklist
---------
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have added tests for all changes in this PR
- [ ] I have added documentation for the changes (in the manual)
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
- [ ] I have run `gofmt` on the code in all commits
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
- [ ] I'm done, this Pull Request is ready for review
<!--
You do not need to check all the boxes below all at once. Feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box. Enable a checkbox by replacing [ ] with [x].
-->
- [ ] I have read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- [ ] I have [enabled maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I have run `gofmt` on the code in all commits.
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
- [ ] I'm done! This pull request is ready for review.

13
.github/dependabot.yml vendored Normal file
View File

@ -0,0 +1,13 @@
version: 2
updates:
# Dependencies listed in go.mod
- package-ecosystem: "gomod"
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
# Dependencies listed in .github/workflows/*.yml
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"

66
.github/workflows/docker.yml vendored Normal file
View File

@ -0,0 +1,66 @@
name: Create and publish a Docker image
on:
push:
tags:
- 'v*'
branches:
- 'master'
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
if: github.repository == 'restic/restic'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Log in to the Container registry
uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- name: Set up QEMU
uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@2b51285047da1547ffb1b2203d8be4c0af6b1f20
- name: Ensure consistent binaries
run: |
echo "removing git directory for consistency with release binaries"
rm -rf .git
# remove VCS information from release builds, keep VCS for nightly builds on master
if: github.ref != 'refs/heads/master'
- name: Build and push Docker image
uses: docker/build-push-action@2cdde995de11925a030ce8070c3d77a52ffcf1c0
with:
push: true
context: .
file: docker/Dockerfile.release
platforms: linux/386,linux/amd64,linux/arm,linux/arm64
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

335
.github/workflows/tests.yml vendored Normal file
View File

@ -0,0 +1,335 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
merge_group:
permissions:
contents: read
env:
latest_go: "1.22.x"
GO111MODULE: on
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.22.x
os: windows-latest
- job_name: macOS
go: 1.22.x
os: macOS-latest
test_fuse: false
- job_name: Linux
go: 1.22.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
- job_name: Linux (race)
go: 1.22.x
os: ubuntu-latest
test_fuse: true
test_opts: "-race"
- job_name: Linux
go: 1.21.x
os: ubuntu-latest
test_fuse: true
- job_name: Linux
go: 1.20.x
os: ubuntu-latest
test_fuse: true
- job_name: Linux
go: 1.19.x
os: ubuntu-latest
test_fuse: true
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v4
- name: Build with build.go
run: |
go run build.go
- name: Minimal test
run: |
./restic init
./restic backup .
env:
RESTIC_REPOSITORY: ../testrepo
RESTIC_PASSWORD: password
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ${{matrix.test_opts}} ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
# Skip for Dependabot pull requests as these are run without secrets
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#responding-to-events
if: ((github.repository == 'restic/restic' && github.event_name == 'push') || github.event.pull_request.head.repo.full_name == github.repository) && (github.actor != 'dependabot[bot]') && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go install github.com/restic/calens@latest
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
matrix:
# run cross-compile in three batches parallel so the overall tests run faster
subset:
- "0/3"
- "1/3"
- "2/3"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for subset ${{ matrix.subset }}
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v4
- name: Cross-compile for subset ${{ matrix.subset }}
run: |
mkdir build-output build-output-debug
go run ./helpers/build-release-binaries/main.go -o build-output -s . --platform-subset ${{ matrix.subset }}
go run ./helpers/build-release-binaries/main.go -o build-output-debug -s . --platform-subset ${{ matrix.subset }} --tags debug
lint:
name: lint
runs-on: ubuntu-latest
permissions:
contents: read
# allow annotating code in the PR
checks: write
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v4
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.57.1
args: --verbose --timeout 5m
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum
analyze:
name: Analyze results
needs: [test, cross_compile, lint]
if: always()
permissions: # no need to access code
contents: none
runs-on: ubuntu-latest
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe
with:
jobs: ${{ toJSON(needs) }}
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
# list of Docker images to use as base name for tags
images: |
restic/restic
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push
id: docker_build
uses: docker/build-push-action@v5
with:
push: false
context: .
file: docker/Dockerfile
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

3
.gitignore vendored
View File

@ -1,2 +1,5 @@
/.idea
/restic
/restic.exe
/.vagrant
/.vscode

69
.golangci.yml Normal file
View File

@ -0,0 +1,69 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# show how code can be simplified
- gosimple
# make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- revive
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# parse and typecheck code
- typecheck
# ensure that http response bodies are closed
- bodyclose
- importas
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# revive: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) .* should have comment or be unexported
# revive: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase
# revive: lots of packages don't have such a comment
- "package-comments: should have a package comment"
# staticcheck: there's no easy way to replace these packages
- "SA1019: \"golang.org/x/crypto/poly1305\" is deprecated"
- "SA1019: \"golang.org/x/crypto/openpgp\" is deprecated"
exclude-rules:
# revive: ignore unused parameters in tests
- path: (_test\.go|testing\.go|backend/.*/tests\.go)
text: "unused-parameter:"
linters-settings:
importas:
alias:
- pkg: github.com/restic/restic/internal/test
alias: rtest

View File

@ -1,2 +0,0 @@
go:
enabled: true

22
.readthedocs.yaml Normal file
View File

@ -0,0 +1,22 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
# Build HTMLZip
formats:
- htmlzip
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: doc/conf.py
# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: doc/requirements.txt

View File

@ -1,61 +0,0 @@
language: go
sudo: false
matrix:
include:
- os: linux
go: "1.10.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.11.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.12.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.12.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
- master
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- go env
script:
- go run run_integration_tests.go
after_success:
- test -r all.cov && bash <(curl -s https://codecov.io/bash) -f all.cov

File diff suppressed because it is too large Load Diff

View File

@ -6,17 +6,18 @@ Ways to Help Out
Thank you for your contribution! Please **open an issue first** (or add a
comment to an existing issue) if you plan to work on any code or add a new
feature. This way, duplicate work is prevented and we can discuss your ideas
and design first.
and design first. Small bugfixes are an exception to this rule, just open a
pull request in this case.
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great
help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the `#restic` channel on `irc.freenode.net`.
as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.libera.chat](https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:6697/#restic).
If you want to find an area that currently needs improving have a look at the
open issues listed at the
@ -25,7 +26,10 @@ for discussing enhancement to the restic tools.
If you are unsure what to do, please have a look at the issues, especially
those tagged
[minor complexity](https://github.com/restic/restic/labels/minor%20complexity).
[minor complexity](https://github.com/restic/restic/labels/help%3A%20minor%20complexity)
or [good first issue](https://github.com/restic/restic/labels/help%3A%20good%20first%20issue).
If you are already a bit experienced with the restic internals, take a look
at the issues tagged as [help wanted](https://github.com/restic/restic/labels/help%3A%20wanted).
Reporting Bugs
@ -45,34 +49,38 @@ environment was used and so on. Please tell us at least the following things:
Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run -mod=vendor build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
In addition, you can instruct restic to create a debug log by setting the
environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
For Go < 1.11, you need to remove the `-mod=vendor` option from the build
command.
Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
If restic gets stuck, please also include a stacktrace in the description.
On non-Windows systems, you can send a SIGQUIT signal to restic or press
`Ctrl-\` to achieve the same result. This causes restic to print a stacktrace
and then exit immediately. This will not damage your repository, however,
it might be necessary to manually clean up stale lock files using
`restic unlock`.
On Windows, please set the environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT`
to `true` and press `Ctrl-C` to create a stacktrace.
If you think restic uses too much memory or a too large cache directory, then
please include the output of `restic stats --mode debug`.
Development Environment
=======================
The repository contains several sets of directories with code: `cmd/` and
`internal/` contain the code written for restic, whereas `vendor/` contains
copies of libraries restic depends on. The libraries are managed with the
command `go mod vendor`.
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
Go >= 1.11
----------
For Go version 1.11 or later, you should clone the repo (without having
`$GOPATH` set) and `cd` into the directory:
Make sure you have the minimum required Go version installed. Clone the repo
(without having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
@ -82,39 +90,41 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.9.2-dev (compiled manually) compiled with go1.11 on linux/amd64
restic 0.14.0-dev (compiled manually) compiled with go1.19 on linux/amd64
To create a debug build use:
$ go build -tags debug ./cmd/restic
You can run all tests with the following command:
$ go test ./...
Go < 1.11
---------
In order to compile restic with Go before 1.11, it needs to be checked out at
the right path within a `GOPATH`. The concept of a `GOPATH` is explained in
["How to write Go code"](https://golang.org/doc/code.html).
Performance and Memory Usage Issues
===================================
If you do not have a directory with Go code yet, executing the following
instructions in your shell will create one for you and check out the restic
repo:
Debug builds of restic support the `--block-profile`, `--cpu-profile`,
`--mem-profile`, and `--trace-profile` options which collect performance data
that later on can be analyzed using the go tools:
$ export GOPATH="$HOME/go"
$ mkdir -p "$GOPATH/src/github.com/restic"
$ cd "$GOPATH/src/github.com/restic"
$ git clone https://github.com/restic/restic
$ cd restic
$ restic --cpu-profile . [...]
$ go tool pprof -http localhost:12345 cpu.pprof
You can then build restic as follows:
To analyze a trace profile use `go tool trace -http=localhost:12345 trace.out`.
$ go build ./cmd/restic
$ ./restic version
restic compiled manually
compiled with go1.8.3 on linux/amd64
As the memory usage of restic changes over time, it may be useful to capture a
snapshot of the current heap. This is possible using then `--listen-profile`
option. Then while restic runs you can query and afterwards analyze the heap statistics.
The following commands can be used to run all the tests:
$ restic --listen-profile localhost:12345 [...]
$ curl http://localhost:12345/debug/pprof/heap -o heap.pprof
$ go tool pprof -http localhost:12345 heap.pprof
Further useful tools are setting the environment variable `GODEBUG=gctrace=1`,
which provides information about garbage collector runs. For a graphical variant
combine this with gcvis.
$ go test ./...
Providing Patches
=================
@ -128,21 +138,19 @@ down to the following steps:
GitHub. For a new feature, please add an issue before starting to work on
it, so that duplicate work is prevented.
1. First we would kindly ask you to fork our project on GitHub if you haven't
done so already.
1. Next, fork our project on GitHub if you haven't done so already.
2. Clone the repository locally and create a new branch. If you are working on
the code itself, please set up the development environment as described in
the previous section. Especially take care to place your forked repository
at the correct path (`src/github.com/restic/restic`) within your `GOPATH`.
2. Clone your fork of the repository locally and **create a new branch** for
your changes. If you are working on the code itself, please set up the
development environment as described in the previous section.
3. Then commit your changes as fine grained as possible, as smaller patches,
that handle one and only one issue are easier to discuss and merge.
3. Commit your changes to the new branch as fine grained as possible, as
smaller patches, for individual changes, are easier to discuss and merge.
4. Push the new branch with your changes to your fork of the repository.
5. Create a pull request by visiting the GitHub website, it will guide you
through the process.
through the process. Please [allow edits from maintainers](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
6. You will receive comments on your code and the feature or bug that they
address. Maybe you need to rework some minor things, in this case push new
@ -150,17 +158,22 @@ down to the following steps:
existing commit, use common sense to decide which is better), they will be
automatically added to the pull request.
7. If your pull request changes anything that users should be aware of (a
bugfix, a new feature, ...) please add an entry to the file
['CHANGELOG.md'](CHANGELOG.md). It will be used in the announcement of the
next stable release. While writing, ask yourself: If I were the user, what
would I need to be aware of with this change.
7. If your pull request changes anything that users should be aware of
(a bugfix, a new feature, ...) please add an entry as a new file in
`changelog/unreleased` including the issue number in the filename (e.g.
`issue-8756`). Use the template in `changelog/TEMPLATE` for the content.
It will be used in the announcement of the next stable release. While
writing, ask yourself: If I were the user, what would I need to be aware
of with this change?
8. Once your code looks good and passes all the tests, we'll merge it. Thanks
8. Do not edit the man pages under `doc/man` or `doc/manual_rest.rst` -
these are autogenerated before new releases.
9. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and
open up a pull request for each.
open up a pull request for each, as this simplifies discussion and merging.
The restic project uses the `gofmt` tool for Go source indentation, so please
run
@ -173,6 +186,14 @@ Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix

View File

@ -3,7 +3,7 @@
all: restic
restic:
go run -mod=vendor build.go || go run build.go
go run build.go
clean:
rm -f restic

109
README.md Normal file
View File

@ -0,0 +1,109 @@
[![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://github.com/restic/restic/workflows/test/badge.svg)](https://github.com/restic/restic/actions?query=workflow%3Atest)
[![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic)
# Introduction
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the [documentation](https://restic.readthedocs.io/en/latest).
You can ask questions in our [Discourse forum](https://forum.restic.net).
## Quick start
Once you've [installed](https://restic.readthedocs.io/en/latest/020_installation.html) restic, start
off with creating a repository for your backups:
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use `restic restore` to restore files or use `restic
mount` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the [online documentation](https://restic.readthedocs.io/en/latest/).
# Backends
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local)
- [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](https://restic.readthedocs.io/en/latest/100_references.html#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [Amazon S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift)
- [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2)
- [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage)
- [Google Cloud Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage)
- And many other services via the [rclone](https://rclone.org) [Backend](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone)
# Design Principles
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy**: Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
# Reproducible Builds
The binaries released with each restic version starting at 0.6.1 are
[reproducible](https://reproducible-builds.org/), which means that you can
reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
[builder repository](https://github.com/restic/builder).
## News
You can follow the restic project on Mastodon [@resticbackup](https://fosstodon.org/@restic) or subscribe to
the [project blog](https://restic.net/blog/).
## License
Restic is licensed under [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause). You can find the
complete text in [`LICENSE`](LICENSE).
## Sponsorship
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by [AppsCode](https://appscode.com)!
[![Sponsored by AppsCode](https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png)](https://appscode.com)

View File

@ -1,137 +0,0 @@
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
Introduction
------------
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.
You can ask questions in our `Discourse forum <https://forum.restic.net>`__.
Quick start
-----------
Once you've `installed
<https://restic.readthedocs.io/en/latest/020_installation.html>`__ restic, start
off with creating a repository for your backups:
.. code-block:: console
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
.. code-block:: console
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use ``restic restore`` to restore files or use ``restic
mount`` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the `online documentation <https://restic.readthedocs.io/en/latest/>`__.
Backends
--------
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- `Local directory <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local>`__
- `sftp server (via SSH) <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp>`__
- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/100_references.rst#rest-backend>`__ `rest-server <https://github.com/restic/rest-server>`__)
- `AWS S3 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3>`__ (either from Amazon or using the `Minio <https://minio.io>`__ server)
- `OpenStack Swift <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift>`__
- `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
- `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
- `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
Design Principles
-----------------
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
Reproducible Builds
-------------------
The binaries released with each restic version starting at 0.6.1 are
`reproducible <https://reproducible-builds.org/>`__, which means that you can
easily reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
`builder repository <https://github.com/restic/builder>`__.
News
----
You can follow the restic project on Twitter `@resticbackup <https://twitter.com/resticbackup>`__ or by subscribing to
the `development blog <https://restic.net/blog/>`__.
License
-------
Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
complete text in ``LICENSE``.
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by `AppsCode <https://appscode.com>`__!
|AppsCode|
.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
:target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
:target: https://travis-ci.com/restic/restic
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
:target: https://ci.appveyor.com/project/fd0/restic/branch/master
.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
:target: https://goreportcard.com/report/github.com/restic/restic
.. |Say Thanks| image:: https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg
:target: https://saythanks.io/to/restic
.. |TestCoverage| image:: https://codecov.io/gh/restic/restic/branch/master/graph/badge.svg
:target: https://codecov.io/gh/restic/restic
.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
:target: https://appscode.com
.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
:target: https://houndci.com

View File

@ -1 +1 @@
0.9.5
0.16.4

View File

@ -1,32 +0,0 @@
clone_folder: c:\restic
environment:
GOPATH: c:\gopath
branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.12.1.windows-amd64.msi
- msiexec /i go1.12.1.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run -mod=vendor run_integration_tests.go

277
build.go
View File

@ -3,22 +3,15 @@
// This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH.
//
// For Go < 1.11, it'll create a new GOPATH in a temporary directory, then run
// `go build` on the package configured as Main in the Config struct.
//
// For Go >= 1.11 if the file go.mod is present, it'll use Go modules and not
// setup a GOPATH. It builds the package configured as Main in the Config
// struct with `go build -mod=vendor` to use the vendored dependencies.
// The variable GOPROXY is set to `off` so that no network calls are made. All
// files are copied to a temporary directory before `go build` is called within
// that directory.
// This program checks for a minimum Go version. It will use Go modules for
// compilation. It builds the package configured as Main in the Config struct.
// BSD 2-Clause License
//
// Copyright (c) 2016-2018, Alexander Neumann <alexander@bumpern.de>
// All rights reserved.
//
// This file has been copied from the repository at:
// This file has been derived from the repository at:
// https://github.com/fd0/build-go
//
// Redistribution and use in source and binary forms, with or without
@ -42,6 +35,7 @@
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//go:build ignore_build_go
// +build ignore_build_go
package main
@ -49,7 +43,6 @@ package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
@ -65,7 +58,7 @@ var config = Config{
Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 10, Patch: 0}, // minimum Go version supported
MinVersion: GoVersion{Major: 1, Minor: 18, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@ -79,125 +72,13 @@ type Config struct {
}
var (
verbose bool
keepGopath bool
runTests bool
enableCGO bool
enablePIE bool
goVersion = ParseGoVersion(runtime.Version())
verbose bool
runTests bool
enableCGO bool
enablePIE bool
goVersion = ParseGoVersion(runtime.Version())
)
// copy all Go files in src to dst, creating directories on the fly, so calling
//
// copy("/tmp/gopath/src/github.com/restic/restic", "/home/u/restic")
//
// with "/home/u/restic" containing the file "foo.go" yields the following tree
// at "/tmp/gopath":
//
// /tmp/gopath
// └── src
// └── github.com
// └── restic
// └── restic
// └── foo.go
func copy(dst, src string) error {
verbosePrintf("copy contents of %v to %v\n", src, dst)
return filepath.Walk(src, func(name string, fi os.FileInfo, err error) error {
if name == src {
return err
}
if name == ".git" {
return filepath.SkipDir
}
if err != nil {
return err
}
if fi.IsDir() {
return nil
}
intermediatePath, err := filepath.Rel(src, name)
if err != nil {
return err
}
fileSrc := filepath.Join(src, intermediatePath)
fileDst := filepath.Join(dst, intermediatePath)
return copyFile(fileDst, fileSrc)
})
}
func directoryExists(dirname string) bool {
stat, err := os.Stat(dirname)
if err != nil && os.IsNotExist(err) {
return false
}
return stat.IsDir()
}
func fileExists(filename string) bool {
stat, err := os.Stat(filename)
if err != nil && os.IsNotExist(err) {
return false
}
return stat.Mode().IsRegular()
}
// copyFile creates dst from src, preserving file attributes and timestamps.
func copyFile(dst, src string) error {
fi, err := os.Stat(src)
if err != nil {
return err
}
fsrc, err := os.Open(src)
if err != nil {
return err
}
if err = os.MkdirAll(filepath.Dir(dst), 0755); err != nil {
fmt.Printf("MkdirAll(%v)\n", filepath.Dir(dst))
return err
}
fdst, err := os.Create(dst)
if err != nil {
_ = fsrc.Close()
return err
}
_, err = io.Copy(fdst, fsrc)
if err != nil {
_ = fsrc.Close()
_ = fdst.Close()
return err
}
err = fdst.Close()
if err != nil {
_ = fsrc.Close()
return err
}
err = fsrc.Close()
if err != nil {
return err
}
err = os.Chmod(dst, fi.Mode())
if err != nil {
return err
}
return os.Chtimes(dst, fi.ModTime(), fi.ModTime())
}
// die prints the message with fmt.Fprintf() to stderr and exits with an error
// code.
func die(message string, args ...interface{}) {
@ -211,7 +92,6 @@ func showUsage(output io.Writer) {
fmt.Fprintf(output, "OPTIONS:\n")
fmt.Fprintf(output, " -v --verbose output more messages\n")
fmt.Fprintf(output, " -t --tags specify additional build tags\n")
fmt.Fprintf(output, " -k --keep-tempdir do not remove the temporary directory after build\n")
fmt.Fprintf(output, " -T --test run tests\n")
fmt.Fprintf(output, " -o --output set output file name\n")
fmt.Fprintf(output, " --enable-cgo use CGO to link against libc\n")
@ -219,7 +99,6 @@ func showUsage(output io.Writer) {
fmt.Fprintf(output, " --goos value set GOOS for cross-compilation\n")
fmt.Fprintf(output, " --goarch value set GOARCH for cross-compilation\n")
fmt.Fprintf(output, " --goarm value set GOARM for cross-compilation\n")
fmt.Fprintf(output, " --tempdir dir use a specific directory for compilation\n")
}
func verbosePrintf(message string, args ...interface{}) {
@ -230,49 +109,30 @@ func verbosePrintf(message string, args ...interface{}) {
fmt.Printf("build: "+message, args...)
}
// cleanEnv returns a clean environment with GOPATH, GOBIN and GO111MODULE
// removed (if present).
func cleanEnv() (env []string) {
removeKeys := map[string]struct{}{
"GOPATH": struct{}{},
"GOBIN": struct{}{},
"GO111MODULE": struct{}{},
}
for _, v := range os.Environ() {
data := strings.SplitN(v, "=", 2)
name := data[0]
if _, ok := removeKeys[name]; ok {
// printEnv prints Go-relevant environment variables in a nice way using verbosePrintf.
func printEnv(env []string) {
verbosePrintf("environment (GO*):\n")
for _, v := range env {
// ignore environment variables which do not start with GO*.
if !strings.HasPrefix(v, "GO") {
continue
}
env = append(env, v)
verbosePrintf(" %s\n", v)
}
return env
}
// build runs "go build args..." with GOPATH set to gopath.
func build(cwd string, env map[string]string, args ...string) error {
a := []string{"build"}
// -trimpath removes all absolute paths from the binary.
a := []string{"build", "-trimpath"}
if goVersion.AtLeast(GoVersion{1, 10, 0}) {
verbosePrintf("Go version is at least 1.10, using new syntax for -gcflags\n")
// use new prefix
a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
} else {
a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", cwd))
}
if enablePIE {
a = append(a, "-buildmode=pie")
}
a = append(a, args...)
cmd := exec.Command("go", a...)
cmd.Env = append(cleanEnv(), "GOPROXY=off")
cmd.Env = os.Environ()
for k, v := range env {
cmd.Env = append(cmd.Env, k+"="+v)
}
@ -280,6 +140,8 @@ func build(cwd string, env map[string]string, args ...string) error {
cmd.Env = append(cmd.Env, "CGO_ENABLED=0")
}
printEnv(cmd.Env)
cmd.Dir = cwd
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
@ -294,7 +156,7 @@ func build(cwd string, env map[string]string, args ...string) error {
func test(cwd string, env map[string]string, args ...string) error {
args = append([]string{"test", "-count", "1"}, args...)
cmd := exec.Command("go", args...)
cmd.Env = append(cleanEnv(), "GOPROXY=off")
cmd.Env = os.Environ()
for k, v := range env {
cmd.Env = append(cmd.Env, k+"="+v)
}
@ -305,6 +167,8 @@ func test(cwd string, env map[string]string, args ...string) error {
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
printEnv(cmd.Env)
verbosePrintf("chdir %q\n", cwd)
verbosePrintf("go %q\n", args)
@ -314,7 +178,7 @@ func test(cwd string, env map[string]string, args ...string) error {
// getVersion returns the version string from the file VERSION in the current
// directory.
func getVersionFromFile() string {
buf, err := ioutil.ReadFile("VERSION")
buf, err := os.ReadFile("VERSION")
if err != nil {
verbosePrintf("error reading file VERSION: %v\n", err)
return ""
@ -455,7 +319,7 @@ func (v GoVersion) String() string {
func main() {
if !goVersion.AtLeast(config.MinVersion) {
fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion)
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s\n", goVersion, config.MinVersion)
os.Exit(1)
}
@ -464,15 +328,13 @@ func main() {
skipNext := false
params := os.Args[1:]
goEnv := map[string]string{}
buildEnv := map[string]string{
"GOOS": runtime.GOOS,
"GOARCH": runtime.GOARCH,
"GOARM": "",
env := map[string]string{
"GO111MODULE": "on", // make sure we build in Module mode
"GOOS": runtime.GOOS,
"GOARCH": runtime.GOARCH,
"GOARM": "",
}
tempdir := ""
var outputFilename string
for i, arg := range params {
@ -484,8 +346,6 @@ func main() {
switch arg {
case "-v", "--verbose":
verbose = true
case "-k", "--keep-gopath":
keepGopath = true
case "-t", "-tags", "--tags":
if i+1 >= len(params) {
die("-t given but no tag specified")
@ -495,9 +355,6 @@ func main() {
case "-o", "--output":
skipNext = true
outputFilename = params[i+1]
case "--tempdir":
skipNext = true
tempdir = params[i+1]
case "-T", "--test":
runTests = true
case "--enable-cgo":
@ -506,13 +363,13 @@ func main() {
enablePIE = true
case "--goos":
skipNext = true
buildEnv["GOOS"] = params[i+1]
env["GOOS"] = params[i+1]
case "--goarch":
skipNext = true
buildEnv["GOARCH"] = params[i+1]
env["GOARCH"] = params[i+1]
case "--goarm":
skipNext = true
buildEnv["GOARM"] = params[i+1]
env["GOARM"] = params[i+1]
case "-h":
showUsage(os.Stdout)
return
@ -523,10 +380,20 @@ func main() {
}
}
solarisMinVersion := GoVersion{Major: 1, Minor: 20, Patch: 0}
if env["GOARCH"] == "solaris" && !goVersion.AtLeast(solarisMinVersion) {
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s for Solaris\n", goVersion, solarisMinVersion)
os.Exit(1)
}
verbosePrintf("detected Go version %v\n", goVersion)
preserveSymbols := false
for i := range buildTags {
buildTags[i] = strings.TrimSpace(buildTags[i])
if buildTags[i] == "debug" || buildTags[i] == "profile" {
preserveSymbols = true
}
}
verbosePrintf("build tags: %s\n", buildTags)
@ -538,7 +405,7 @@ func main() {
if outputFilename == "" {
outputFilename = config.Name
if buildEnv["GOOS"] == "windows" {
if env["GOOS"] == "windows" {
outputFilename += ".exe"
}
}
@ -553,7 +420,11 @@ func main() {
if version != "" {
constants["main.version"] = version
}
ldflags := "-s -w " + constants.LDFlags()
ldflags := constants.LDFlags()
if !preserveSymbols {
// Strip debug symbols.
ldflags = "-s -w " + ldflags
}
verbosePrintf("ldflags: %s\n", ldflags)
var (
@ -567,54 +438,18 @@ func main() {
}
buildTarget := filepath.FromSlash(mainPackage)
buildCWD := ""
if goVersion.AtLeast(GoVersion{1, 11, 0}) && fileExists("go.mod") {
verbosePrintf("Go >= 1.11 and 'go.mod' found, building with modules\n")
buildCWD = root
buildArgs = append(buildArgs, "-mod=vendor")
testArgs = append(testArgs, "-mod=vendor")
} else {
if tempdir == "" {
tempdir, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
if err != nil {
die("TempDir(): %v\n", err)
}
}
verbosePrintf("Go < 1.11 or 'go.mod' not found, create GOPATH at %v\n", tempdir)
targetdir := filepath.Join(tempdir, "src", filepath.FromSlash(config.Namespace))
if err = copy(targetdir, root); err != nil {
die("copying files from %v to %v/src failed: %v\n", root, tempdir, err)
}
defer func() {
if !keepGopath {
verbosePrintf("remove %v\n", tempdir)
if err = os.RemoveAll(tempdir); err != nil {
die("remove GOPATH at %s failed: %v\n", tempdir, err)
}
} else {
verbosePrintf("leaving temporary GOPATH at %v\n", tempdir)
}
}()
buildCWD = targetdir
goEnv["GOPATH"] = tempdir
buildEnv["GOPATH"] = tempdir
buildCWD, err := os.Getwd()
if err != nil {
die("unable to determine current working directory: %v\n", err)
}
verbosePrintf("environment:\n go: %v\n build: %v\n", goEnv, buildEnv)
buildArgs = append(buildArgs,
"-tags", strings.Join(buildTags, " "),
"-ldflags", ldflags,
"-o", output, buildTarget,
)
err = build(buildCWD, buildEnv, buildArgs...)
err = build(buildCWD, env, buildArgs...)
if err != nil {
die("build failed: %v\n", err)
}
@ -624,7 +459,7 @@ func main() {
testArgs = append(testArgs, config.Tests...)
err = test(buildCWD, goEnv, testArgs...)
err = test(buildCWD, env, testArgs...)
if err != nil {
die("running tests failed: %v\n", err)
}

View File

@ -0,0 +1,9 @@
Enhancement: Support specifying multiple host flags for various commands
Previously commands didn't take more than one `--host` or `-H` argument into account, which could be limiting with e.g.
the `forget` command.
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`, `stats` and `tag` commands will now take into account
multiple `--host` and `-H` flags.
https://github.com/restic/restic/issues/1570

View File

@ -0,0 +1,8 @@
Enhancement: Optimize `restic mount`
We've optimized the FUSE implementation used within restic.
`restic mount` is now more responsive and uses less memory.
https://github.com/restic/restic/issues/1680
https://github.com/restic/restic/pull/2587
https://github.com/restic/restic/pull/2787

View File

@ -0,0 +1,6 @@
Bugfix: Report correct number of directories processed by backup
The directory statistics calculation was fixed to report the actual number
of processed directories instead of always zero.
https://github.com/restic/restic/issues/1863

View File

@ -0,0 +1,5 @@
Enhancement: Display snapshot date when using `restic find`
Added the respective snapshot date to the output of `restic find`.
https://github.com/restic/restic/issues/2072

View File

@ -0,0 +1,9 @@
Enhancement: Allow specifying user and host when creating keys
When adding a new key to the repository, the username and hostname for the new
key can be specified on the command line. This allows overriding the defaults,
for example if you would prefer to use the FQDN to identify the host or if you
want to add keys for several different hosts without having to run the key add
command on those hosts.
https://github.com/restic/restic/issues/2175

View File

@ -0,0 +1,9 @@
Bugfix: Fix tar issues when dumping `/`
We've fixed an issue with dumping either `/` or files on the first sublevel
e.g. `/foo` to tar. This also fixes tar dumping issues on Windows where this
issue could also happen.
https://github.com/restic/restic/issues/2254
https://github.com/restic/restic/issues/2357
https://github.com/restic/restic/pull/2255

View File

@ -0,0 +1,5 @@
Enhancement: Add support for ppc64le
Adds support for ppc64le, the processor architecture from IBM.
https://github.com/restic/restic/issues/2277

View File

@ -0,0 +1,7 @@
Bugfix: Handle format verbs like '%' properly in `find` output
The JSON or "normal" output of the `find` command can now deal with file names
that contain substrings which the Golang `fmt` package considers "format verbs"
like `%s`.
https://github.com/restic/restic/issues/2281

View File

@ -0,0 +1,8 @@
Bugfix: Do not hang when run as a background job
Restic did hang on exit while restoring the terminal configuration when it was
started as a background job, for example using `restic ... &`. This has been
fixed by only restoring the terminal configuration when restic is interrupted
while reading a password from the terminal.
https://github.com/restic/restic/issues/2298

View File

@ -0,0 +1,8 @@
Bugfix: Fix mangled json output of backup command
We've fixed a race condition in the json output of the backup command
that could cause multiple lines to get mixed up. We've also ensured that
the backup summary is printed last.
https://github.com/restic/restic/issues/2389
https://github.com/restic/restic/pull/2545

View File

@ -0,0 +1,5 @@
Bugfix: Refresh lock timestamp
Long-running operations did not refresh lock timestamp, resulting in locks becoming stale. This is now fixed.
https://github.com/restic/restic/issues/2390

View File

@ -0,0 +1,12 @@
Enhancement: Ignore sync errors when operation not supported by local filesystem
The local backend has been modified to work with filesystems which doesn't support
the `sync` operation. This operation is normally used by restic to ensure that data
files are fully written to disk before continuing.
For these limited filesystems, saving a file in the backend would previously fail with
an "operation not supported" error. This error is now ignored, which means that e.g.
an SMB mount on macOS can now be used as storage location for a repository.
https://github.com/restic/restic/issues/2395
https://forum.restic.net/t/sync-errors-on-mac-over-smb/1859

View File

@ -0,0 +1,7 @@
Enhancement: Add flag `--iexclude-file` to backup command
The backup command now supports the flag `--iexclude-file` which is a
case-insensitive version of `--exclude-file`.
https://github.com/restic/restic/issues/2427
https://github.com/restic/restic/pull/2898

View File

@ -0,0 +1,6 @@
Bugfix: backup --json reports total_bytes_processed as 0
We've fixed the json output of total_bytes_processed. The non-json output
was already fixed with pull request #2138 but left the json output untouched.
https://github.com/restic/restic/issues/2429

View File

@ -0,0 +1,5 @@
Bugfix: Fix incorrect bytes stats in `diff` command
In some cases, the wrong number of bytes (e.g. 16777215.998 TiB) were reported by the `diff` command. This is now fixed.
https://github.com/restic/restic/issues/2469

View File

@ -0,0 +1,9 @@
Change: Remove vendored dependencies
We've removed the vendored dependencies (in the subdir `vendor/`). When
building restic, the Go compiler automatically fetches the dependencies. It
will also cryptographically verify that the correct code has been fetched by
using the hashes in `go.sum` (see the link to the documentation below).
https://github.com/restic/restic/issues/2482
https://golang.org/cmd/go/#hdr-Module_downloading_and_verification

View File

@ -0,0 +1,16 @@
Bugfix: Do not crash with Synology NAS sftp server
It was found that when restic is used to store data on an sftp server on a
Synology NAS with a relative path (one which does not start with a slash), it
may go into an endless loop trying to create directories on the server. We've
fixed this bug by using a function in the sftp library instead of our own
implementation.
The bug was discovered because the Synology sftp server behaves erratic with
non-absolute path (e.g. `home/restic-repo`). This can be resolved by just using
an absolute path instead (`/home/restic-repo`). We've also added a paragraph in
the FAQ.
https://github.com/restic/restic/issues/2518
https://github.com/restic/restic/issues/2363
https://github.com/restic/restic/pull/2530

View File

@ -0,0 +1,5 @@
Bugfix: Fix incorrect size calculation in `stats --mode restore-size`
The restore-size mode of stats was counting hard-linked files as if they were independent.
https://github.com/restic/restic/issues/2531

View File

@ -0,0 +1,5 @@
Bugfix: Fix incorrect file counts in `stats --mode restore-size`
The restore-size mode of stats was failing to count empty directories and some files with hard links.
https://github.com/restic/restic/issues/2537

View File

@ -0,0 +1,8 @@
Enhancement: Support excluding files by their size
The `backup` command now supports the `--exclude-larger-than` option to exclude files which are
larger than the specified maximum size. This can for example be useful to exclude unimportant
files with a large file size.
https://github.com/restic/restic/issues/2569
https://github.com/restic/restic/pull/2914

View File

@ -0,0 +1,16 @@
Enhancement: Self-heal missing file parts during backup of unchanged files
We've improved the resilience of restic to certain types of repository corruption.
For files that are unchanged since the parent snapshot, the backup command now
verifies that all parts of the files still exist in the repository. Parts that are
missing, e.g. from a damaged repository, are backed up again. This verification
was already run for files that were modified since the parent snapshot, but is
now also done for unchanged files.
Note that restic will not backup file parts that are referenced in the index but
where the actual data is not present on disk, as this situation can only be
detected by restic check. Please ensure that you run `restic check` regularly.
https://github.com/restic/restic/issues/2571
https://github.com/restic/restic/pull/2827

View File

@ -0,0 +1,9 @@
Enhancement: Support filtering snapshots by tag and path in the stats command
We've added filtering snapshots by `--tag tagList` and by `--path path` to
the `stats` command. This includes filtering of only 'latest' snapshots or
all snapshots in a repository.
https://github.com/restic/restic/issues/2858
https://github.com/restic/restic/pull/2859
https://forum.restic.net/t/stats-for-a-host-and-filtered-snapshots/3020

View File

@ -0,0 +1,20 @@
Enhancement: Add command for copying snapshots between repositories
We've added a copy command, allowing you to copy snapshots from one
repository to another.
Note that this process will have to read (download) and write (upload) the
entire snapshot(s) due to the different encryption keys used on the source
and destination repository. Also, the transferred files are not re-chunked,
which may break deduplication between files already stored in the
destination repo and files copied there using this command.
To fully support deduplication between repositories when the copy command is
used, the init command now supports the `--copy-chunker-params` option,
which initializes the new repository with identical parameters for splitting
files into chunks as an already existing repository. This allows copied
snapshots to be equally deduplicated in both repositories.
https://github.com/restic/restic/issues/323
https://github.com/restic/restic/pull/2606
https://github.com/restic/restic/pull/2928

View File

@ -0,0 +1,10 @@
Enhancement: Use optimized library for hash calculation of file chunks
We've switched the library used to calculate the hashes of file chunks, which
are used for deduplication, to the optimized Minio SHA-256 implementation.
Depending on the CPU it improves the hashing throughput by 10-30%. Modern x86
CPUs with the SHA Extension should be about two to three times faster.
https://github.com/restic/restic/issues/551
https://github.com/restic/restic/pull/2709

View File

@ -0,0 +1,23 @@
Enhancement: Simplify and improve restore performance
Significantly improves restore performance of large files (i.e. 50M+):
https://github.com/restic/restic/issues/2074
https://forum.restic.net/t/restore-using-rclone-gdrive-backend-is-slow/1112/8
https://forum.restic.net/t/degraded-restore-performance-s3-backend/1400
Fixes "not enough cache capacity" error during restore:
https://github.com/restic/restic/issues/2244
NOTE: This new implementation does not guarantee order in which blobs
are written to the target files and, for example, the last blob of a
file can be written to the file before any of the preceding file blobs.
It is therefore possible to have gaps in the data written to the target
files if restore fails or interrupted by the user.
The implementation will try to preallocate space for the restored files
on the filesystem to prevent file fragmentation. This ensures good read
performance for large files, like for example VM images. If preallocating
space is not supported by the filesystem, then this step is silently skipped.
https://github.com/restic/restic/pull/2195
https://github.com/restic/restic/pull/2893

View File

@ -0,0 +1,8 @@
Enhancement: Improve speed of check command
We've improved the check command to traverse trees only once independent of
whether they are contained in multiple snapshots. The check command is now much
faster for repositories with a large number of snapshots.
https://github.com/restic/restic/pull/2328
https://github.com/restic/restic/issues/2284

View File

@ -0,0 +1,5 @@
Enhancement: support user@domain parsing as user
Added the ability for user@domain-like users to be authenticated over SFTP servers.
https://github.com/restic/restic/pull/2423

View File

@ -0,0 +1,19 @@
Change: Return exit code 3 when failing to backup all source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).
This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:
- 0 when the command was successful
- 1 when there was a fatal error (no snapshot created)
- 3 when some source data could not be read (incomplete snapshot created)
https://github.com/restic/restic/pull/2546
https://github.com/restic/restic/issues/956
https://github.com/restic/restic/issues/2064
https://github.com/restic/restic/issues/2526
https://github.com/restic/restic/issues/2364

View File

@ -0,0 +1,9 @@
Enhancement: Improve the chunking algorithm
We've updated the chunker library responsible for splitting files into smaller
blocks. It should improve the chunking throughput by 5-15% depending on the
CPU.
https://github.com/restic/restic/pull/2576
https://github.com/restic/restic/pull/2845
https://github.com/restic/restic/issues/2820

View File

@ -0,0 +1,6 @@
Bugfix: SFTP backend supports IPv6 addresses
The SFTP backend now supports IPv6 addresses natively, without relying on
aliases in the external SSH configuration.
https://github.com/restic/restic/pull/2592

View File

@ -0,0 +1,6 @@
Enhancement: Improve speed of diff command
We've improved the performance of the diff command when comparing snapshots
with similar content. It should run up to twice as fast as before.
https://github.com/restic/restic/pull/2598

View File

@ -0,0 +1,6 @@
Enhancement: Slightly reduce memory usage of prune and stats commands
The prune and the stats command kept directory identifiers in memory twice
while searching for used blobs.
https://github.com/restic/restic/pull/2599

View File

@ -0,0 +1,14 @@
Change: Update dependencies, require Go >= 1.13
Restic now requires Go to be at least 1.13. This allows simplifications in the
build process and removing workarounds.
This is also probably the last version of restic still supporting mounting
repositories via fuse on macOS. The library we're using for fuse does not
support macOS any more and osxfuse is not open source any more.
https://github.com/restic/restic/pull/2600
https://github.com/restic/restic/pull/2852
https://github.com/restic/restic/pull/2927
https://github.com/bazil/fuse/issues/224
https://github.com/osxfuse/osxfuse/issues/590

View File

@ -0,0 +1,7 @@
Bugfix: Honor RESTIC_CACHE_DIR environment variable on Mac and Windows
On Mac and Windows, the RESTIC_CACHE_DIR environment variable was ignored.
This variable can now be used on all platforms to set the directory where
restic stores caches.
https://github.com/restic/restic/pull/2607

View File

@ -0,0 +1,6 @@
Bugfix: Don't abort the stats command when data blobs are missing
Running the stats command in the blobs-per-file mode on a repository with
missing data blobs previously resulted in a crash.
https://github.com/restic/restic/pull/2668

View File

@ -0,0 +1,8 @@
Bugfix: Add stricter prune error checks
Additional checks were added to the prune command in order to improve
resiliency to backend, hardware and/or networking issues. The checks now
detect a few more cases where such outside factors could potentially cause
data loss.
https://github.com/restic/restic/pull/2674

View File

@ -0,0 +1,6 @@
Enhancement: S3 backend: Add support for WebIdentityTokenFile
We've added support for EKS IAM roles for service accounts feature to the S3 backend.
https://github.com/restic/restic/pull/2733
https://github.com/restic/restic/issues/2703

View File

@ -0,0 +1,6 @@
Enhancement: Optimize handling of new index entries
Restic now uses less memory for backups which add a lot of data, e.g. large initial backups.
In addition, we've improved the stability in some edge cases.
https://github.com/restic/restic/pull/2773

View File

@ -0,0 +1,8 @@
Enhancement: Reduce memory consumption of in-memory index
We've improved how the index is stored in memory.
This change can reduce memory usage for large repositories by up to 50%
(depending on the operation).
https://github.com/restic/restic/pull/2781
https://github.com/restic/restic/pull/2812

View File

@ -0,0 +1,6 @@
Enhancement: Optimize `list blobs` command
We've changed the implementation of `list blobs` which should be now a bit faster
and consume almost no memory even for large repositories.
https://github.com/restic/restic/pull/2786

View File

@ -0,0 +1,6 @@
Enhancement: Optimized file access in restic mount
Reading large (> 100GiB) files from restic mountpoints is now faster,
and the speedup is greater for larger files.
https://github.com/restic/restic/pull/2790

View File

@ -0,0 +1,8 @@
Change: Honor the --no-lock flag in the mount command
The mount command now does not lock the repository if given the
--no-lock flag. This allows to mount repositories which are archived
on a read only backend/filesystem.
https://github.com/restic/restic/issues/1597
https://github.com/restic/restic/pull/2821

View File

@ -0,0 +1,7 @@
Enhancement: Speed-up file deletion in forget, prune and rebuild-index
We've sped up the file deletion for the commands forget, prune and
rebuild-index, especially for remote repositories.
Deletion was sequential before and is now run in parallel.
https://github.com/restic/restic/pull/2840

View File

@ -0,0 +1,9 @@
Bugfix: Fix possible crash in the progress bar of check --read-data
We've fixed a possible crash while displaying the progress bar for the
check --read-data command. The crash occurred when the length of the
progress bar status exceeded the terminal width, which only happened for
very narrow terminal windows.
https://github.com/restic/restic/pull/2899
https://forum.restic.net/t/restic-rclone-pcloud-connection-issues/2963/15

View File

@ -0,0 +1,11 @@
Bugfix: Restore timestamps and permissions on intermediate directories
When using the `--include` option of the restore command, restic restored
timestamps and permissions only on directories selected by the include pattern.
Intermediate directories, which are necessary to restore files located in sub-
directories, were created with default permissions. We've fixed the restore
command to restore timestamps and permissions for these directories as well.
https://github.com/restic/restic/issues/1212
https://github.com/restic/restic/issues/1402
https://github.com/restic/restic/pull/2906

View File

@ -0,0 +1,15 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications
of file permissions.
restic now marks files stored in a local repository as read-only on the
filesystem on non-Windows systems. The error handling is improved to support
more filesystems.
https://github.com/restic/restic/issues/1756
https://github.com/restic/restic/issues/2157
https://github.com/restic/restic/pull/2989

View File

@ -0,0 +1,10 @@
Bugfix: Hide password in REST backend repository URLs
When using a password in the REST backend repository URL,
the password could in some cases be included in the output
from restic, e.g. when initializing a repo or during an error.
The password is now replaced with "***" where applicable.
https://github.com/restic/restic/issues/2241
https://github.com/restic/restic/pull/2658

View File

@ -0,0 +1,12 @@
Bugfix: Correctly dump directories into tar files
The dump command previously wrote directories in a tar file in a way which
can cause compatibility problems. This caused, for example, 7zip on Windows
to not open tar files containing directories. In addition it was not possible
to dump directories with extended attributes. These compatibility problems
are now corrected.
In addition, a tar file now includes the name of the owner and group of a file.
https://github.com/restic/restic/issues/2319
https://github.com/restic/restic/pull/3039

View File

@ -0,0 +1,9 @@
Bugfix: Don't require `self-update --output` placeholder file
`restic self-update --output /path/to/new-restic` used to require that
new-restic was an existing file, to be overwritten. Now it's possible
to download an updated restic binary to a new path, without first
having to create a placeholder file.
https://github.com/restic/restic/issues/2491
https://github.com/restic/restic/pull/2937

View File

@ -0,0 +1,7 @@
Bugfix: Fix rare cases of backup command hanging forever
We've fixed an issue with the backup progress reporting which could cause
restic to hang forever right before finishing a backup.
https://github.com/restic/restic/issues/2834
https://github.com/restic/restic/pull/2963

View File

@ -0,0 +1,6 @@
Bugfix: Fix manpage formatting
The manpage formatting in restic v0.10.0 was garbled, which is fixed now.
https://github.com/restic/restic/issues/2938
https://github.com/restic/restic/pull/2977

View File

@ -0,0 +1,7 @@
Bugfix: Make --exclude-larger-than handle disappearing files
There was a small bug in the backup command's --exclude-larger-than
option where files that disappeared between scanning and actually
backing them up to the repository caused a panic. This is now fixed.
https://github.com/restic/restic/issues/2942

View File

@ -0,0 +1,9 @@
Bugfix: restic generate, help and self-update no longer check passwords
The commands `restic cache`, `generate`, `help` and `self-update` don't need
passwords, but they previously did run the RESTIC_PASSWORD_COMMAND (if set in
the environment), prompting users to authenticate for no reason. They now skip
running the password command.
https://github.com/restic/restic/issues/2951
https://github.com/restic/restic/pull/2987

View File

@ -0,0 +1,9 @@
Enhancement: Optimize check for unchanged files during backup
During a backup restic skips processing files which have not changed since the last backup run.
Previously this required opening each file once which can be slow on network filesystems. The
backup command now checks for file changes before opening a file. This considerably reduces
the time to create a backup on network filesystems.
https://github.com/restic/restic/issues/2969
https://github.com/restic/restic/pull/2970

View File

@ -0,0 +1,9 @@
Bugfix: Make snapshots --json output [] instead of null when no snapshots
Restic previously output `null` instead of `[]` for the `--json snapshots`
command, when there were no snapshots in the repository. This caused some
minor problems when parsing the output, but is now fixed such that `[]` is
output when the list of snapshots is empty.
https://github.com/restic/restic/issues/2979
https://github.com/restic/restic/pull/2984

View File

@ -0,0 +1,12 @@
Enhancement: Add support for Volume Shadow Copy Service (VSS) on Windows
Volume Shadow Copy Service allows read access to files that are locked by
another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitly using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340
https://github.com/restic/restic/pull/2274

View File

@ -0,0 +1,7 @@
Enhancement: Authenticate to Google Cloud Storage with access token
When using the GCS backend, it is now possible to authenticate with OAuth2
access tokens instead of a credentials file by setting the GOOGLE_ACCESS_TOKEN
environment variable.
https://github.com/restic/restic/pull/2849

View File

@ -0,0 +1,10 @@
Enhancement: New option --repository-file
We've added a new command-line option --repository-file as an alternative
to -r. This allows to read the repository URL from a file in order to
prevent certain types of information leaks, especially for URLs containing
credentials.
https://github.com/restic/restic/issues/1458
https://github.com/restic/restic/issues/2900
https://github.com/restic/restic/pull/2910

View File

@ -0,0 +1,8 @@
Enhancement: Warn if parent snapshot cannot be loaded during backup
During a backup restic uses the parent snapshot to check whether a file was
changed and has to be backed up again. For this check the backup has to read
the directories contained in the old snapshot. If a tree blob cannot be
loaded, restic now warns about this problem with the backup repository.
https://github.com/restic/restic/pull/2978

View File

@ -0,0 +1,10 @@
Bugfix: Make `mount` not create missing mount point directory
When specifying a non-existent directory as mount point for the `mount`
command, restic used to create the specified directory automatically.
This has now changed such that restic instead gives an error when the
specified directory for the mount point does not exist.
https://github.com/restic/restic/issues/1681
https://github.com/restic/restic/pull/3008

View File

@ -0,0 +1,8 @@
Bugfix: Ignore `no data available` filesystem error during backup
Restic was unable to backup files on some filesystems, for example certain
configurations of CIFS on Linux which return a `no data available` error
when reading extended attributes. These errors are now ignored.
https://github.com/restic/restic/issues/1800
https://github.com/restic/restic/pull/3034

View File

@ -0,0 +1,8 @@
Enhancement: Allow specifying percentage in `check --read-data-subset`
We've enhanced the `check` command's `--read-data-subset` option to also accept
a percentage (e.g. `2.5%` or `10%`). This will check the given percentage of
pack files (which are randomly selected on each run).
https://github.com/restic/restic/issues/2186
https://github.com/restic/restic/pull/3038

View File

@ -0,0 +1,14 @@
Enhancement: Report permanent/fatal backend errors earlier
When encountering errors in reading from or writing to storage backends,
restic retries the failing operation up to nine times (for a total of ten
attempts). It used to retry all backend operations, but now detects some
permanent error conditions so that it can report fatal errors earlier.
Permanent failures include local disks being full, SSH connections
dropping and permission errors.
https://github.com/restic/restic/issues/2453
https://github.com/restic/restic/pull/3170
https://github.com/restic/restic/issues/3180
https://github.com/restic/restic/pull/3181

View File

@ -0,0 +1,21 @@
Enhancement: Add Alibaba/Aliyun OSS support in the `s3` backend
A new extended option `s3.bucket-lookup` has been added to support
Alibaba/Aliyun OSS in the `s3` backend. The option can be set to one
of the following values:
- `auto` - Existing behaviour
- `dns` - Use DNS style bucket access
- `path` - Use path style bucket access
To make the `s3` backend work with Alibaba/Aliyun OSS you must set
`s3.bucket-lookup` to `dns` and set the `s3.region` parameter. For
example:
restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init
Note that `s3.region` must be set, otherwise the MinIO SDK tries to
look it up and it seems that Alibaba doesn't support that properly.
https://github.com/restic/restic/issues/2528
https://github.com/restic/restic/pull/2535

View File

@ -0,0 +1,9 @@
Bugfix: Report the correct owner of directories in FUSE mounts
Restic 0.10.0 changed the FUSE mount to always report the current user
as the owner of directories within the FUSE mount, which is incorrect.
This is now changed back to reporting the correct owner of a directory.
https://github.com/restic/restic/issues/2563
https://github.com/restic/restic/pull/3141

View File

@ -0,0 +1,31 @@
Bugfix: Make `backup` and `tag` commands separate tags by comma
Running `restic backup --tag foo,bar` previously created snapshots with one
single tag containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`).
Similarly, the `tag` command's `--set`, `--add` and `--remove` options would
treat `foo,bar` as one tag instead of two tags. This was inconsistent with
other commands and often unexpected when one intended `foo,bar` to mean two
tags.
To be consistent in all commands, restic now interprets `foo,bar` to mean two
separate tags (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere,
including in the `backup` and `tag` commands.
NOTE: This change might result in unexpected behavior in cases where you use
the `forget` command and filter on tags like `foo,bar`. Snapshots previously
backed up with `--tag foo,bar` will still not match that filter, but snapshots
saved from now on will match that filter.
To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can
first generate a list of the relevant snapshots using a command like:
restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id'
and then use `restic tag --set foo --set bar snapshotID [...]` to set the new
tags. Please adjust the commands to include real tag names and any additional
tags, as well as the list of snapshots to process.
https://github.com/restic/restic/issues/2688
https://github.com/restic/restic/pull/2690
https://github.com/restic/restic/pull/3197

View File

@ -0,0 +1,17 @@
Enhancement: Configurable progress reports for non-interactive terminals
The `backup`, `check` and `prune` commands never printed any progress
reports on non-interactive terminals. This behavior is now configurable
using the `RESTIC_PROGRESS_FPS` environment variable. Use for example a
value of `1` for an update every second, or `0.01666` for an update every
minute.
The `backup` command now also prints the current progress when restic
receives a `SIGUSR1` signal.
Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1`
signal prints a status report even when `--quiet` was specified.
https://github.com/restic/restic/issues/2706
https://github.com/restic/restic/issues/3194
https://github.com/restic/restic/pull/3199

View File

@ -0,0 +1,5 @@
Bugfix: Make the `cat` command respect the `--no-lock` option
The `cat` command would not respect the `--no-lock` flag. This is now fixed.
https://github.com/restic/restic/issues/2739

View File

@ -0,0 +1,13 @@
Enhancement: Add `backup` options `--files-from-{verbatim,raw}`
The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a
list of files to back up from a file. Unlike the existing `--files-from`
option, these options do not interpret the listed filenames as glob patterns;
instead, whitespace in filenames is preserved as-is and no pattern expansion is
done. Please see the documentation for specifics.
These new options are highly recommended over `--files-from`, when using a
script to generate the list of files to back up.
https://github.com/restic/restic/issues/2944
https://github.com/restic/restic/issues/3013

View File

@ -0,0 +1,18 @@
Enhancement: Allow usage of deprecated S3 `ListObjects` API
Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken
`ListObjectsV2` implementation which causes problems for restic when using
their API endpoints. When a broken server implementation is used, restic prints
errors similar to the following:
List() returned error: Truncated response should have continuation token set
As a temporary workaround, restic now allows using the older `ListObjects`
endpoint by setting the `s3.list-objects-v1` extended option, for instance:
restic -o s3.list-objects-v1=true snapshots
Please note that this option may be removed in future versions of restic.
https://github.com/restic/restic/issues/3083
https://github.com/restic/restic/pull/3085

View File

@ -0,0 +1,10 @@
Bugfix: The `--use-fs-snapshot` option now works on windows/386
Restic failed to create VSS snapshots on windows/386 with the following error:
GetSnapshotProperties() failed: E_INVALIDARG (0x80070057)
This is now fixed.
https://github.com/restic/restic/issues/3087
https://github.com/restic/restic/pull/3090

View File

@ -0,0 +1,17 @@
Change: Deleting files on Google Drive now moves them to the trash
When deleting files on Google Drive via the `rclone` backend, restic used to
bypass the trash folder required that one used the `-o rclone.args` option to
enable usage of the trash folder. This ensured that deleted files in Google
Drive were not kept indefinitely in the trash folder. However, since Google
Drive's trash retention policy changed to deleting trashed files after 30 days,
this is no longer needed.
Restic now leaves it up to rclone and its configuration to use or not use the
trash folder when deleting files. The default is to use the trash folder, as
of rclone 1.53.2. To re-enable the restic 0.11 behavior, set the
`RCLONE_DRIVE_USE_TRASH` environment variable or change the rclone
configuration. See the rclone documentation for more details.
https://github.com/restic/restic/issues/3095
https://github.com/restic/restic/pull/3102

View File

@ -0,0 +1,10 @@
Bugfix: Do not require gs bucket permissions when running `init`
Restic used to require bucket level permissions for the `gs` backend
in order to initialize a restic repository.
It now allows a `gs` service account to initialize a repository if the
bucket does exist and the service account has permissions to write/read
to that bucket.
https://github.com/restic/restic/issues/3100

View File

@ -0,0 +1,9 @@
Bugfix: Correctly detect output redirection for `backup` command on Windows
On Windows, since restic 0.10.0 the `backup` command did not properly detect
when the output was redirected to a file. This caused restic to output
terminal control characters. This has been fixed by correcting the terminal
detection.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3150

View File

@ -0,0 +1,11 @@
Enhancement: Support additional environment variables for Swift authentication
The `swift` backend now supports the following additional environment variables
for passing authentication details to restic:
`OS_USER_ID`, `OS_USER_DOMAIN_ID`, `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID`
Depending on the `openrc` configuration file these might be required when the
user and project domains differ from one another.
https://github.com/restic/restic/issues/3147
https://github.com/restic/restic/pull/3158

View File

@ -0,0 +1,9 @@
Bugfix: Don't create invalid snapshots when `backup` is interrupted
When canceling a backup run at a certain moment it was possible that
restic created a snapshot with an invalid "null" tree. This caused
`check` and other operations to fail. The `backup` command now properly
handles interruptions and never saves a snapshot when interrupted.
https://github.com/restic/restic/issues/3151
https://github.com/restic/restic/pull/3164

View File

@ -0,0 +1,9 @@
Bugfix: Improve error handling in the `restore` command
The `restore` command used to not print errors while downloading file contents
from the repository. It also incorrectly exited with a zero error code even
when there were errors during the restore process. This has all been fixed and
`restore` now returns with a non-zero exit code when there's an error.
https://github.com/restic/restic/issues/3166
https://github.com/restic/restic/pull/3207

View File

@ -0,0 +1,8 @@
Enhancement: Add release binaries for MIPS architectures
We've added a few new architectures for Linux to the release binaries: `mips`,
`mipsle`, `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded
systems.
https://github.com/restic/restic/issues/3191
https://github.com/restic/restic/pull/3208

Some files were not shown because too many files have changed in this diff Show More