Commit Graph

1792 Commits

Author SHA1 Message Date
Michael Eischer ee305e6041 backup: rework error reporting for subcommand 2023-10-27 23:58:52 +02:00
Michael Eischer 8bceb8e359 fs: add tests for CommandReader 2023-10-27 23:58:52 +02:00
Michael Eischer 317144c1d6 fs: merge command startup into CommandReader 2023-10-27 23:58:51 +02:00
Michael Eischer 7d879705ad fs: cleanup CommandReader implementation 2023-10-27 23:58:51 +02:00
Enrico204 37a312e505 restic-from-command: use standard behavior when no output and exit code 0 from command
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204 81f8d473df restic-from-command: abort snapshot on non-zero exit codes 2023-10-27 23:58:51 +02:00
Enrico204 4e5caab114 stdin-from-command: implemented suggestions in #4254
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß c133065a9f Check command result before snapshotting
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß 25350a9c55 Extend SnapshotOptions w/ command data
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Michael Eischer a8fdcf79b7 restorer: Make hardlink index generic
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
2023-10-27 23:40:42 +02:00
Michael Eischer 45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Leo R. Lundgren aafb806a8c doc: Correct two typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr 41e6a02bcc Fix typos 2023-10-27 18:56:32 +02:00
Michael Eischer c7b770eb1f convert MemorizeList to be repository based
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer 1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer ceb0774af1 backend: make LoadAll independent of restic package 2023-10-25 22:58:39 +02:00
Michael Eischer b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer 7881309d63 backend: move backend implementation helpers to util package
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
Michael Eischer 8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer a28940ea29 check: Suggest usage of `restic repair packs` for corrupted blobs
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Arash Farr d15ffd9c92 retry: Do not retry Load() if file does not exist 2023-10-22 13:25:32 -05:00
Michael Eischer 7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer 6a4d6d5da4 sftp: test that Args and Command options cannot be set at the same time 2023-10-21 19:26:39 +02:00
Michael Manganiello 41f70f1f4f backend/sftp: Add sftp.args option
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.

Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer 6c7560e537 backend/b2: Switch to official library `Backblaze/blazer`
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.

Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace f1877e721e
feat: support reading REST credentials from env 2023-10-20 13:04:58 +02:00
Michael Eischer d27cfd10a9
Merge pull request #4487 from lbausch/tweak-restore-summary
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Eischer d173d1c532
Merge pull request #4486 from MichaelEischer/separate-large-blobs
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer cb9cbe55d9 repository: store oversized blobs in separate pack files
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer f750aa8dfb backend/limiter: replace juju/ratelimit with x/time/rate 2023-10-17 22:49:25 +02:00
Michael Eischer 91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer 3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Lorenz Bausch 2657217574
Tweak restore summary 2023-09-20 10:59:55 +02:00
Michael Eischer 6e586b64e4
Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
greatroar 691c01963b internal/restic: Return summary from BlobSet.String
Fixes #4449: error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer 2bec99dc6f master_index: fix inconsistent length blob length in test
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer feb6abb7bb
Merge pull request #4394 from greatroar/acl
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar aaf5254e26 dump: Rewrite Linux ACL handling
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Michael Eischer 4a33370072 backend: Fix unreliable TestListCancel/Timeout test
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.

The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer f12bbd9229 restic: check that Node.LinkTarget can handle non-utf8 characters 2023-07-23 00:05:14 +02:00
Michael Eischer 6adb629608 Add support for non-utf8 symlink targets 2023-07-23 00:05:14 +02:00
Michael Eischer 25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer 3f63b53090 lock: Shrink critical section in RefreshStaleLock
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer 96eada3d5f backend/sema: add test for freeze functionality 2023-07-22 23:23:17 +02:00
Michael Eischer 8154f6a77a Ensure consistent naming for `<snapshot>:<subfolder>` syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer 321cc35cde restic: add test for FindTreeDirectory 2023-07-22 19:55:57 +02:00
Michael Eischer a97915642c restic: Ensure snapshots created by TestCreateSnapshot are valid
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00