3f878aa8e7
Merge pull request #4845 from greatroar/errors
...
Fix error handling bug + clean up error messages
2024-06-07 17:07:07 +00:00
0a70bbcea5
Merge pull request #4844 from MichaelEischer/improve-timeout-error
...
backend: Improve timeout error message
2024-06-07 19:05:39 +02:00
b2bbbe805f
azure: Improve error message in azure.Create()
2024-06-03 23:37:17 +02:00
660679c2f6
Merge pull request #4835 from MichaelEischer/fix-list-cancel
...
backend/retry: do not log final error if context was canceled
2024-06-01 19:17:30 +02:00
db2398f35b
backend: increase request progress timeout to 5 minutes
...
Apparently, 2 minutes are too short in some cases and can result in
canceled List requests.
2024-06-01 19:01:51 +02:00
6bf3d4859f
backend: improve error on http request timeout
...
Now yields a "request timeout" error instead of "context canceled".
2024-06-01 18:52:39 +02:00
4a874000b7
gs: Replace some errors.Wrap calls
...
The first one in Create is already a WithStack error. The rest were
referencing code that hasn't existed for quite some time. Note that
errors from Google SDKs tends to start with "google:" or "googleapi:".
Also, use restic/internal/errors.
2024-06-01 15:11:06 +02:00
38654a3bd7
backend/retry: do not log final error if context was canceled
...
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
2024-05-30 18:48:52 +02:00
de7b418bbe
http: allow custom `User-Agent` for outgoing HTTP requests
2024-05-30 15:38:06 +02:00
8e5d7d719c
cache: move to backend package
2024-05-24 23:04:06 +02:00
860b595a8b
backend: increase watchdog test timeout for deflaking
2024-05-24 21:33:17 +02:00
e4a48085ae
backend/retry: feature flag new retry behavior
2024-05-24 20:24:02 +02:00
98709a4372
retry: reduce total number of retries
...
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions
The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.
Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
512cd6ef07
retry: ensure that there's always at least one retry
...
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
a60ee9b764
retry: limit retries based on elapsed time not count
...
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
a3633cad9e
retry: explicitly log failed requests
...
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
c56ecec9aa
azure: deduplicate cli and default credentials case
2024-05-18 22:15:54 +02:00
355f520936
Azure: add option to force use of CLI credential
2024-05-18 22:15:54 +02:00
0c1ba6d95d
backend: remove unused Location method
2024-05-18 21:38:31 +02:00
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
47232bf8b0
backend: move LimitReadCloser to util package
...
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
53d15bcd1b
retry: add circuit breaker to load method
...
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
394c8ca3ed
rest/rclone/s3/sftp/swift: move short file detection behind feature gate
...
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
aeb7eb245c
retry: do not retry permanent errors
...
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
bf8cc59889
Use generic backend-error-redesign feature flag instead of http-timeouts
...
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
4740528a0b
backend: add tests for IsPermanentError
2024-05-18 19:54:52 +02:00
6a85df7297
backend: add IsPermanentError() method to interface
2024-05-18 19:54:52 +02:00
cfc420664a
mem: stricter handling of out of bounds requests
2024-05-18 19:54:52 +02:00
d40f23e716
azure/b2/gs/s3/swift: adapt cloud backend
2024-05-18 19:54:51 +02:00
e793c002ec
local: stricter handling of short files
2024-05-18 19:54:21 +02:00
b4895ebd76
rest: rework error reporting and report too short files
2024-05-18 19:54:21 +02:00
eaa3f81d6b
sftp: check for truncated files without an extra backend request
2024-05-18 19:54:21 +02:00
c6d74458ee
sftp: improve handling of too short files
2024-05-18 19:54:21 +02:00
b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance
...
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
ebd01a4675
backend: add tests for watchdogRoundTripper
2024-05-09 23:46:17 +02:00
8778670232
backend: cancel stuck http requests
...
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
0987c731ec
backend: configure protocol-level connection health checks
...
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
935327d480
sftp: slightly increase write concurrency
...
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
669a669603
sftp: Fix upload performance issue
...
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
...
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
35277b7797
backend/mem: cleanup not found error message
2024-04-14 13:45:06 +02:00
55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout
...
Deprecate s3legacy layout
2024-04-11 22:09:34 +02:00
591b421c4a
Deprecate s3legacy layout
2024-04-10 21:27:56 +02:00
6e775d3787
Enhancement: option to send HTTP over unix socket
...
add tests for unix socket connection
switch HTTP rest-server test to use any free port
allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
c0514dd8ba
Fix linter errors (except for tests)
2024-02-10 22:58:10 +01:00
a763a5c67d
s3: minor cleanups for archive storage class handling
2024-01-20 11:25:28 +01:00
8ca58b487c
S3: do not set storage class for metadata when using archive storage
2024-01-20 11:04:15 +01:00
bd883caae1
CI: enable bodyclose linter
2024-01-19 21:17:18 +01:00
b1a8fd1d03
rest: fix and cleanup closing of http response body
...
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.
This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
...
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer
Michael Eischer
Leo R. Lundgren
greatroar
Srigovind Nayak
Maik Riechert
Adam Eijdenberg
Alexander Neumann
Vladislav Belous