Commit Graph

39 Commits

Author SHA1 Message Date
Michael Eischer c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer 678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
greatroar 66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar 746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
Michael Eischer 45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
aawsome 0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
greatroar 8cf3bb8737 Revert "Put host last in SSH command line"
This reverts commit e1969d1e33.
2020-03-08 16:45:33 +01:00
greatroar e1969d1e33 Put host last in SSH command line
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar 6ac6bca7a1 Support IPv6 in SFTP backend
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
rawtaz 680a14afa1
Merge pull request #2530 from restic/fix-sftp-mkdirall
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
Lars Lehtonen 3ed54e762e
internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Alexander Neumann 2cd9c7ef16 sftp: Use MkdirAll provided by the client
Closes #2518
2020-01-01 17:26:38 +01:00
streambinder 97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
Jakob Unterwurzacher dd3b9910ee sftp: persist "ssh command exited" error
If our ssh process has died, not only the next, but all subsequent
calls to clientError() should indicate the error.

restic output when the ssh process is killed with "kill -9":

  Save(<data/afb68adbf9>) returned error, retrying after 253.661803ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/afb68adbf9>) returned error, retrying after 580.752212ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 790.150468ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 1.769595051s: ssh command exited: signal: killed
  [...]
  error in cleanup handler: ssh command exited: signal: killed

Before this patch:

  Save(<data/de698d934f>) returned error, retrying after 252.84163ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 660.236963ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 568.049909ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 2.428813824s: OpenFile: failed to send packet header: write |1: file already closed
  [...]
  error in cleanup handler: failed to send packet header: write |1: file already closed
2018-05-30 19:28:14 +02:00
Alexander Neumann 577faa7570 local/sftp: Handling non-existing dirs in List() 2018-04-10 21:35:30 +02:00
Alexander Neumann cf4cf94418 Move backend/sftp.StartForeground to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann 34f27edc03 Refactor SplitShellStrings 2018-04-01 10:16:31 +02:00
Alexander Neumann 345b6c4694 Move backend/sftp.SplitShellArgs to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann be0a5b7f06 Merge pull request #1649 from jasperla/solaris
Minimal set of patches to get restic working on Solaris
2018-03-05 20:00:17 +01:00
Jasper Lievisse Adriaanse 96311d1a2b Add support for illumos/Solaris
This does come without xattr/fuse support at this point.

NB: not hooking up the integration tests as restic won't compile without
    cgo with Go < 1.10.
2018-03-04 20:11:29 +00:00
Alexander Neumann 99f7fd74e3 backend: Improve Save()
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
2018-03-03 15:49:44 +01:00
Igor Fedorenko d58ae43317 Reworked Backend.Load API to retry errors during ongoing download
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:12:14 -05:00
Alexander Neumann 00e905ebe6 sftp: Allow running ssh without a tty 2018-01-26 19:21:14 +01:00
Alexander Neumann e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann 0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann 05958caf6e sftp: Prompt for password, don't terminate on SIGINT
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
2018-01-17 23:02:47 +01:00
Alexander Neumann b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann 7d8765a937 backend: Only return top-level files for most dirs
Fixes #1478
2017-12-14 19:14:16 +01:00
George Armhold d8938e259a sftp ReadDir: add path to return error messages (gh-1323)
fix missing "Close" string in debug log fmt
2017-10-28 14:16:27 -04:00
Alexander Neumann 4a995105a9 sftp: Fix Delete() 2017-10-14 16:08:15 +02:00
Alexander Neumann 0e722efb09 backend: Add Delete() to restic.Backend interface 2017-10-14 15:56:25 +02:00
Herbert 3473c3f7b6 Remove all dot-imports 2017-10-02 15:06:39 +02:00
Alexander Neumann 556a63de19 sftp: Return error when path starts with a tilde (~) 2017-09-30 10:34:23 +02:00
Alexander Neumann fb9729fdb9 sftp: Allow password entry
This was a bit tricky: We start the ssh binary, but we want it to ignore
SIGINT. In contrast, restic itself should process SIGINT and clean up
properly. Before, we used `setsid()` to give the ssh process its own
process group, but that means it cannot prompt the user for a password
because the tty is gone.

So, now we're passing in two functions that ignore SIGINT just before
the ssh process is started and re-install it after start.
2017-09-23 11:43:33 +02:00
Alexander Neumann 9842eff887 local/sftp: Remove unneeded stat() call 2017-09-21 21:47:03 +02:00
Alexander Neumann f9a934759f sftp: Improve error handling for non-existing dir 2017-08-27 20:53:04 +02:00
Alexander Neumann 23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann 6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann 83d1a46526 Moves files 2017-07-23 14:19:13 +02:00