rdelaage
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,566 Commits 8 Branches 45 Tags 76 MiB
Commit Graph

821 Commits

Author SHA1 Message Date
Alexander Neumann 2bd9c9247c checker: Remove Load() from test error backend 2017-01-23 17:54:12 +01:00
Alexander Neumann fc235317fe backend: Use Get instead of Load for ReaderAt 2017-01-23 17:54:12 +01:00
Alexander Neumann 82d9163955 backend: Ensure Reader is closed on error 2017-01-23 17:54:12 +01:00
Alexander Neumann 4a354befe5 Fix checker test 2017-01-23 17:54:12 +01:00
Alexander Neumann 212936eb52 Make backend.LoadAll() similar to ioutil.ReadAll() 2017-01-23 17:54:12 +01:00
Alexander Neumann 05afedd950 Add backend.Get() 2017-01-23 17:54:11 +01:00
Alexander Neumann a36c01372d Use streaming functions for saving data in repo 2017-01-23 17:54:11 +01:00
Alexander Neumann 9b48da5b4e Change backend Save() function signature 2017-01-23 17:54:11 +01:00
Alexander Neumann c93f79f0f3 Add hashing package 2017-01-23 17:54:11 +01:00
Alexander Neumann 89a5152f7d Fix Archiver test: Clean up temp files 2017-01-23 17:53:59 +01:00
Alexander Neumann 47bd9cdf2f Add options for creating a memory or CPU profile 2017-01-23 17:52:26 +01:00
Alexander Neumann 668a36a652 Add option for debug pprof service 2017-01-23 17:27:42 +01:00
Alexander Neumann af1cc0717b Add integration test for forget and prune 2017-01-22 22:23:30 +01:00
Alexander Neumann 5e3365d233 Index: Store pack ID 2017-01-22 22:10:36 +01:00
Alexander Neumann 4f780a01f9 Index: Test pack ID 2017-01-22 22:09:56 +01:00
Alexander Neumann dc6a832cc3 Correct BenchmarkIndexSave 2017-01-22 09:59:19 +01:00
Alexander Neumann 164ba823e5 Merge pull request #731 from restic/improve-memory-usage 
Improve memory usage
 2017-01-20 15:56:31 +01:00
Alexander Neumann 8dd7fe82ff Add TestIndexSave 2017-01-20 14:46:14 +01:00
Mirko Dziadzka d47758a540 fix typo 2017-01-18 10:46:04 +01:00
Alexander Neumann dac18e3bf8 Improve BenchmarkIndexSave 2017-01-17 13:00:59 +01:00
Alexander Neumann c4f44c7bcb Reduce memory consuption of TestCreateSnapshot 2017-01-17 12:56:20 +01:00
Alexander Neumann 73ad3d418d Index: Remove unneeded allocation 2017-01-17 12:46:41 +01:00
Alexander Neumann 36276c41b2 Add Benchmark for IndexSave 2017-01-17 10:40:58 +01:00
Alexander Neumann d40f566e41 Index: Use slices instead of maps, reduce data 2017-01-17 10:40:58 +01:00
Alexander Neumann cd9b526203 Preallocate pack entries list 2017-01-17 10:40:58 +01:00
Alexander Neumann caabc4ec44 Reduce memory usage while decoding index 2017-01-17 10:40:58 +01:00
Alexander Neumann 73e7a2bea8 Add BenchmarkLoadIndex 2017-01-17 10:40:58 +01:00
Alexander Neumann e463587bad Add BenchmarkDecodeIndex 2017-01-17 10:40:57 +01:00
Alexander Neumann e571b6a656 Use the same buffer for decryption 2017-01-17 10:40:57 +01:00
Alexander Neumann 710499cf46 Add benchmark for LoadAndDecrypt 2017-01-17 10:40:57 +01:00
Alexander Neumann 32a5c2c1f6 Add a few functions to calculate Blob buffer len 2017-01-17 10:40:57 +01:00
Alexander Neumann 91dcb958e0 Fix tests 2017-01-17 10:40:57 +01:00
Alexander Neumann 9a5b9253c4 LoadBlob: use buffer as scratch space 
benchmark               old bytes     new bytes     delta
    BenchmarkLoadBlob-4     1010128       2256          -99.78%
 2017-01-17 10:40:57 +01:00
Alexander Neumann 215af5c60a Add LoadBlob benchmark 2017-01-17 10:40:57 +01:00
Alexander Neumann 8734c2466c Fix call to debug.Log() 2017-01-17 10:40:57 +01:00
Alexander Neumann 0556687584 Reduce memory usage for prune 2017-01-17 10:40:57 +01:00
Alexander Neumann 094e80f4a4 Merge pull request #723 from ulziibuyan/ls-latest-cmd 
Added latest keyword in ls command.
 2017-01-15 12:40:58 +01:00
Uzi c13a0953c8 User interface inconsistency fixed 2017-01-14 11:19:47 +08:00
Uzi 02c02283cf Added latest keyword in ls command. 2017-01-12 19:24:08 +08:00
Matthew Holt 0a34a2d5d8 Consider the environment 2017-01-02 12:21:30 -07:00
Matthew Holt a394b675b0 CLI options now override env vars 2017-01-02 11:14:22 -07:00
Alexander Neumann f9501e97a2 Only add entries to indexes inside PackerManager 
This was a nasty bug. Users reported that restic aborts with panic:

    panic: store new item in finalized index

The code calling panic() is in the Store() method of an index and guards
the failure case that an index is to be modified while it has already
been saved in the repo.

What happens here (at least that's what I suspect): PackerManager calls
Current() on a MasterIndex, which yields one index A. Concurrently,
another goroutine calls Repository.SaveFullIndex(), which in turn calls
MasterIndex.FullIndexes(), which (among others) yields the index A. Then
all indexes are marked as final. Then the other goroutine is executed
which adds an entry to the index A, which is now marked as final. Then
the panic occurs.

The commit solves this by removing MasterIndex.Current() and adding a
Store() method that stores the entry in one non-finalized index. This
method uses the same RWMutex as the other methods (e.g. FullIndexes()),
thereby ensuring that the full indexes can only be processed before or
after Store() is called.

Closes #367
 2017-01-02 14:14:51 +01:00
Zlatko Čalušić 2267aca296 Rest server moved to https://github.com/restic/rest-server 2017-01-01 16:22:46 +01:00
Alexander Neumann 80457018d7 Make sure cleanup is executed before exiting 
Closes #708
 2016-12-28 10:53:31 +01:00
Alexander Neumann b0997d05fb Merge pull request #704 from restic/remove-timestamp 
Remove timestamp from `version` command
 2016-12-19 22:22:43 +01:00
Alexander Neumann 166d1811a1 Remove timestamp from `version` command 
This enables reproducible builds, for details see
https://reproducible-builds.org/docs/timestamps/
 2016-12-19 21:14:12 +01:00
Sjoerd Simons e1fc455079 Avoid duplicate backup paths 
Target directories from the from-files argument get added to the command
line args, after which all command line args were appended to the same
variable again causing duplicates. Split the used variables to avoid
this.

Signed-off-by: Sjoerd Simons <sjoerd@luon.net>
 2016-12-18 23:23:57 +01:00
Alexander Neumann 9885aeac3b Make sure SaveFile always returns a node 2016-12-14 18:56:11 +01:00
Alexander Neumann e6a40af06d Treat changed files as a warning, not an error 2016-12-10 17:14:13 +01:00
Alexander Neumann 3fcbb4ac25 Use new Node if file has changed 
Closes #604
 2016-12-10 16:54:20 +01:00
Alexander Neumann 7d71bad4eb Test if modified files are correctly saved 2016-12-10 16:36:58 +01:00
Alexander Neumann dbdfed6343 Merge pull request #690 from zcalusic/master 
Even if file changes size during backup, still save it
 2016-12-10 12:36:56 +01:00
Alexander Neumann 5e48c1fadc Merge pull request #688 from restic/fix-686 
Save snapshot after saving all pack files
 2016-12-10 12:33:58 +01:00
Zlatko Čalušić deb6dd7f72 Even if file changes size during backup, still save it 
Previously such files (typically log files) wouldn't be backed up at
all!

The proper behaviour is to backup what we can, and warn the operator
that file is possibly not complete. But it is a warning, not an error.

Closes #689
 2016-12-10 12:24:45 +01:00
Alexander Neumann c265673c8e Save snapshot after saving all pack files 
Closes #686
 2016-12-10 11:49:09 +01:00
Jan Niehusmann c5897e0d62 Update debug message 
Since client.BucketExists was changed to return a separate 'found' value, instead of reporting an error when the bucket doesn't exist, the error code path does no longer imply a call to client.MakeBucket. So the second part of the debug message, "...trying to create the bucket" doesn't apply any more.
Also, changed the name of the return value from 'ok' to 'found', matching the API documentation at https://docs.minio.io/docs/golang-client-api-reference#BucketExists.
 2016-12-05 23:12:30 +01:00
Jan Niehusmann 9267c25aa0 Omit "archived as %v" messages in quiet mode. 2016-12-03 10:28:49 +01:00
JP Mens 5996d671a0 Small typo in dry-run of remove snapshot 2016-12-02 17:33:05 +01:00
Alexander Neumann e5617b5fd1 Merge pull request #675 from restic/parent-check-hostname 
Use the hostname filter to find a parent snasphot
 2016-11-19 12:42:40 +01:00
Alexander Neumann 2828003d60 Test that existing files and dirs are restored 2016-11-15 21:41:41 +01:00
Alexander Neumann 16cef3b4c6 Use the hostname filter to find a parent snasphot 
Closes #674
 2016-11-15 21:04:51 +01:00
Alexander Neumann 699f39e3cf FindLatestSnapshot: Rename parameter to clarify meaning 2016-11-15 21:03:54 +01:00
Seb Patane 33b6a7381b Don't consider a pre-existing directory in the restore path to be a failure 
* When a directory already exists, CreateDirAt returns an error stating so
  * This means that the restoreMetadata step is skipped, so for directories which already exist no file permissions, owners, groups, etc will be restored on them
* Not returning the error if it's a "directory exists" error means the metadata will get restored
  * It also removes the superfluous "error for ...: mkdir ...: file exists" messages
* This makes the behaviour of directories consistent with that of files (which always have their content & metadata restored, regardless of whether they existed or not)
 2016-11-14 17:53:09 +10:00
Alexander Thaller b7b03dbd4a Added new flag to backup subcommand that reads the files to backup from a file 2016-11-12 15:45:32 +01:00
Alexander Neumann b56bde3f61 Remove fadvise 
This commit removes the use of FADV_DONTNEED, which also purges active
cached pages for other processes.
 2016-11-10 22:21:22 +01:00
Zlatko Čalušić d8f0e7cbd1 Fix REST backend HTTP keepalive 
This is subtle.  A combination od fast client disk (read: SSD) with lots
of files and fast network connection to restic-server would suddenly
start getting lots of "dial tcp: connect: cannot assign requested
address" errors during backup stage.  Further inspection revealed that
client machine was plagued with TCP sockets in TIME_WAIT state.  When
ephemeral port range was finally exhausted, no more sockets could be
opened, so restic would freak out.

To understand the magnitude of this problem, with ~18k ports and default
timeout of 60 seconds, it means more than 300 HTTP connections per
seconds were created and teared down.  Yeah, restic-server is that
fast. :)

As it turns out, this behavior was product of 2 subtle issues:

1) The body of HTTP response wasn't read completely with io.ReadFull()
   at the end of the Load() function.  This deactivated HTTP keepalive,
   so already open connections were not reused, but closed instead, and
   new ones opened for every new request.  io.Copy(ioutil.Discard,
   resp.Body) before resp.Body.Close() remedies this.

2) Even with the above fix, somehow having MaxIdleConnsPerHost at its
   default value of 2 wasn't enough to stop reconnecting.  It is hard to
   understand why this would be so detrimental, it could even be some
   subtle Go runtime bug.  Anyhow, setting this value to match the
   connection limit, as set by connLimit global variable, finally nails
   this ugly bug.

I fixed several other places where the response body wasn't read in
full (or at all).  For example, json.NewDecoder() is also known not to
read the whole body of response.

Unfortunately, this is not over yet. :( The check command is firing up
to 40 simultaneous connections to the restic-server.  Then, once again,
MaxIdleConnsPerHost is too low to support keepalive, and sockets in the
TIME_WAIT state pile up.  But, as this kind of concurrency absolutely
kill the poor disk on the server side, this is a completely different
bug then.
 2016-11-10 09:32:07 +01:00
Alexander Neumann 51322a1055 selectFunc: handle nil 2016-11-05 12:38:33 +01:00
Alexander Neumann c5bc802ff0 fs.DeviceID(): Return errors when fi is nil 2016-11-05 12:38:17 +01:00
Justin Clift ecc1f92787 Remove redundant check of error var e 
As per #649
 2016-10-25 18:10:53 +01:00
Alexander Neumann 1dd72693f9 forget: Remove unneeded index loading 2016-10-24 14:01:23 +02:00
Alexander Neumann fe1013e779 cmds/ls: Format timestamp 2016-10-19 22:11:37 +02:00
David 5f8a6cea6f don't print status info if running in the background 
clean

fix OS issues & format code

fix issues
 2016-10-15 18:12:19 +00:00
Alexander Neumann 250b36eeb1 Improve error message for 'forget' 
$ bin/restic forget /d 7 /w 4 /m 12
    argument "/d" is not a snapshot ID, ignoring
    argument "7" is not a snapshot ID, ignoring
    argument "/w" is not a snapshot ID, ignoring
    argument "4" is not a snapshot ID, ignoring
    argument "/m" is not a snapshot ID, ignoring
    cound not find a snapshot for ID "12", ignoring
 2016-10-10 20:55:02 +02:00
Daniel Örn ba8d960c8f using backtics instead of doublequotes 2016-10-05 08:26:32 +02:00
Daniel Örn 84421a7c68 structured file with gofmt 2016-10-05 07:30:46 +02:00
Daniel Örn 5c7325f44a Added long paths fix for samba network shares 2016-10-05 07:09:56 +02:00
Christian Kemper a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann 22f5fc5739 Improve help text for slice options 2016-09-29 20:39:55 +02:00
Alexander Neumann e994cacbfe Fix short-hand option clash 2016-09-29 20:37:45 +02:00
Alexander Neumann feed54caef Remove timing, simplify function matching 2016-09-28 20:10:40 +02:00
Alexander Neumann 4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann 2ae06a7a01 Rework debug log function 2016-09-28 19:56:03 +02:00
Alexander Neumann 25945718a1 Fix recursive call to debug.Log 2016-09-28 19:56:03 +02:00
Alexander Neumann 3601c39177 Add comments 2016-09-27 20:22:01 +02:00
Alexander Neumann 565d72ef36 Use cobra for all commands 2016-09-27 19:53:03 +02:00
Alexander Neumann a257a613d7 Fix debug log 2016-09-27 19:52:48 +02:00
Alexander Neumann 0a752b9fab test helpers: Always print stack trace 2016-09-27 19:50:26 +02:00
Alexander Neumann 04d6b5da2f Remove more unused bits 2016-09-21 20:45:18 +02:00
Alexander Neumann 1dfd3b8aa3 Remove unused bits and pieces 
Reported by https://github.com/dominikh/go-unused
 2016-09-21 20:22:32 +02:00
Alexander Neumann 0a9cbd47c7 Merge pull request #626 from rfjakob/master 
Add "-x", "--one-file-system" option
 2016-09-18 20:03:58 +02:00
Jakob Unterwurzacher 53701891a1 Add "-x", "--one-file-system" option 
Equivalent to rsync's "-x" option.

Notes to the naming:

"--exclude-other-filesystems"
is used by Duplicity,

"--one-file-system"
is used rsync and tar.

This latter should be more familiar to the user.
 2016-09-18 18:52:30 +02:00
Alexander Neumann 68b462d057 fuse: Add test for same timestamps 2016-09-18 18:30:25 +02:00
Alexander Neumann 649f789190 fuse: Fix test for timestamps with same second 2016-09-18 18:13:39 +02:00
Alexander Neumann 5494c1858e fuse: correctly handle snapshots 
The fuse code kept adding snapshots to the top-level dir "snapshots". In
addition, snapshots with the same timestamp (same second) were not added
correctly, they will now be suffixed by an incrementing counter, e.g.:

    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:44+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00-1

Closes #624
 2016-09-18 15:04:39 +02:00
Alexander Neumann b090c73bd4 Remove wrapper functions in errors package 
This way, our own errors package does not appear in the stack traces.
 2016-09-18 13:28:59 +02:00
Alexander Neumann 2b9a408ccc Return a fatal for location.Parse 2016-09-18 13:28:41 +02:00
Alexander Neumann 83c35bd6b5 Do not print stack trace when open repo failed 
Closes #622
 2016-09-18 13:24:46 +02:00
Alexander Neumann a9af896ddd Add verbose error when marshalling a node fails 
This code is introduced to watch for issue #529, in which two users
describe that restic failed encoding a time in a node to JSON with the
error message:

    panic: json: error calling MarshalJSON for type *restic.Node: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]

The error message now is:

    panic: Marshal: json: error calling MarshalJSON for type *restic.Node: node /home/fd0/shared/work/restic/restic/.git/hooks/applypatch-msg.sample has invalid ModTime year -1: -0001-01-02 03:04:05.000000006 +0053 LMT
 2016-09-17 10:43:04 +02:00
Alexander Neumann 64fe9ec048 Update github.com/jessevdk/go-flags 2016-09-15 22:29:49 +02:00
Alexander Neumann d4a2d70089 Retry umount for integration tests 2016-09-15 21:37:50 +02:00
Alexander Neumann 4ffca0f4b4 Improve integration tests for fuse 2016-09-15 21:17:20 +02:00
Alexander Neumann a0f3e94655 fuse: handle duplicate timestamps for snapshots 
This closes #606, which fails because several snapshots are created with
exactly the same timestamp, and the code checks that for each snapshot
there is a dir in the fuse mount. This fails for colliding timestamps,
so we now add a suffix "-1", "-2" etc for each duplicate timestamp.
 2016-09-15 21:15:49 +02:00
Alexander Neumann 6485a6cdc0 Simplify mount logic 2016-09-15 19:59:07 +02:00
Alexander Neumann 3975d76f23 Correct filenames for expire policy tests 2016-09-13 21:20:41 +02:00
Alexander Neumann bf6602bc1b Update golden file 2016-09-13 21:19:57 +02:00
Alexander Neumann 828267aaa3 Fix status for stdin archiver 2016-09-13 21:01:29 +02:00
Alexander Neumann a77c615909 Fix 'forget' command with tags 2016-09-13 20:56:18 +02:00
Alexander Neumann cfdf4c92f7 Add `--keep-tag` to `forget` command 2016-09-13 20:37:11 +02:00
Alexander Neumann 0f9fb37c78 Add tags to forget command 2016-09-13 20:20:55 +02:00
Alexander Neumann 673bce936e Add tags to 'backup' and 'snapshots' commands 2016-09-13 20:20:52 +02:00
Alexander Neumann 1f83635267 Add tags to snapshots and filter 2016-09-13 20:12:55 +02:00
Alexander Neumann ceb4a3ecc0 Merge pull request #613 from restic/read-password-from-file 
Read password from file
 2016-09-12 20:37:08 +02:00
Alexander Neumann 223dc78acb Improve statistics for `prune` 
Sample:

    counting files in repo
    building new index for repo
    [0:00] 100.00%  22 / 22 packs
    repository contains 22 packs (1377 blobs) with 90.610 MiB bytes
    processed 1377 blobs: 0 duplicate blobs, 0B duplicate
    load all snapshots
    find data that is still in use for 1 snapshots
    [0:00] 100.00%  1 / 1 snapshots
    found 409 of 1377 data blobs still in use, removing 968 blobs
    will delete 10 packs and rewrite 10 packs, this frees 64.232 MiB
    creating new index
    [0:00] 100.00%  7 / 7 packs
    saved new index as df467c6e
    done

Closes #581
 2016-09-12 14:26:47 +02:00
Alexander Neumann 65afeba19a Add option to read the password from a file 2016-09-12 14:09:22 +02:00
Alexander Neumann e443454c4b Add OS and Arch to 'version' output 2016-09-04 15:46:50 +02:00
Alexander Neumann b628bcee27 Remove redundant ParseID 2016-09-04 14:38:18 +02:00
Alexander Neumann dfc0cbf3a8 Use one test password 2016-09-04 14:30:14 +02:00
Alexander Neumann 512a92895f Rename WithTestEnvironment -> Env 2016-09-04 14:29:04 +02:00
Alexander Neumann 6ab425f130 Remove SetupRepo 2016-09-04 13:24:51 +02:00
Alexander Neumann f5b9ee53a3 Fix mock.Repository 2016-09-04 13:18:25 +02:00
Alexander Neumann ea073f58cf Correct comment 2016-09-04 13:08:09 +02:00
Alexander Neumann bef5c4acb8 Add mock.Repository, Rework SetupRepo 2016-09-04 12:52:43 +02:00
Alexander Neumann b5b3c0eaf8 Add repository.SaveTree 2016-09-03 21:10:25 +02:00
Alexander Neumann 1fb80bf0e2 Fix fuse mount 2016-09-03 21:10:25 +02:00
Alexander Neumann 436332d5f2 LoadDataBlob -> LoadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann fe8c12c798 Replace repolitoy.SaveAndEncrypt to SaveBlob() 2016-09-03 21:10:25 +02:00
Alexander Neumann 1cc59010f5 Remove LoadJSONPack, un-export loadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann 878c1cd936 Add more comments 2016-09-03 21:10:25 +02:00
Alexander Neumann 5170c4898a Address hound comments 2016-09-03 21:10:25 +02:00
Alexander Neumann 2054e3c026 Fix tests 2016-09-03 21:10:25 +02:00
Alexander Neumann ffbe05af9b Rework crypto, use restic.Repository everywhere 2016-09-03 21:10:25 +02:00
Alexander Neumann 84f95a09d7 Introduce LoadTreeBlob and LoadDataBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann 573410afab Fix archiver test 2016-09-03 21:10:25 +02:00
Alexander Neumann 619939ccd9 Reorder methods in interface Repository 2016-09-03 21:10:25 +02:00
Alexander Neumann 714a5d1dc4 Move tree walker to restic/walk 2016-09-03 21:10:25 +02:00
Alexander Neumann bc42dbdf87 Create package restic/errors 2016-09-03 21:10:24 +02:00
Alexander Neumann 765b5437bd Fix command 'dump' 2016-09-03 21:10:24 +02:00
Alexander Neumann 5d7b38cabf Remove sentinel errors 2016-09-03 21:10:24 +02:00
Alexander Neumann debf1fce54 Remove IDSize, TestRandomID -> NewRandomID 2016-09-03 21:10:24 +02:00
Alexander Neumann 0045f2fb61 Remove functions 2016-09-03 21:10:24 +02:00
Alexander Neumann 5764b55aee Rename Node.FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann 5e3a41dbd2 Rename struct member FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann 88d0f24ce7 Reduce lock timeout to zero 2016-09-03 21:10:24 +02:00
Alexander Neumann eb6e3ba8b3 Fix imported package 2016-09-03 21:10:24 +02:00
Alexander Neumann 528c301891 Last fixes for integration tests 2016-09-03 21:10:24 +02:00
Alexander Neumann f7ae0cb78f Fix cmds/restic for new structure 2016-09-03 21:10:24 +02:00
Alexander Neumann 3695ba5882 Tests pass for restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann 4c95d2cfdc wip 2016-09-03 21:10:24 +02:00
Alexander Neumann cc6a8b6e15 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 51d8e6aa28 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann f0600c1d5f wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 90da66261a Copy ID from backend to restic 2016-09-03 21:10:24 +02:00
Alexander Neumann 82c2dafb23 Copy interfaces and basic types to package restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann bfdd26c541 Remove (unused) cache implementation 2016-09-03 21:10:24 +02:00
Alexander Neumann e699f6d1bd Update doc comment 2016-09-03 21:10:24 +02:00
Alexander Neumann 1aa52e5e1e Replace lowlevel syscall to restore symlink times 2016-08-30 21:45:16 +02:00
Alexander Neumann 8d90588020 Add better error message for 'cat' 2016-08-30 21:19:04 +02:00
Alexander Neumann 9cf63c99cf Wrap errors #3 2016-08-29 22:16:58 +02:00
Alexander Neumann 4a0f77650b Wrap errors #2 2016-08-29 21:54:50 +02:00
Alexander Neumann b53679a24d Wrap errors 2016-08-29 21:38:34 +02:00
Alexander Neumann b06845c545 Always use errors.Cause() for testing error values 2016-08-29 19:52:03 +02:00
Alexander Neumann c55b6ee544 Add restic.Fatal/f 
This is a new error which implements the restic.Fataler interface.
Errors of this type are written to stderr, the restic exits. For all
other errors, restic prints the stack trace (if available).
 2016-08-29 19:52:00 +02:00
Alexander Neumann 045f545085 repository: Handle errors correctly 2016-08-29 19:23:50 +02:00
Alexander Neumann d3f4c816c7 Print error stack if available 2016-08-29 19:23:50 +02:00
Alexander Neumann 72aa6be38d Replace fmt.Errorf() by errors.Errorf() 2016-08-29 19:23:50 +02:00
Alexander Neumann 444a268ce0 Replace stdlib errors with github.com/pkg/errors 2016-08-29 19:23:50 +02:00
Alexander Neumann 17a38faa43 Drop dependency github.com/juju/errors 2016-08-29 19:23:50 +02:00
Alexander Neumann f51bc8e9b9 Fix panic for debug.Log() with empty string 2016-08-28 22:43:05 +02:00
Alexander Neumann 3af8f53097 Allow 'cat' for tree blobs 2016-08-28 21:23:46 +02:00
Alexander Neumann 6c6b0e2395 cat: Add warning when pack was modified 2016-08-28 21:21:04 +02:00
Alexander Neumann 26351522c5 Merge pull request #594 from restic/fix-checker 
Remove check for filemode 0
 2016-08-28 21:09:02 +02:00
Alexander Neumann dec2e4788e Remove flaky test 2016-08-28 21:06:27 +02:00
Alexander Neumann f9cd736b33 Fix flaky test 2016-08-28 21:04:35 +02:00
Alexander Neumann 88634dac3a Remove check for filemode 0 2016-08-28 20:04:09 +02:00
Alexander Neumann 83924d0864 Improve error message when sftp fails 
Also add a prefix for all errors written to stderr by the client
 2016-08-28 19:56:46 +02:00
Alexander Neumann 22bde5b277 sftp: Add debug log messages 2016-08-28 19:47:12 +02:00
Alexander Neumann cdbdf74811 Remove debug output for tests 2016-08-28 19:30:56 +02:00
Alexander Neumann db16702263 Report errors to stderr for tests 2016-08-28 19:30:56 +02:00
Alexander Neumann 5dd137d53e Improve error handling with the ssh subprocess 2016-08-28 19:30:56 +02:00
Alexander Neumann ed09887d9e Fix panic when parsing sftp URIs 
Closes #587
 2016-08-28 19:30:56 +02:00
Alexander Neumann 196bbbd25b local/sftp: Fix broken error handling 
This yields the error messages for a full backup location:

    panic: write /home/fd0/mnt/temp/tmp/temp-987810174: no space left on device

Closes #540

Also connected to #574
 2016-08-28 18:54:58 +02:00
Lluís Batlle i Rossell 3acf03986a On prune report, packs instead of files + fix counter 2016-08-27 20:04:35 +02:00
Alexander Neumann 12a904eb4b Fix reading password from stdin 
This fixes a bug introduced in #585, it must by checked for stdin and
stdout separately whether it is a terminal.
 2016-08-27 18:31:46 +02:00
Alexander Neumann 7f06ec98b8 Merge pull request #585 from trbs/progress_without_terminal 
show progress every second when run non interactively
 2016-08-27 10:10:18 +02:00
Alexander Neumann b2a67d458c Remove unneeded packs without repacking 2016-08-25 22:35:22 +02:00
Alexander Neumann de88fb2022 Simplify pack.List 2016-08-25 22:25:55 +02:00
trbs 71263b5090 show progress every second when run non interactively 2016-08-25 22:13:47 +02:00
Alexander Neumann 3fd1e4a992 Add backend.ReaderAt 2016-08-25 21:49:00 +02:00
Alexander Neumann 9f752b8306 Rework function for listing packs 2016-08-25 21:08:16 +02:00
Alexander Neumann e07ae7631c Add more safety checks for Unpacker 2016-08-23 22:21:29 +02:00
Jan Stürtz 91c458bf74 Fixed gofmt 2016-08-22 22:07:10 +02:00
Jan Stürtz 374b1144de Dont't guess the max width, get it from the terminal 2016-08-22 17:27:58 +02:00
Jan Stürtz f05b0871e9 fixed maxlen computation (off by one) on small terminals 2016-08-22 17:27:03 +02:00
Jan Stürtz 4cb8fe3210 Fixed style hints from hound 
- no else, when if has a return
- Improve Comment on Function
 2016-08-21 23:10:28 +02:00
Jan Stürtz 08eb5b42eb Fix progress output on Windows 
The windows cmd shell is not aware of ANSI escape sequences and
does print them uninterpreted to the console. This is ugly.
Added a function to generate platform specific string for the escape sequence. On Windows this will be 79 white spaces with
a trailing \r.
 2016-08-21 22:38:22 +02:00
Alexander Neumann ebd3723a06 Properly close the minio object on Stat() 
Closes #544
 2016-08-21 16:15:41 +02:00
Alexander Neumann 06b23edb39 Fix code for newer minio-go 2016-08-21 16:14:58 +02:00
Alexander Neumann a3492d69dd Use low-security scrypt KDF parameters for testing 2016-08-21 13:42:04 +02:00
Alexander Neumann 8e24c51233 Fix commets for constants 2016-08-21 13:13:05 +02:00
Alexander Neumann d8107f77aa Limit the number of key files checked on SearchKey 2016-08-21 13:10:16 +02:00
Alexander Neumann 79e950b710 Remove dead code 2016-08-21 13:10:15 +02:00
Alexander Neumann f0d7f3f1bd Calibrate scrypt for the current hardware 
Closes #17
 2016-08-21 13:10:08 +02:00
Alexander Neumann 9afec53c55 Remove crypto reader/writer (unused) 2016-08-21 13:10:08 +02:00
Alexander Neumann 11098d6eb0 Move KDF() to kdf.go 2016-08-21 13:10:08 +02:00
Alexander Neumann d965d703d1 Reduce duplicate code in wrappers for os 2016-08-21 10:42:07 +02:00
Alexander Neumann b20921d836 Use constants from /x/sys/unix 2016-08-21 10:36:20 +02:00
Alexander Neumann 2be0aa9dbc Merge pull request #518 from restic/implement-prune 
Implement prune
 2016-08-21 09:22:22 +02:00
Alexander Neumann aa29c68189 Fix progress for new index 2016-08-20 20:44:57 +02:00
Alexander Neumann d3da30e8fb Use UTC for snapshot time based tests 2016-08-20 18:49:02 +02:00
Alexander Neumann 458448357c Add help texts which cross-line prune/forget 2016-08-20 18:33:24 +02:00
Alexander Neumann 27d0909302 forget: Remove message when no policy is specified 2016-08-20 18:15:36 +02:00
Alexander Neumann 5f0ebb71b2 forget: Allow filtering for a hostname 2016-08-20 17:59:47 +02:00
Alexander Neumann 00f647dc92 forget: Join paths by ":" 2016-08-20 17:59:10 +02:00
Alexander Neumann 8e7202bd6a Rename function in debug 'dump' command 2016-08-20 17:54:27 +02:00
Alexander Neumann 5cf7c827b8 forget: Do nothing if no policy is configured 2016-08-20 17:53:03 +02:00
Alexander Neumann 71f7f4f543 Add ExpirePolicy.Empty() 2016-08-20 17:51:48 +02:00
Alexander Neumann bf47dba1c4 Add 'forget' command 2016-08-20 17:43:25 +02:00
Alexander Neumann cbd457e557 Add Hourly expire functions 2016-08-20 15:55:23 +02:00
Alexander Neumann 6cf4b81558 Add functions to filter snapshots 2016-08-20 15:22:40 +02:00
Alexander Neumann bb84d351f1 Revert "ID: move Str() to non-pointer receiver" 
This reverts commit f102406cd7.
 2016-08-19 20:45:19 +02:00
Alexander Neumann a107e3cc84 Correct comment 2016-08-19 20:36:24 +02:00
Alexander Neumann 2a2fb74ba8 Merge pull request #569 from restic/fix-568 
Use the platform-independent function for joining
 2016-08-19 17:53:09 +02:00
Alexander Neumann bd819a5e81 Fix panic 2016-08-16 21:59:43 +02:00
Alexander Neumann 162629571d Add BenchmarkFindUsedBlobs 2016-08-16 21:30:14 +02:00
Alexander Neumann 2c04ad3c29 TestCreateSnapshot: free buffer 2016-08-16 21:30:14 +02:00
Alexander Neumann 238d3807e9 prune: Format duplicate bytes properly 2016-08-16 21:30:14 +02:00
Alexander Neumann 7f9d227725 Use progress in prune command 2016-08-16 21:30:14 +02:00
Alexander Neumann 8de6e5a627 Add progress option to index 2016-08-16 21:30:14 +02:00
Alexander Neumann 8d735cf6a9 Explicitely specify supersedes for new index 2016-08-16 21:30:14 +02:00
Alexander Neumann 29bb845f0e Rebuild index at the end of prune 2016-08-16 21:30:14 +02:00
Alexander Neumann 1bb2d59e38 Add Save() method to Index 2016-08-16 21:30:14 +02:00
Alexander Neumann 3ceb2ad3cf Progress: Call OnUpdate before OnDone 2016-08-16 21:30:14 +02:00
Alexander Neumann 009c803c8a prune: Use new Index 2016-08-16 21:30:14 +02:00
Alexander Neumann c0ef1ec6fd Add RemovePack for index 2016-08-16 21:30:14 +02:00
Alexander Neumann 69c2e8ce7e Add first version of the `prune` command 2016-08-16 21:30:14 +02:00
Alexander Neumann f102406cd7 ID: move Str() to non-pointer receiver 2016-08-16 21:30:14 +02:00
Alexander Neumann 302619a11a Move interfaces to package restic/types 2016-08-16 21:30:14 +02:00
Alexander Neumann 80bcae44e2 Decouple ListAllPacks from repository 2016-08-16 21:30:14 +02:00
Alexander Neumann 1f263a7683 Decouple index/ and repository/ 2016-08-16 21:30:14 +02:00
Alexander Neumann 3b57075109 Add global interface Repository 2016-08-16 21:30:14 +02:00
Alexander Neumann 3fa7304e94 Add interfaces to ListAllPacks 2016-08-16 21:30:14 +02:00
Alexander Neumann 47950b82a0 Add test for loading index from documentation 2016-08-16 21:30:14 +02:00
Alexander Neumann 9ecf7070af Implement Lookup() and Save() for new Index 2016-08-16 21:30:14 +02:00
Alexander Neumann 2310773798 Compute negative offsets ourselves in the s3 backend 2016-08-16 21:30:14 +02:00
Alexander Neumann a60e3b5030 Make backend tests less verbose 2016-08-16 21:30:14 +02:00
Alexander Neumann b350b443d0 Stop backend tests early on failure 2016-08-16 21:30:14 +02:00
Alexander Neumann 2c517e4a33 Add Index structures for Blobs 2016-08-16 21:30:14 +02:00
Alexander Neumann 4bdd59b4ad Index: Add DuplicateBlobs() 2016-08-16 21:30:14 +02:00
Alexander Neumann f5daf33322 Add pack size to ListAllPacks 2016-08-16 21:30:14 +02:00
Alexander Neumann 1058a91b39 Add option to create duplicate blobs in TestCreateSnapshot 2016-08-16 21:30:14 +02:00
Alexander Neumann 240b8f273a Add more index tests 2016-08-16 21:30:14 +02:00
Alexander Neumann 6808523d34 Add String() for Blob 2016-08-16 21:30:14 +02:00
Alexander Neumann bad6184ab5 Add new Index data structure 2016-08-16 21:30:14 +02:00
Alexander Neumann 6b384287f3 Return error when it occurs 2016-08-16 21:30:14 +02:00
Alexander Neumann ef33cf12ca Fix Unpacker for packs < 2048 byte 2016-08-16 21:30:14 +02:00
Alexander Neumann a5cbbb8b5a Fix BufferLoader for negative offset 2016-08-16 21:30:14 +02:00
Alexander Neumann 71924fb7c0 Add tests for Load() with negative offset 2016-08-16 21:30:14 +02:00
Alexander Neumann b0565015cc Remove ReadSeeker 2016-08-16 21:30:14 +02:00
Alexander Neumann fa283c6ecd Remove unused GetReader() 2016-08-16 21:30:14 +02:00
Alexander Neumann 94d157d97a Introduce interface pack.Loader 2016-08-16 21:30:14 +02:00
Alexander Neumann f72f3dbc6a Buffer last 2048 bytes of a file for unpack 2016-08-16 21:28:55 +02:00
Alexander Neumann 3c3a180417 Move RandomID() to backend package 2016-08-16 21:28:55 +02:00
Alexander Neumann fd6c854a21 Add TestResetRepository and BenchmarkCreateSnapshot 2016-08-16 21:28:55 +02:00
Alexander Neumann e9cddc0be5 Fix TestFindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann d7e5f11b78 Export FindUsedBlobs 2016-08-16 21:28:55 +02:00
Alexander Neumann 2b1b6d8c2a Export ListAllPacks 2016-08-16 21:28:55 +02:00
Alexander Neumann acc2fa5816 Fix TestRepack 
* Decrease number of blobs for use in test
 * Fail the test when there's a duplicate blob
 2016-08-16 21:28:54 +02:00
Alexander Neumann 6285f31604 Use pack.BlobSet instead of backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann 3cca831b2e Fix invalid type in newly created packs 2016-08-16 21:28:54 +02:00
Alexander Neumann cff6fea32a Fix 'cat' command 2016-08-16 21:28:54 +02:00
Alexander Neumann 17e1872544 Switch order of parameters to repo.LoadBlob() 2016-08-16 21:28:54 +02:00
Alexander Neumann 246302375d Index: Add multiple packs per blob, pack.Type 
Change the index so that a blob can be contained in multiple packs.

Require passing the blob type to all lookup functions.
 2016-08-16 21:28:54 +02:00
Alexander Neumann 231da4ff80 Remove old repacking code 2016-08-16 21:28:54 +02:00
Alexander Neumann 1b4b469440 Add pack.Handle and pack.Handles 2016-08-16 21:28:54 +02:00
Alexander Neumann 35e3762e37 Remove dead code 2016-08-16 21:28:54 +02:00
Alexander Neumann 7e732dbd2d Allow multiple entries in the index 2016-08-16 21:28:54 +02:00
Alexander Neumann 8b4d4ec25f Fix TestCreateSnapshot, do not store duplicate data 2016-08-16 21:28:54 +02:00
Alexander Neumann 035d0aeb31 Do not create duplicate content for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann f1bc181c5b Add more checks for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann 50b724ca23 Fix stylistic issues with FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann 6227821b4e Move functions to correct file 2016-08-16 21:28:54 +02:00
Alexander Neumann 810056c2bc Correct packages for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann 34b3e3a095 Split index/repack functions to different files 2016-08-16 21:28:54 +02:00
Alexander Neumann bdd085e9f1 Prevent loops when finding used blobs 2016-08-16 21:28:54 +02:00
Alexander Neumann ffc3503e6f Add first version of FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann 51b16ad57d Add handy functions to backend.IDSet 2016-08-16 21:28:54 +02:00
Alexander Neumann 723592d923 Move FindUsedBlobs to package restic 2016-08-16 21:28:54 +02:00
Alexander Neumann 22aa17091b Add test for FindUsedBlobs 2016-08-16 21:28:54 +02:00
Alexander Neumann 4720a7d807 Allow specifying chunker polynomial for tests 2016-08-16 21:28:54 +02:00
Alexander Neumann d5323223f4 Change repository Init() function to allow better testing 2016-08-16 21:28:54 +02:00
Alexander Neumann fe79177b40 Make TestCreateSnapshot return the snapshot itself 2016-08-16 21:28:54 +02:00
Alexander Neumann 5c32ae15c2 Move test checking repo code to checker package 2016-08-16 21:28:54 +02:00
Alexander Neumann 6c2334f505 Make TestCreateSnapshot less verbose 2016-08-16 21:28:54 +02:00
Alexander Neumann b55ac2afd6 Make test files in test repo less random 2016-08-16 21:28:54 +02:00
Alexander Neumann d9012b4a64 Add trees recursively to test snapshot 2016-08-16 21:28:54 +02:00
Alexander Neumann 952f124238 Use RandReader instead of rand directly 
This is a fix to be backwards-compatible with Go < 1.6.
 2016-08-16 21:28:54 +02:00
Alexander Neumann 14db71d3fa Move RandReader to repository package 2016-08-16 21:28:54 +02:00
Alexander Neumann f59ffcaeae Correct comment 2016-08-16 21:28:54 +02:00
Alexander Neumann d609e4a986 Extended plaintext buffer if necessary 2016-08-16 21:28:54 +02:00
Alexander Neumann 0e6c72ad1d Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann d5f42201c5 Fix test for Repack 2016-08-16 21:28:54 +02:00
Alexander Neumann 122a0944a6 Do not repack blobs that shouldn't be kept 2016-08-16 21:28:54 +02:00
Alexander Neumann fa26ecc8f9 Make `rebuild-index` use the code in package repository 2016-08-16 21:28:54 +02:00
Alexander Neumann 00139648a0 Implement Repack() 2016-08-16 21:28:54 +02:00
Alexander Neumann 6ba38e9a38 Add tests for Repack() 2016-08-16 21:28:54 +02:00
Jan Stürtz b108966b12 Fix 567 (#570) 
* Patch for  https://github.com/restic/restic/issues/567
Backup also files on windows with longer pathnames than 255 chars (e.g. from node).

as fd0 says "So, as far as I can see, we need to have custom methods for all functions that accept a path, so that on Windows we can substitute the normal (possibly relative) path used within restic by an (absolute) UNC path, and only then call the underlying functions like os.Stat(), os.Lstat(), os.Open() and so on.

I've already thought about adding a generic abstraction for the file system (so we can mock this easier in tests), and this looks like a good opportunity to build it."

* fixed building tests

* Restructured patches
Add Wrapper for filepath.Walk

* using \\?\ requires absolute pathes to be used.
Now all tests run

* used gofmt on the code

* Restructured Code. No patches dir, integrate the file functions into restic/fs/

There is still an issue, because restic.fs.Open has a different api the os.Open, which returns the result of OpenFile, but takes only a string

* Changed the last os.Open() calls to fs.Open() after extending the File interface

* fixed name-clash of restic.fs and fuse.fs detected by travis

* fixed fmt with gofmt

* c&p failure: removed fixpath() call.

* missing include

* fixed includes in linux variant

* Fix for Linux. Fd() is required on File interface

* done gofmt
 2016-08-15 21:59:13 +02:00
Alexander Neumann fa4570bde8 Always use forward slashes in file names 2016-08-11 19:41:47 +02:00
Alexander Neumann f6c2787d80 Use the platform-independent function for joining 2016-08-11 19:37:22 +02:00
Alexander Neumann 4b8b625b90 Merge pull request #562 from damekr/implement-gomaxprocs-env 
Issue-535: restic respect GOMAXPROCS env variable depending on go version
 2016-08-11 19:09:54 +02:00
damekr be00d91967 Respect GOMAXPROCS variable 
Closes #535
 2016-08-08 21:37:20 +02:00
Alexander Neumann fc9b27c533 Revert "Fix TestCreateSnapshot, do not generate duplicate data" 
This reverts commit 628fb0fb72.
 2016-08-02 22:11:55 +02:00
Alexander Neumann 628fb0fb72 Fix TestCreateSnapshot, do not generate duplicate data 2016-08-01 22:01:34 +02:00
Alexander Neumann 8418fed18e Handle empty files correctly 2016-07-29 21:18:32 +02:00
Alexander Neumann 3de989b7bb Fix panic with empty files 2016-07-29 21:05:36 +02:00
Alexander Neumann 5afda94a3c Handle reads with large offsets 2016-07-29 20:55:09 +02:00
Ben Agricola edb1843f24 Explicitly Close() obj after ReadFull() 
Signed-off-by: Ben Agricola <bagricola@squiz.co.uk>
 2016-07-29 14:18:02 +01:00
mappu 32985f7904 idset.go: micro-optimise away redundant scan 2016-07-26 09:36:31 +12:00
Alexander Neumann 6b7ddf1b03 Don't report valid types as invalid 
Closes #528
 2016-07-20 20:46:57 +02:00
Alexander Neumann 902f619a06 Fix call to minio.New() 
The last parameter changed semantics from `insecure` to `secure`.
 2016-06-08 21:33:18 +02:00
Gerdus van Zyl 73e9cac5c4 gofmt + small doc fix 2016-05-10 22:20:03 +02:00
Gerdus van Zyl 8010a0d90c fix 2016-05-10 21:57:30 +02:00
Gerdus van Zyl 3cb68ddb0d Add option to restore latest snapshot with optional path and source filters 
eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"

Add option to restore latest snapshot with optional path and source filters

eg restic -r r1 restore latest --target restore2 --path "D:\dev\restic\bin\s1"
path and source filters also added to snapshot cmd
eg restic -r r1 snapshots --source nucore --path="D:\dev\restic\bin\s1"
 2016-05-10 21:41:26 +02:00
Gerdus van Zyl 49f82f54b0 rebase, change source to host and add description to manual 2016-05-10 21:40:32 +02:00
Alexander Neumann 6bc7a71e55 Merge pull request #516 from restic/fix-flaky-test 
Fix flaky worker cancel test
 2016-05-09 22:12:39 +02:00
Alexander Neumann 2c1e590e47 Merge pull request #509 from restic/read-from-stdin 
Allow reading data from stdin
 2016-05-09 22:12:22 +02:00
Alexander Neumann 84f7d28abf Merge pull request #515 from viric/fix_traverse_order 
Traverse paths in the same order as parent snapshot
 2016-05-09 22:10:36 +02:00
Alexander Neumann cb75737770 Merge pull request #514 from viric/fix_parent_search 
Better backup parent snapshot search. Part of #513
 2016-05-09 22:10:32 +02:00
Alexander Neumann fb45ea139d Add barrier 2016-05-09 21:29:13 +02:00
Alexander Neumann bce0bbeda2 Add Benchmark for ArchiveReader 2016-05-09 21:16:59 +02:00
Alexander Neumann c6d934a685 Fix flaky worker cancel test 2016-05-09 20:41:55 +02:00
Alexander Neumann 4146c09a04 Add test for ArchiveReader() 2016-05-09 20:11:32 +02:00
Alexander Neumann 43f9c2d36e backup: Save file size when reading from stdin 2016-05-09 20:11:32 +02:00
Alexander Neumann 5e0813ca04 fuse: Use correct file size in case it's zero 2016-05-09 20:11:32 +02:00
Alexander Neumann 6ee9baa9c5 fuse: Add debug logs 2016-05-09 20:11:32 +02:00
Alexander Neumann 7c76ff3aaf Allow reading backups from stdin 2016-05-09 20:11:32 +02:00
Lluís Batlle i Rossell 4818a8e356 Fix gofmt 2016-05-09 16:31:59 +02:00
Lluís Batlle i Rossell 83aa63365a Not exporting baseNameSlice. Noone else wants it. 2016-05-09 14:46:14 +02:00
Lluís Batlle i Rossell aed73be93d Improve comment according to hound guidelines 2016-05-09 14:44:03 +02:00
Lluís Batlle i Rossell 4ea62ecbcc Traverse paths in the same order as parent snapshot 
This is the 2nd partial fix to #513.

The archivepipe requires the snapshot paths and the backup paths to be
traversed in the same order, and they were sorted differently: the backup paths
by full path, and the snapshot by basename path.
 2016-05-09 14:32:17 +02:00
Lluís Batlle i Rossell 60c8c90d35 Better backup parent snapshot search. Part of #513 
I look for the newest snapshot that contains all supplied paths to backup.
 2016-05-09 12:42:12 +02:00
Alexander Neumann 20afed4058 Checker: handle symlinks 2016-05-08 23:16:17 +02:00
Alexander Neumann a2224e380b Address style issues identified by Hound 2016-05-08 22:38:38 +02:00
Alexander Neumann 31030baca3 Add comment 2016-05-08 13:51:33 +02:00
Alexander Neumann 173940cbdf Add repository.ListPack 2016-05-08 13:51:21 +02:00
Alexander Neumann 6fc3590838 Remove repository.SaveFrom() 2016-05-08 13:13:29 +02:00
Alexander Neumann 43f7a1fcd9 Correct log statement 2016-05-08 13:09:36 +02:00
Alexander Neumann 7faf272996 Progress: Use reference to sync.Once 2016-05-08 13:04:58 +02:00
Alexander Neumann 514a43f74b Add more tests 2016-05-08 12:25:01 +02:00
Alexander Neumann 6655511ab8 checker: test file mode 2016-05-08 12:25:01 +02:00
Alexander Neumann 168cfc2f6d Add testing helper functions 2016-05-08 12:25:01 +02:00
Alexander Neumann 6cfa0d502d Add LoadAllSnapshots() 2016-05-08 12:25:01 +02:00
Alexander Neumann a996dbb9d6 check: Add more checks for nodes 2016-05-08 12:25:01 +02:00
Alexander Neumann 3f8da47a0c Fix restic s3 backend for new minio-go version 2016-05-07 23:38:41 +02:00
Alexander Neumann 4531456be5 Merge pull request #497 from Thor77/excludeFileExpandEnv 
Expand environment-variables in exclude-files
 2016-04-18 21:40:30 +02:00
Alexander Neumann 039019689a Merge pull request #500 from restic/fix-499 
Fix exclude filters with trailing slash
 2016-04-18 21:02:27 +02:00
Alexander Neumann 6fe25548bd Add another filter test 2016-04-17 22:04:42 +02:00
Alexander Neumann 9002eaa259 Fix exclude filters with trailing slash 2016-04-17 21:54:12 +02:00
Alexander Neumann ead6d11ecf Backend tests: remove debug 2016-04-17 17:39:14 +02:00
Thor77 9644399074 add environment-var expanding for exclude-files 2016-04-16 22:04:29 +02:00
Alexander Neumann 23d7464306 Umount fuse in tests 
This corrects the order when the fuse mount is terminated by closing the
done channel: Before, restic would close the fuse connection and only
afterwards try to remove the mount, that does not work.

Closes #494
 2016-04-13 20:18:54 +02:00
Alexander Neumann 32a5778602 Merge pull request #490 from Thor77/backupExcludeFile 
add backup --exclude-file
 2016-04-06 00:09:46 +02:00
Thor77 1c1eacfc94 add backup --exclude-file 2016-04-01 13:53:22 +02:00
Alexander Neumann ddfadae6f6 Fix compilation for Go 1.3 2016-03-28 16:09:28 +02:00
Alexander Neumann c8f46ce81d fs: Require Go1.4 for Linux 2016-03-28 15:48:18 +02:00
Alexander Neumann c30f4a9134 fs: remove unneeded code 2016-03-28 15:33:10 +02:00
Alexander Neumann b7713d2d34 local backend: Drop file content from cache after write 2016-03-28 15:31:25 +02:00
Alexander Neumann 5b5bb070b9 fs: Split out ClearCache from File 2016-03-28 15:31:25 +02:00
Alexander Neumann feb664620a Use fadvise() to not cache the content of files read 2016-03-28 15:26:46 +02:00
Matt Holt e6ba9e5849 Change ErrNoKeyFound message 
For #438. I was just going to change it to "wrong password" but then I saw that it might actually be the case that no key could be found, so I changed it to what I did. Let me know if you'd like something different!
 2016-03-11 08:21:01 -07:00
Alexander Neumann e4a6dd8c8c Use newRandReader instead of rand.New() 
This needs to be done since for Go < 1.6 rand.Rand does not implement
io.Reader.
 2016-03-06 14:21:02 +01:00
Alexander Neumann 18c3024171 Unexport NewPackerManager 2016-03-06 14:20:48 +01:00
Alexander Neumann 1e1368eea3 Add randReader for tests 
This can be removed once we require Go 1.6.
 2016-03-06 13:59:06 +01:00
Alexander Neumann cda7616c82 Remove tempdir for packerManager 2016-03-06 13:14:06 +01:00
Alexander Neumann 015cea0c50 PackerManager: Remove debug comment 2016-03-06 12:35:21 +01:00
Alexander Neumann c0b5f5a8af Fix all code which uses repository.New() 2016-03-06 12:34:23 +01:00
Alexander Neumann f956f60f9f PackerManager: use tempfiles instead of memory buffers 2016-03-06 12:26:25 +01:00
Alexander Neumann f893ec57cb Add test and benchmark for PackerManager 2016-03-05 15:58:39 +01:00
Alexander Neumann 8d1a5731f3 Remove integration test for 'optimize' 2016-02-27 13:38:05 +01:00
Alexander Neumann 04b3ce00e2 Remove command 'optimize' 
It was discovered that the current code may delete still-referenced
blobs, so we'll remove the command for now.

This closes #434
 2016-02-27 13:12:22 +01:00
Alexander Neumann 9386bfbafa checker: Do not use reference in checker errors 2016-02-27 13:10:35 +01:00
Alexander Neumann a613e23e34 checker: Use backend.IDSet instead of custom struct 2016-02-27 13:10:35 +01:00
Alexander Neumann 5ce1375ddd Rename non-exported function 2016-02-27 13:10:35 +01:00
Alexander Neumann 4cefd456bb Refactor rebuild-index code 
This code reads all pack headers from all packs and rebuilds the index
from scratch. Afterwards, all indexes are removed. This is needed
because in #434 the command `optimize` produced a broken index that
did not contain a blob any more. Running `rebuild-index` should fix
this.
 2016-02-27 13:10:35 +01:00
Alexander Neumann bc911f4609 cmd_dump: Only load pack header 2016-02-27 13:06:21 +01:00
Alexander Neumann 090920039f pack: Add test with backend.NewReadSeeker 
This uses the new backend ReadSeeker with the unpacker.
 2016-02-27 13:06:21 +01:00
Alexander Neumann 21a99397ff worker: fix tests 
The test failed on Windows, probably because the machine used for CI was
too slow. The new test doesn't depend on timing any more.
 2016-02-27 13:06:21 +01:00
Alexander Neumann 482fc9f51d Add backend.readSeeker 
This struct implements an io.ReadSeeker on top of a backend. This is the
easiest way to make the packer compatible to the new backend without
loading a complete pack into a bytes.Buffer.
 2016-02-27 13:06:21 +01:00
Alexander Neumann b114ab7108 cmd_dump: Allow dumping all blobs in all packs 
I had the suspicion that one of my repositories contained redundant
data blobs that is not recorder in the index. In order to check this I
needed to dump information about the pack files without consulting the
index. The code here iterates over all packs and dumps the headers, and
dumps information about them in JSON (so we can use other tools to parse
that information again).
 2016-02-27 13:06:21 +01:00
Alexander Neumann 4cb4a3ac7f Add separate goroutine that closes the output chan 
This allows iterating over the output channel without having to start
another Goroutine outside of the worker pool. This also removes the need
for calling Wait().
 2016-02-27 13:06:21 +01:00
Alexander Neumann ee422110c8 Make worker pools input/output chans symmetric 
Input and output channel are now both of type `chan Job`, this makes it
possible to chain multiple worker pools together.
 2016-02-27 13:06:21 +01:00
Alexander Neumann e5ee4eba53 Add worker pool 
A worker pool is needed whenever something should be done concurrently.
This small library makes it easy to create a worker pool by specifying
channels, concurrency and a function that should be executed for each
job and returns a result and an error.
 2016-02-27 13:06:21 +01:00
Alexander Neumann 1e0b7dbdd2 Merge pull request #477 from restic/rest-backend 
rest backend: Remove indirection on http.Client
 2016-02-25 22:24:00 +01:00
Alexander Neumann 17f5b524a6 local: Replace matching code with proper Readdir() 2016-02-24 22:43:04 +01:00
Alexander Neumann 4ae16d7661 repository: Use backend.ID to load index 
This commit uses ParallelWorkFuncParseID() to load all indexes and
ignores file names with invalid format.

This fixes #475.
 2016-02-24 22:41:32 +01:00
Alexander Neumann 77d85cee52 Merge pull request #472 from restic/update-chunker 
Update chunker
 2016-02-24 21:25:15 +01:00
Alexander Neumann 2ce49ea0ee Update code to use the new Chunker interface 2016-02-23 23:14:35 +01:00
Fabian Wickborn 6005bd9833 Fix relative paths in integrations tests 2016-02-23 09:21:50 +01:00
Fabian Wickborn 442780f214 Move commands to src/cmds 2016-02-23 07:21:28 +01:00
Alexander Neumann 9c47a8abfc Merge pull request #467 from fawick/master 
Merging restic-server
 2016-02-22 20:48:18 +01:00
Fabian Wickborn dd5680dab6 restic-server: Create tmp folder 2016-02-22 20:14:11 +01:00
Fabian Wickborn 1cdbc8e1aa restic-server: Fix folder permissions 2016-02-22 20:14:11 +01:00
Fabian Wickborn e4168fdde5 restic-server: Reduce memory footprint for saving blobs 
Before, the restic-server read the whole blob (up to 8MB) into memory
prior to writing it to disk. Concurrent writes consumed a lot
of memory. This change writes the blob to a tmp file directly and
renames it afterwards in case there where no errors.
 2016-02-22 20:14:11 +01:00
Fabian Wickborn 4749e610af restic-server: Fix content length for HEAD requests 2016-02-22 20:14:11 +01:00
Fabian Wickborn 51d86370a5 Fixes for the PR 
- Removed external dependencies for test
- Prevent building restic-server w/ Go 1.3

Go versions 1.0, 1.1., and 1.2 are going to fail as well, but they
are "excluded" by README.md already.
 2016-02-22 20:13:55 +01:00
Fabian Wickborn d86c093480 Merged the restic-server by @bchapuis 
Commit ID in fd0/restic-server at time of merge is
07fae00e7ddd8751b150e2ebf0bff8b2871c77ce
 2016-02-22 20:12:50 +01:00
Alexander Neumann bb7b9ef3fc Merge pull request #466 from ckemper67/sftp-path-clean 
Cleaned up the sftp parsing logic.
 2016-02-22 19:04:56 +01:00
Alexander Neumann 5dd65a5c19 Merge pull request #464 from restic/rest-backend 
Add REST backend
 2016-02-22 18:52:37 +01:00
Christian Kemper 6eb97ca6cc Cleaned up the sftp parsing logic. 
Simplified and cleaned up the sftp parsing logic. Added support to
path.Clean the directory. Added additional tests.
 2016-02-21 10:50:36 -08:00
Christian Kemper c2716755f1 Added missing handle to the s3.Stat log message output 2016-02-21 09:59:27 -08:00
Alexander Neumann 7087efaa79 rest backend: Remove indirection on http.Client 2016-02-21 17:06:35 +01:00
Alexander Neumann 8ad98e8040 rest backend: Fixes 2016-02-21 16:35:25 +01:00
Alexander Neumann f7a10a9b9c backend tests: Test accessing config 
This commit adds real testing for accessing the config file with
different names.
 2016-02-21 16:02:13 +01:00
Alexander Neumann bd621197f8 Add rest backend to ui parser 2016-02-21 15:33:13 +01:00
Alexander Neumann ec34da2d66 Add rest backend to location 2016-02-21 15:33:13 +01:00
Alexander Neumann c2348ba768 Add REST backend 
This is a port of the original work by @bchapuis in
https://github.com/restic/restic/pull/253
 2016-02-21 15:33:13 +01:00
Alexander Neumann 625c987d23 Move sftp test 2016-02-20 20:53:40 +01:00
Alexander Neumann 841326d713 Move build.go and run_integration_tests.go to root 2016-02-20 17:31:21 +01:00
Alexander Neumann 7b6629802b Move "doc" to root dir 2016-02-20 17:31:21 +01:00
Alexander Neumann c0bd660a9e Rename package 
* github.com/restic/restic -> restic
 2016-02-20 17:31:21 +01:00
Alexander Neumann 0a8ef79dad Move top-level files 2016-02-20 17:31:21 +01:00
Alexander Neumann b63399d606 Move things around for gb 
This moves all restic source files to src/, and all vendored
dependencies to vendor/src.
 2016-02-20 17:31:20 +01:00