Commit Graph

87 Commits

Author SHA1 Message Date
Michael Pratt fa0be82da8 gs: allow backend creation without storage.buckets.get
If the service account used with restic does not have the
storage.buckets.get permission (in the "Storage Admin" role), Create
cannot use Get to determine if the bucket is accessible.

Rather than always trying to create the bucket on Get error, gracefully
fall back to assuming the bucket is accessible. If it is, restic init
will complete successfully. If it is not, it will fail on a later call.

Here is what init looks like now in different cases.

Service account without "Storage Admin":

Bucket exists and is accessible (this is the case that didn't work
before):

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
created restic backend c02e2edb67 at gs:this-bucket-does-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

Bucket exists but is not accessible:

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
create key in backend at gs:this-bucket-does-exist:/ failed:
service.Objects.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have
storage.objects.create access to object this-bucket-exists/keys/0fa714e695c8ecd58cb467cdeb04d36f3b710f883496a90f23cae0315daf0b93., forbidden

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
create backend at gs:this-bucket-does-not-exist:/ failed:
service.Buckets.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have storage.buckets.create access to bucket this-bucket-does-not-exist., forbidden

Service account with "Storage Admin":

Bucket exists and is accessible: Same

Bucket exists but is not accessible: Same. Previously this would fail
when Create tried to create the bucket. Now it fails when trying to
create the keys.

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
enter password for new backend:
enter password again:
created restic backend c3c48b481d at gs:this-bucket-does-not-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
2017-09-25 22:25:51 -07:00
Alexander Neumann 0bb2a8e0d0 cache: Synchronize downloading
This commit adds code to synchronize downloading files to the cache.
Before, requests that came in for files currently downloading would fail
because the file was not completed in the cache. Now, the code waits
until the download is completed.

Closes #1278
2017-09-25 15:58:20 +02:00
Alexander Neumann 2e72b57f2f Correct debug message 2017-09-25 14:35:37 +02:00
Alexander Neumann e262f35d0a cache: Auto-remove invalid files 2017-09-24 23:11:47 +02:00
Alexander Neumann 176bfa6529 backend: Improve ReadAt 2017-09-24 23:11:23 +02:00
Alexander Neumann 240c4cf2fd cache: In case of an error, fall back backend 2017-09-24 23:11:23 +02:00
Alexander Neumann db5ec5d876 repo: Automatically cache tree-only pack files 2017-09-24 23:11:23 +02:00
Alexander Neumann e1dfaf5d87 cache: Allow proactive caching of tree packs
This commit adds a function to the cache which can decide to proactively
load the complete pack file and store it in the cache. This is helpful
for pack files containing only tree blobs, as it is likely that the same
file is accessed again in the future.
2017-09-24 23:11:23 +02:00
Alexander Neumann 5436154f0d cache: Add PerformReadahead 2017-09-24 23:11:23 +02:00
Alexander Neumann 809e218d20 cache: Improve debug logs 2017-09-24 23:11:23 +02:00
Alexander Neumann 1eaad6cebb index: Add TreePacks() 2017-09-24 21:54:53 +02:00
Alexander Neumann 3890a947ca Clear data files in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann e299272378 repo: Try cached pack files first 2017-09-24 21:54:53 +02:00
Alexander Neumann 70248bd05a repo: Clear indexes 2017-09-24 21:54:53 +02:00
Alexander Neumann 7a5fde8f5a repository: Save pack files for trees in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann 610b676444 Automatically exclude current restic cache 2017-09-24 21:54:53 +02:00
Alexander Neumann 58699e3c90 Write CACHEDIR.TAG to cache base directory 2017-09-24 21:54:53 +02:00
Alexander Neumann 9be24a1c9f Add cache
This commits adds rudimentary support for a cache directory, enabled by
default. The cache directory is created if it does not exist. The cache
is used if there's anything in it, newly created snapshot and index
files are written to the cache automatically.
2017-09-24 21:54:53 +02:00
Michael Pratt 3b2106ed30 gs: document required permissions
In the manual, state which standard roles the service account must
have to work correctly, as well as the specific permissions required,
for creating even more specific custom roles.
2017-09-24 11:25:57 -07:00
Michael Pratt 5f4f997126 gs: minor comment cleanups
* Remove a reference to S3.
* Config can only be used for GCS, not other "gcs compatibile servers".
* Make comments complete sentences.
2017-09-24 10:10:56 -07:00
Alexander Neumann 9c6b7f688e Merge pull request #1270 from restic/sftp-allow-password-prompt
sftp: Allow password entry
2017-09-23 22:13:04 +02:00
Alexander Neumann 429106340f Merge pull request #1267 from harshavardhana/possible-fix-memory
Implement Size() and Len() to know the optimal size.
2017-09-23 14:04:15 +02:00
Alexander Neumann fb9729fdb9 sftp: Allow password entry
This was a bit tricky: We start the ssh binary, but we want it to ignore
SIGINT. In contrast, restic itself should process SIGINT and clean up
properly. Before, we used `setsid()` to give the ssh process its own
process group, but that means it cannot prompt the user for a password
because the tty is gone.

So, now we're passing in two functions that ignore SIGINT just before
the ssh process is started and re-install it after start.
2017-09-23 11:43:33 +02:00
Harshavardhana 98369f6a5d Implement Size() and Len() to know the optimal size. 2017-09-22 12:09:17 -07:00
Alexander Neumann 3541d06d07 repo: Split packers for tree and data
The code now bundles tree blobs and data blobs into different pack
files, so we'll end up with pack files that either only contain data or
trees. This is in preparation to adding a cache (#1040), because
tree-only pack files can easily be cached later on.
2017-09-22 15:36:47 +02:00
Alexander Neumann db0e3cd772 repo: Remove packer limits
This commit simplifies finding a packer: The first open packer is taken,
and the upper limit for the pack file is removed.
2017-09-22 15:36:47 +02:00
Alexander Neumann d610c60991 repo: Remove unused sync.Pool 2017-09-22 12:37:10 +02:00
Alexander Neumann 3f6e11d26e Allow sorting nodes in trees 2017-09-22 12:37:05 +02:00
Alexander Neumann 9842eff887 local/sftp: Remove unneeded stat() call 2017-09-21 21:47:03 +02:00
Tobias Klein 1a83a739dc fuse: added symlink 'latest' to snapshots-dir 2017-09-21 16:41:20 +02:00
Alexander Neumann 5479daa6d4 Merge pull request #1247 from restic/fix-1246
Fix backend List()
2017-09-20 11:04:05 +02:00
Alexander Neumann 4c6b626db6 backend: Improve TestList 2017-09-18 13:18:42 +02:00
Alexander Neumann 835ba16c27 b2: Add pagination for List() 2017-09-18 12:13:35 +02:00
Alexander Neumann 3b6a580b32 backend: Make pagination for List configurable 2017-09-18 12:01:54 +02:00
Tobias Klein 9b38980ed9 fuse: typo 2017-09-17 17:39:28 +02:00
Alexander Neumann 649c536250 backend: Improve test for pagination in list 2017-09-17 11:36:45 +02:00
Alexander Neumann dd49e2b12d Azure: Fix List(), use pagination marker 2017-09-17 11:32:05 +02:00
Alexander Neumann f61dab1774 backend: Add test for List() 2017-09-17 11:09:16 +02:00
Alexander Neumann 40edf00182 gs: implement pagination 2017-09-17 11:08:51 +02:00
Alexander Neumann c35518a865 Azure/GS: Remove ReadDir() 2017-09-17 11:05:30 +02:00
Alexander Neumann c784a15aaa Merge pull request #1244 from restic/fix-swift-backend-tests
Ignore "not exist" errors for swift backend tests
2017-09-16 14:54:32 +02:00
Alexander Neumann ce180de9b8 Merge pull request #1243 from restic/improve-error-reporting
Improve error reporting
2017-09-16 14:54:30 +02:00
Alexander Neumann 2a1633621b Ignore "not exist" errors for swift backend tests 2017-09-16 13:59:55 +02:00
Alexander Neumann d4e994de7b Improve error reporting
This will print the error (including a stack trace) if available before
exiting.
2017-09-16 10:55:13 +02:00
Alexander Neumann a60e751217 Use .Equal() instead of == for time.Time
Closes #1238
2017-09-15 20:57:35 +02:00
Alexander Neumann 5bf2228596 local: Fix creating data dirs 2017-09-11 21:48:25 +02:00
Alexander Neumann 227b01395f local: Add test for open non-existing dir 2017-09-11 21:34:26 +02:00
Alexander Neumann 36eaa22ed0 Merge pull request #1205 from mungomat/backup_time
Backup time
2017-09-11 19:01:41 +02:00
Alexander Neumann 62df316356 Merge pull request #1194 from mungomat/bugfix_bucker_always
fix: bucker
2017-09-11 19:01:23 +02:00
Alexander Neumann 9fa909ccd6 Update golden files 2017-09-11 17:52:22 +02:00