Let’s say you have access to a remote machine and use it to copy backups occasionally, eg with rsync. Your local machine has credentials stored that allow write access on the remote machine, however if the local account was compromised that could also allow access to the remote machine and the data stored there.
How can you grant access to an account to write remotely, but also protect the data from this account? One possibility could be to change the permissions on the data after it is copied to prevent deletion/interference, although I’m just making this up. Is there a standard practise for this?
The suggestion I have heard is to have the remote machine connect to the machine on a schedule and pull the backups onto itself. Then your local machine doesn’t have direct access to the backups, making it harder to compromise the backups if hacked. But this also assumes the backup machine is locked down and isolated so it is lower risk than the local machine.
Do pull backups instead of push backups: Backup server connects to local machine.
That’s an interesting idea
What’s the rationale for this? Genuinely curious.
The reasoning is that your backup server should be more secure than production. Production has to have a bunch of stuff open in order to be useful and convenient. The backup server does not. It can be basically fully locked down.
To add - by doing pulls the backup server uses different credentials to run than the credentials used to perform pulls.
Backup server has it’s own credentials database, machines being backed up have their own database. Backup service in backup server uses appropriate credentials from machine being backed up to access the data there (shares, etc). So credentials from compromised machine are unrelated to credentials for backup server.
And if backups are done properly (full on a schedule, daily incrementals, or something similar) you should be able to revert to a known-good state with minimal data loss.
If the main site gets compromised the credentials there must be considered lost and known to che attackers.
with a pull backup that’s not an issue because the main site has no access to the remote system; it is a process on the remote site that has credentials to access the main site and not the other way around.
the remote system may
receiveretrieve a compromised copy of the data, but the attacker cannot tamper with previous backups so recovery is still possible.That makes sense. I use NFS, so there are other controls for security because “offsite” is another building on my property, but still in the same pool of subnets…
This is the main reason I had in my head about pull backups. Thanks for the explanation.
Why downvote this?
Encrypt before send, and if you want to have protection against deletions of the data have a cold backup offline other than during the copy.
Restic has quite a good solution for this: https://restic.readthedocs.io/en/stable/060_forget.html#security-considerations-in-append-only-mode
Funny thing is that blockchains are actually good for this type of thing. Too bad crypto bros got a hold of it.
Just a small sidenote: If you do not trust your local machine you should think about why and how to change that.
Well I think she amount of caution is still appropriate no matter how much you manage to secure the local machine
Of course, of course :)
A system like proxmox backup server can do this scurely. There you can create a user that can only add new backups and read the existing ones, but cannot delete any or read anything else on the remote host.
Otherwise if you only care to protect the remote machine, then something like an ssh chroot jail would also work.
Append only mode.
Scheduled snapshots (btrfs or zfs). If the compromised account deletes or modifies files, they’re still there in the past snapshots
Filesystem-level snapshots are quite space-efficient because they don’t make copies of all the files or even whole files; just the blocks that changed.
I think you could do it somewhat like hetzner does for their storage boxes. You get an account that has read and write access to a directory and nothing outside. The accound can only run a limited set of commands, like ls, cat, nano, rsync etc. but has no access to commands that modify the filesystem.
Then you can use a copy on write fs like btrfs and make scheduled staggered snapshots.
I usually do 1x per year, 1x per month of current year, 4 per week of current montg, 7 per day in current week.
I have no clue what they use to limit the user accounts like that btw. but maybe that gives you a new jump off point for further research.
Nano and rsync are 100% designed to modify the filesystem. But yes the idea is correct.
Same with got over ssh, you restrict the connection to the got shell that can only do the things you want.
Are they? I thought they only write/modify/delete data to the fs, not change the fs itself.
Yeah precice phrasing ia hard sometime. I was refering to delete/modify of files as “changes to the fs”. Not sure how changing the actuall fs would be relevant to the backup question.
OP needs a restricted shell that can take backup data and write it to disk but not be able to modify anything that is already there. Nano and rsync can both do that.
OP asked:
How can you grant access to an account to write remotely, but also protect the data from this account?
So I was thinking that the account should not be able to delete the filesystem in an unrecoverable way. Like overriding the current fs with random data or an encrypted fs and filling it etc.
Like I said on a Hetzner storage box, multiple users get access to the same system, but each one only has file editing commands, not fs editing and they can only access their assigned directory. So if the system does scheduled snapshots (outside of that user’s scope of access) there is no way for a user to delete the files beyond recoverability. (no matter if their own files or other users files).
The user can still delete their own data. But because the fs is cow with snapshots (like btrfs) and they can not touch that, the data can be recovered easily.
I have solved that by giving the distant machine the credentials to connect to the local machine. And the distant machine can’t be accessed from the outside.
Yes that sounds like pull backups, which is worth exploring
Append only, like others are saying.
What you are talking about is Immutability and an append only backup. The s3 file system and some others has Immutability built in. Not all backups can do append only.
Yep offline backups are useful, although it does require remembering & making the effort to do it each time
Certainly sounds relevant, although overall it is quite a different approach than in currently using
For doing snapshots did that means the local system identifies the changes? Or it all gets copied each time?
