Nullish's Logo

Clean My Digital House

50% Complete

As the sole steward of my digital footprint, I've recognized a deficiency in how I'm managing my personal files.

A few years ago I built a NAS out of spare parts, and have subsequently enjoyed the benefits of having a nice central storage solution (with disk redundancy) on my local network. However, in the typical standard of "3-2-1" backups (three copies across two forms of media, with one being offsite), my offsite backup situation needs improvement.

Ok, being more honest, it doesn't exist at all. I am (shamefully, inexcusably) between offsite backup solutions.

The reason for this is kind of dumb: I'm not actually sure what kind of a solution I need because I'm not sure how much data I actually have. In the last couple years I've made a focused effort to get my data and my eyeballs away from big tech. Consequently, I have a number of archives which I've carelessly extracted onto my NAS, and which I suspect amount to a great deal of duplicate data. I need to clean it up and eliminate that duplication; I need to sort out the remaining files and decide what I actually need to keep; and then I'll be in a position to pick an offsite backup strategy that is able to meet my needs.

This is the plan, in five steps:

  1. Consolidate and organize my most important documents
  2. Secure the above documents as needed
  3. Run some tool-assisted de-duplication
  4. Pick a cloud storage provider for backups
  5. Implement a strategy for offsite backups

I can immediately predict that step #3 is going to be rather drawn out and time-intensive, so this plan will be an ongoing process. The tense of the writing might jump around unceremoniously. I've done stuff, I'm doing stuff, I will do stuff. That's just how it goes with this kind of a project.

Maybe upon completion I'll come back and make some revisions, but I think these plans are more interesting if I'm able to iterate on the documenting (here), as I iterate on the execution.

Consolidate

Past tense.

I started this process by sorting out important documents. Tax returns, the contents of my "death binder" (to be written about another day), really important security stuff like the audit of sites I have logins to, 2FA backup codes, that sort of thing. Most of this was in fairly good shape.

I consider myself to be at generally financially literate, and definitely privacy and security-minded (at least compared to the average person), so most of these files were at semi-organized. It took a few hours to improve the situation, but overall this was not too bad, and this consolidation allowed me to proceed to the next step.

Secure

Past tense.

I decided that for these important, sensitive, but rarely changed and rarely accessed files, I wanted to implement file-level encryption on my NAS. Why? Just paranoid, I guess.

I don't currently expose my NAS to the internet, and I'm not worried about others in my household getting into this stuff, but in the next part of this process I'm planning to use some software assistance to help me sort out all my photo duplicates.

Now granted, the program I'm planning to try is widely used, open source, and highly likely to be trustworthy. However, you never really know with these things, and so I'd feel a bit better if my most important data was encrypted.

I looked into a few different options for this. As usual, PrivacyGuides.org was an excellent resource, and I looked closely into both Cryptomator and VeraCrypt. Both are compelling, and probably would have met my needs just fine.

However, for simplicity I decided to use Kleopatra, an excellent KDE application that comes standard in openSUSE Tumbleweed, my current distro of choice. It's a user-friendly front-end for GnuPG; as I'm sorting through my files, if I have a directory full of sensitive documents all I need to do is right click it, "Archive & Encrypt Folder", and provide a secure passphrase.

At this point, everything that's in need of extra security has been secured, and I'm moving on to the de-duplication process. This will probably take the longest out of all the steps in this .plan.

Deduplicate

Present tense.

I'm in the middle of some heavy duty deduplicating. Family photos and videos are the largest chunk of my data overall, and also the data that is in the worst shape in terms of being generally disorganized and duplicated. After some research, I decided to use the program Czkawa to help with this problem.

Using Czkawa is an ongoing process. There are a few different duplicate detection options, and the ones I've tried work very well. Exact duplicates can be found either by metadata (e.g. file name + size) or by a hash of the image itself, which is extremely effective but a bit slow for large photo libraries. Similar images (with a slider for "how similar") can be identified too. I've only played with this feature a little bit.

There's a music duplicate detection feature that I'm looking forward to trying out as well.

Overall, after taking a few passes with different options, I've been able to delete a couple hundred GB of duplicate files, and I think there's a little more juice left in this grapefruit.

Cloud Storage research

Future tense.

After all the time I spent getting away from big tech, I'm not thrilled with the idea of putting my data back into the Cloud. However, I would be truly devastated if I lost some of this stuff, and so I need to be responsible. I also intend to fully client-side encrypt anything that is backed up into Cloud storage, which largely mitigates the security and privacy aspect, and the trust issues I have with most internet companies these days.

Once I get to this step of the plan, I should have a pretty good idea of how much data I actually have, and how much of it is important to backup offsite regularly or ever.

Going into this, my leading contenders are:

Offsite Backup Implementation

Future tense.

Once I figure out what cloud storage provider I want to go with, I have to actually, you know, set it up in a satisfactory way. This is effectively the last step of this process: implement and test the backup solution.

By the time I get here, I'll have sorted out my data management issues, and will have a robust 3-2-1 backup solution in place.