Data is kinda important

One of those things that I always think 'oh I'll get to that later' and then it takes me way too long to do is making sure that my important data is backed up. I suck at this. I think about it, think I should do that, and then think about it some more. Let's ramble about it.

Putting the data somewhere in the first place

I've leaned heavily into consolidating a lot of my old cronjobs and bash scripts into a centralised place where I can manage things and see if it's gone wrong. Although quite a complex single place to put things, Kubernetes is where this has ended up as this is where most of the workloads and services live now. CronJobs run in Kubernetes and integrate with all of the other tooling to monitor services already and it's all on the existing compute. Alerts are set in Grafana for if jobs fail and all the hacky scripts can be stored in Git if my cluster gets nuked from existence again (oops).

One of the more recent jobs I set up (that I really should have done sooner) is backing up my databases to an S3 store. A Kubernetes CronJob spins up, connects to my local Postgres databases, does a full dump of these and exports them encrypted to an s3 compatible bucket. This happens nightly and backups older than 2 weeks are automatically cleaned up there is hope that I realise things are broken before then. The motivation came to do this after moving around 200GB of photos out from Google Photos and into Immich. The photos themselves are backed up but reimporting that volume of data into Immich in case of data loss sounds painful and a lot of CPU cycles.

Backing up my NAS shares

I run TrueNas for my NAS requirements, I currently have 2x12TB drives in a mirror that is split into a few different datasets depending on the use case. Stuff that isn't super important but I want a local copy of does not need backing up, this data is non important to me. I have two main datasets I care about, my 'Documents' and an archive of my twitch streams. Documents is in quotes because it encompasses a few different things but overall it's files I care about and want to keep safe.

TrueNas has a useful page called data protection and within this, I can set up Cloud Sync tasks to back my data up to an S3 compatible target, in my case, Scaleway. I do nightly backups of my important documents and photos and a weekly archive of my twitch streams (this is low volume and any data lost within that 7 day period I can retrieve again from source). Most importantly is making sure that I get notified if these backups fail, I don't often visit the TrueNAS UI and therefore, I need a heads up if things ever fail due to credentials expiring or some other issue. This is done via webhooks to services and then a notification fires off to go fix things.

Every piece of data that leaves my servers is encrypted, there is no need to directly access the data in the buckets and if required, it's possible to browse them by setting up rclone locally with a crypt backend. This seems like common sense to me, even though I aim to use European cloud providers for data sovereignty, this data is still my own.

Take this as a friendly reminder to make sure your data is backed up and save your future self from losing data, is happens to everyone at least once.