Overview
Virtalis Reach comes with an automated back up system allowing an Administrator to restore to an earlier snapshot in the event of a disaster. We will install Velero to back up the state of your Kubernetes Cluster and uses a custom built solution which leverages Restic to back up the persistent data imported into Virtalis Reach.
You should consider creating regular backups of your buckets which hold the backed-up data in case of failure. This will bed one through your cloud provider or manually if you host your own bucket.
Alternatively, you can consider using your own backup solution. A good option is the Persistent Volume Snapshot which creates a snapshot of a persistent volume at a point in time. The biggest caveat is that they’re only supported on a number of platforms such as Azure and AWS.
If you opt in for a different solution to the one we provide, you have to be mindful of the fact that not all databases used by Virtalis Reach support live backups. This means that the databases have tobe taken offline before backing up.
The following databases in use by Virtalis Reach must be taken offline for the duration of the backup:
· Minio
· Neo4j
Variables and Commands
In this section, variables enclosed in<> arrows should be replaced with the appropriate values. For example:
becomes
Commands to execute in a shell are shown as code and each block of code is designed to be a single command that can becopy and pasted:
Set Up the Deployment Shell
Substitute the variable values and export them:
Navigate to the directory containing Reach installation files:
Installation
Creating a StorageLocation
Recommended:
Follow the “Create S3 Bucket” and “Set permissions for Velero” sections from the link below and ensure that you create the following 2 buckets under your s3 bucket:
· reach-restic
· reach-velero
https://github.com/vmware-tanzu/velero-plugin-for-aws#create-s3-bucket
Export the address and port of the bucket you have created:
Not recommended - create an s3 bucket on the same cluster, alongside Virtalis Reach:
Customize the persistence.size if the total size of your data exceeds 256gb and change the storage class REACH_SC if needed:
Export the address and port of the bucket you have created:
Set Up Variables
For the duration of this installation, you must navigate to the k8s folder that is downloadable by following the Virtalis ReachInstallation Guide.
Make scripts executable:
Export the following variables:
Export the address of the reach-restic bucket:
Optional configuration variables:
Velero Installation
The following steps will assume you named your Velero bucket “reach-velero”.
Add the VMware helm repository and update:
Install Velero:
Install the velero cli client:
Manually create a single backup to verify that the connection to the aws bucket is working:
Watch the status of the backup until it’s finished, this should show up as complete if everything was set up correctly:
Create a scheduled backup:
This schedule will run a backup every Saturday at 23:45PM.
Restic Integration
The custom restic integration uses Kubernetes jobs to mount the data, encrypt it, and send it to a bucket. Kubernetes Custom Resource Definitions are used to store the information about the resticre positories as well as any created backups.
The scheduled data backup runs on every Friday at 23:45PM by default. This can be modified by editing the cronSchedule field in all values.yaml files located in backup-restore/helmCharts/<release_name>/with the exception of common-lib.
All the performed backups are offline backups,therefore Virtalis Reach will be unavailable for that period as a number of databases have to be taken down.
Create an AWS bucket with the name“reach-restic” by following the same guide from the Velero section.
Replace the keys and create a secret containing the reach-restic bucket credentials:
If you instead opted in to deploy an s3 bucket on the same cluster, run this instead:
Export the address of the reach-restic bucket:
Run the installation:
Check if all the -init-repository- jobs have completed:
Query the list of repositories:
The output should look something like this with the status of all repositories showing as Initialized:
Once you are happy to move on, delete the completed job pods:
Trigger a manual backup:
After a while, all the -triggered-backup- jobs should show up as Completed:
Query the list of snapshots:
The output should look something like this with the status of all snapshots showing as Completed:
Once you are happy to move on, delete the completed job pods:
Triggering a Manual Backup
Consider scheduling system downtime and scaling down the ingress to prevent people from accessing the server during the backup procedure.
Note down the replica count for nginx before scaling it down:
Scale down the nginx ingress service:
Create a cluster resource level backup:
Check the status of the velero backup:
Create a database level backup:
Check the status of the database backup:
Once you are happy to move on, delete the completed job pods:
Restoring Data
Restoration Plan
Plan your restoration by gathering a list of the snapshot IDs you will be restoring from and export them.
Begin by querying the list of repositories:
Run a dry run of the restore script to gather a list of the variables you can export:
Sample output:
Query available snapshots or use the commands returned in the output above to query by specific repositories:
This should return a list of available snapshots:
It’s strongly advised to restore all the backed-up data using snapshots from the same day to avoid data corruption.
Note down the replica count for nginx before scaling it down:
Scale down the nginx ingress service to prevent people from accessing Virtalis Reach during the restoration process:
Run the Restore Script
Get list of backups created with velero:
Example:
Restore:
Unset exported restore id’s:
After a while, all the -triggered-restore- jobs should show up as Completed:
Once you are happy to move on, delete the completed job pods:
Watch and wait for all pods that are running to be Ready:
Scale back nginx:
Pruning Backups
Velero
Get list of backups created with velero:
Example:
Delete a single backup:
Example:
Restic
Get list of reach backups:
Example:
Prune backups based on a restic policy.
Learn more about the different policies here: https://restic.readthedocs.io/en/latest/060_forget.html#removing-snapshots-according-to-a-policy
Example:
Keep the latest snapshot and delete everything else:
After:
