Vault Backup fails if secret has been deleted

Problem

The vault backups fail for Anyware Manager if the secret data is deleted.

To Verify:

• Install Anyware Manager
• Complete the initial login and create a deployment.
• Create a Deployment Service account and then delete it (this is what might trigger the bug).
• Run backup, copy backup and key to a preferred location.
• Uninstall K3S and Anyware Manager.
• Reinstall
• Restore and make sure you can log in with your credentials.
• Create a CASM 22.04 Backup and make sure you can restore it to AWM 22.09.

Cause

The issue happens because when the DELETE API is called on a secret. It deletes the data but it won't delete the metadata which confuses the backup script (can't find data for a secret). In order to remove the secret completely you need to delete the metadata as well. Since deletion is risky, the easiest thing to do is add dummy data back to the secrets.

Resolution

If the vault backup fails, perform the following steps on the Anyware Manager instance:

1. Get the vault token by running. NOTE: This data is sensitive so it is recommended to perform this task with screen share turned off. 

   /usr/local/bin/kubectl get secret vault-secret -o jsonpath={.data.roottoken} | base64 -d 
   

2. Exec into the container.

   /usr/local/bin/kubectl exec -it deploy/vault -- sh

3. Log into the vault, it will prompt for the token from step 1.

   # export VAULT_SKIP_VERIFY=1
   # vault login

4. List the secrets in the Vault.

   # vault kv list secret
   Keys
   ----
   62eaf868d29f2435771dec8c
   admin-62eaf851d29f24acdc1dec85
   key62eaf87cd29f247d481deca6
   secret-management-service-health 

5. Check each secret to ensure the data has been deleted. NOTE: This data is sensitive so it is recommended to perform this task with screen share turned off. The output for a deleted secret is as follows:

   # vault kv get -format=json secret/key62eaf87cd29f247d481deca6 
   {
     "request_id": "c1b06ca8-2a15-403c-fbe8-2c3f36aeff87",
     "lease_id": "",
     "lease_duration": 0,
     "renewable": false,
     "data": {
       "data": null,
       "metadata": {
         "created_time": "2022-08-03T22:36:49.794576266Z",
         "deletion_time": "2022-08-03T22:38:03.709024273Z",
         "destroyed": false,
         "version": 1
       }
     },
     "warnings": null
   } 

6. Note that the {"data": {"data": null}}. Insert some dummy data and verify.

   # vault kv put secret/key62eaf87cd29f247d481deca6 dummy=value
   Key              Value
   ---              -----
   created_time     2022-08-03T23:23:57.467904981Z
   deletion_time    n/a
   destroyed        false
   version          1

   # vault kv get secret/key62eaf87cd29f247d481deca6
   ====== Metadata ======
   Key              Value
   ---              -----
   created_time     2022-08-03T23:23:57.467904981Z
   deletion_time    n/a
   destroyed        false
   version          1==== Data ====
   Key     Value
   ---     -----
   test    test

7. Repeat all the steps as needed and perform a backup once the issue is resolved.