Nov 12, 2018

Deleting amazon glacier vault (crash course in AWS/Glacier/CLI)


Part 1


WOW! Who would have thought it to be so painful!

Some background, feel free to skip:
Problem is Glacier UI doesn't support deleting files. Only CLI/SDK are supported.

The problem is, AWS charges for transfer to Glacies, and maintenance. So an entire qnap full of pictures sent to Glacier is ~$50 per months, plus I would guess a few $100 to restore. The plus here is setting up glacier backup to QNAP was pretty damn easy.

Initially when I compared Glacier to other file storage, it looks cheaper and safer, but at current pricing model it's very poorly suited to a bunch of small files. At 1.6 TB of pictures, I ended up with 800'000 files (aka archives) and my transfer bill became $40+ per months with storage bill at $6.20 per month. I had 6 more TB to go.

------- Start -------
Been a while since I logged into AWS console. First, had to create a new IAM user & Give him Glacier full control and S3 full control[1]. Expert advice: make a backup, but keep this tab open

Then, I installed powershell tools, and found out they dont support Glacier. Don't bother installing powershell tools

Next, install developer tools. Turns out they are wrong and windows 10 blocked whatever they had to do, so it didn't work anway. (this is about an hour in at this point). Install the next set of dev tools, and looks like it worked (AWSCLI64PY3.msi)[2]

get aws creds configured with aws configure

Dick around with the aws help glacier and end up nowhere. I can't delete vault cause it has items inside. Cant delete items, because you don't know their IDs and it's poorly documented anyway. There are articles going back to 2013 with people suffering.

You can use linux shorthand so that's useful aws glacier describe-vault --va qnap --a -

It looks like you can get the full inventory, mark all archives for deletion, and then feed it back into delete-archive command.

The command in the helpfile doesn't work, so you have to massage it in the most awkward way with double quotes:
aws glacier initiate-job --acc - --va qnap --job-parameters '{""Type"": ""inventory-retrieval""}'

Mind you, this is probably nearing two hours at this point. Looks like the command takes a long time to run, so i went off watching Westworld. Gonna resume tomorrow.

aws glacier list-jobs --ac - --va qnap

------------------------

Part 2


I finally gave up, and went with https://github.com/leeroybrun/glacier-vault-remove

In order to not contaminate my home box I used vagrant and virtual box to standup some flavor of linux - I think and then hacked inside that. It was pretty quick to get going.

Main point is you need Python 3 (more info here: https://github.com/leeroybrun/glacier-vault-remove/issues/29)


Links:
1. https://console.aws.amazon.com/iam
1h. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html

2. https://docs.aws.amazon.com/acm-pca/latest/userguide/PcaInstallCLI.html