Cleaning Up the Free Tier After AWS Playtime is Over

Photo by Mick Haupt on Unsplash

I just spent a happy year in the AWS free tier, experimenting with new things, building POCs and trying things out. In the process I made a mess. Now I have to clean it up.

The Problem

All happy messes are alike, but every unhappy mess is unhappy in its own way. My own mess had the following characteristics:

1. Some of it was valuable
2. Some of it wasn’t
3. There was some resource tagging but it was not applied consistently or thoroughly
4. The principle of least privilege had largely been jettisoned in favor of the principle of just opening things up as wide as possible in order to get something to work

The clean up was prompted by three things:

1. Some of the completed workloads could be used in semi-permanent demonstrations and potentially end up on production projects. I needed to isolate and protect all that stuff
2. The AWS free tier had come to a close and I needed to ensure that no costs were being run up by anything that wasn’t essential. I also needed to start deleting things, and the delete button is probably the scariest thing in all of IT.
3. The usual long-standing security niggles that keep me awake at night.

Photo by Anton on Unsplash

The Solution

The solution begins with this basic requirement: sort out the valuable stuff from the rest, and protect it.

This means that we can’t really look directly to AWS Organizations for the solution. Organizations is an oft-touted solution for keeping concerns separate. But my problem was prior to that one: finding a way to safely separate things in the first place.

Photo by Simon Maage on Unsplash

That is quite a big issue. It depends on what kind of resources you have already created, what your business aims are and so on. So what I write here won’t be a didactic ‘solves all’ template kind of answer. I am suspicious, to be honest, of any answer that pretends to that. What I am portraying instead is more something born of my own experience that hopefully points a way out of the forest. It is written both that it may help others and that others may help it, with comments and suggestions from their own experience.

So let’s talk tags.

Resource Tagging

Photo by Alexander Popov on Unsplash

Tags are the most basic way to label resources. The process is fundamental to sorting out what to keep from what to dump. It is also fundamental to explaining why. What tags you decide to use and what values you give them will be foundational to how you govern resources over time. It’s important and foundational.

Perhaps you have been pro-active and worked with tags from an early stage. This is to be commended, but if your experience is like mine, it may transpire that tags and tag values have not necessarily been consistently applied. A clean up still needs to be done.

What Tags to Use

AWS broadly recommend five general tags in their documentation:

• Environment — production, test, development etc.…
• Application ID — to group resources by application.
• Automation Opt-In/Opt-Out — this indicates whether a resource should be included in an automated activity.
• Cost Center/Business Unit — to group resources by business and accounting category.
• Owner — who is responsible for the resource.

This is fine in so far as it goes but it assumes that work is being done exclusively on applications and that automation will feature also. One key feature of the free tier, perhaps its very reason for being, is experimentation and variety. So it is not likely to simply be restricted to working applications and automation, especially if you are doing it right. It should rather be filled with all kinds of trial stuff, stumps, experiment and sketches. With that in mind, I cut down these five tags and started my free tier journey armed with just three:

• A tag for the principle: Naming whoever is in charge of that resource
• A tag for the collective use classification: This is basically something like a project name tag — one that allows you to group resources in a way that makes sense to yourself and your business — be that by application, business case or something else.
• A tag for environment: Extended beyond the standard ‘dev’, ‘qa’ etc… to include things like ‘PocInUse’ and ‘ForDeletion’. These will be discussed further below

For example, an EC2 instance in my set up might be tagged:

UserName: Frank

ProjectName: Data Filtering POC

Environment: For Deletion

Photo by Mike Szczepanski on Unsplash

Armed with this tagging strategy, I took three steps in cleaning up:

1. Sort out all resources into tag-based resource groups
2. Adjust their tags accordingly
3. Deal with the ‘weird’ resources (these will become apparent soon)

Let’s take each step in turn.

Sort Out Resources into Tag-Based Resource Groups

In the console GUI, select the ‘AWS Resource Groups’ (& Tag Editor) service. This will, as the name suggests, allow you to view resources by tag, and will helpfully also show lists pre-existing tags and values, in addition to displaying resources that don’t have a chosen tag, or that do have one, but without a value ascribed to it.

As with most AWS resources, you will need to work region by region.

Select ‘Create a New Resource Group’

Stick with ‘Tag based’ for now. ‘CloudFormation stack based’ is, as the name suggests, useful for checking on resources that were created through CloudFormation templates. I think it is fair to say that most users starting out in the free tier will probably not jump straight into using CloudFormation from the outset. So this is really another topic for another article.

The resource grouping criteria are basically a way of filtering by AWS service. If you want to narrow down, for example, to just S3 buckets, you can. I think it is advisable initially to keep this wide open so just select for ‘All supported resource types’

When you click into the tag filter selector (either for key or for value) it will display a dropdown of all your currently existing tags under that choice. This can be a useful reminder of what kind of tags you have used in the past and is a quick way to find resources you previously tagged.

When you select a tag, it should be noted that there are some special values AWS allows:

1. (not tagged)
2. (empty value)

These are very useful for quickly finding resources that have not been tagged properly. Depending on how rigorously you previously approached the free tier, these may be all of your resources or just some of them. For example, using this filter you can immediately list all resources not tagged with, e.g. an ‘Environment’ tag. Then you can go through that list and tag accordingly.

Tagging accordingly

There are two tag values that I find very useful to add in an initial clean up:

Environment: InUse

and

Environment: ForDeletion

The ‘InUse’ value is for everything to be kept and protected — either for demonstrations or potential future products. These are the first resources I look for when I start checking for anything with ‘Environment: (not tagged)’ ascribed to it.

Photo by Paweł Czerwiński on Unsplash

The job now is to basically try to remind yourself which resources belong under which groups and add those tags for yourself. You may need to hunt down, for example, which Lambda function is invoked by which API Gateway and which corresponding bucket is updated when that Lambda runs. And so on. This takes time — the kind of technical debt time that runs slower than normal time, and is the price paid for not being completely tidy in the first place.

There are two ways to tag the resources listed in the results of any search. The easy way is to simply click on the resource link and tag it directly. This is not always possible, though, as I will show below.

A second way is to click into the check boxes next to any resource you want to work with and click the ‘Manage tags of selected resources’ button.

Doing this brings up an ‘Edit tags’ dialogue which enables the addition and editing of tags directly through the resource tag editor. This can be useful for some resources that you cannot tag directly as will be shown for CloudFormation tags below.

Once ‘InUse’ resources are tagged, it is possible to tag most of the rest with the ‘ForDeletion’ tag. As I am sure everyone knows, it is never wise delete anything without giving the action some prior time and thought. Tagging resources as ‘ForDeletion’ allows for a revisit them after a period of time has expired. Then it will be possible to examine when and how resources were last used, by checking CloudWatch and CloudTrail logs, for example. If the resources have been quiet, this may be the time to archive and delete them.

Some Odd Ones

Some services don’t fit so neatly into tagging as a resource clarification strategy. I will list a couple that I have found here. I am sure there are (a lot) more but an exhaustive list would require a deep foundational dive into all AWS resources and I do need to sleep from time to time.

IAM Users and Roles

While IAM Users and Roles can be tagged, they are not compatible with the Resource Groups Tag Editor. This means, in short, that they won’t show up as part of the nice, tidy resource organization under AWS Resource Groups just outlined here.

That is a pity for a few reasons:

1. The first casualty of the free tier is roles. You are likely going to end up with a lot of them, all done ad hoc, just to get something working in the past. Now they are all sitting there, and need to be sorted, but you can’t list them in the tag editor.
2. Knowing your roles is really important. The role is a vital part of any any resource group, and it is key to resource security. So these really should be seen as part of that group, and not as a separate entity.
3. Keeping security open can sometimes be part of just getting something new to work. Doing so means reducing potential bug sources. Of course, as soon as a project is confirmed as running, security should be immediately tightened. But … well, let’s just admit that, as much as we may wish it did, that doesn’t always happen in the experimental arena. Finding relevant roles and applying the principle of least privilege should be essential to the general clean up project.

Perhaps this could be added as a future feature.

CloudFormation Stacks

Tags can be added to a CloudFormation Stack when it is created in the GUI, but not afterwards, when it is edited. I am talking here about external tags for the stack itself, not its internal resources. That’s bad news for our endeavor as we would ideally like to tag old CloudFormation stacks so that we can add them to resource groups (even those that they created). But, as hinted at above, it turns out that it is possible to add external tags to a CloudFormation stack directly through the tag editor.

Click on tag editor, choose AWS::CloudFormation:Stack as your resource type, then run a filtered search as above. You will see a new button ‘Manage tags of selected resources’. Click that and you are now able to add external tags to previously created CloudFormation stacks.

One proviso, and AWS themselves clearly label this, it is not best practice to add external tags to nested stacks. But if you you have a lot of resources and want to tag the main stack that created them so it is part of your resource group, this is one way to do it.

Conclusion

So there you are: some things I have discovered when sorting the virtual wheat from the holographic chaff on AWS. As I said, this piece is not meant to be either definitive or exhaustive. It merely points a way towards a possible clean up and beyond that maybe a security and governance strategy built on that foundation.