Convenience + security: best practices for AWS access key management

In this blog post I am going to talk about several security best practices, particularly for configuring AWS Access Keys. Some of these practices are based on a project that we inherited which was compromised by hackers. Best practices are often learned from mistakes; and when the mistakes are someone else's, so much the better!

Tracking down an exposed access key issue

Several months ago, we got an alert from the AWS Simple Email Service (SES), Amazon's cloud-based email service — SES was shutting down our client's outbound email service due to a large number of spam complaints. This particular client doesn’t use SES for bulk mailing – it typically sends no more than 50 or so mails a day – and so this immediately seemed odd.

I contacted SES support and reviewed the logs, to find out that our customer’s server had sent about 50,000 mails in a 24-hour period. Technically, SES can handle that load, of course – but user reports led them to shut down mail transmission. Clearly, we had been hacked.

My first thought was that bad actors had somehow gotten their hands on AWS account credentials. But after a careful scan of the AWS audit logs, we saw that access keys had somehow been compromised, and the hacker was using a script to hijack SES for their spam runs. The access keys were authorized to have almost complete control of the AWS account, so the door was wide open.

Upon further investigation, we found that our customer (who has management authority over the AWS account) had stored the keys in a public GitHub directory and left the .git directory exposed on the server. It is important to note that even if you remove a file from the current branch, the file is still visible in the history that Git maintains! These directories are constantly scanned by the public (including GitHub’s own AI), so this was a glaring invitation for mischief.

The fix was easy enough. We expired the compromised keys and then created new ones, with limited permissions specific only to the purposes for which we use associated command-line scripts. We cleaned up the IAM user accounts and checked for inter-regional service issues. Problem solved.

As a result of this process, we firmed up some new best practices for using AWS access keys, and basic identity management in general. We think these steps strike a solid balance between the convenience of command-line control and the need to lock down your systems.

Real world vs. ideal world access key best practices

Before I go on, I should stress that the best practices I am going to discuss here are intended for high-trust users. Access keys are inherently risky – they are entry points into one of your business’ most critical assets.

Amazon suggests using Identity and Access Management (IAM) user accounts and short-term credentials for personal access to AWS management features. It also advocates for using a power shell or integrating the command-line with IAM manager, as opposed to creating long-term access keys, which it suggests be limited to genuinely programmatic use (e.g. - an automated maintenance script that runs at 3 a.m. each day).

But, lots of admins use access keys for quick command-line tasks. In fact, it’s one of the use cases included in the access keys generator (see our sidebar for a walkthrough). And authorized users can make their own keys from within IAM.

So, our list of best practices here reflect real-world use, not dogmatic adherence to Amazon’s suggestions. Much of the wisdom is the same, and the major emphasis is identical – store access keys, particularly the signature key of the pair, where bad actors simply cannot find them. Everything else follows.

Six best practices for access key and ID security

So, with that disclaimer out of the way, here are Mugo Web’s best practices for using access keys and securing your AWS admin environment.

1. Base access keys on accounts with limited permissions

The hackers who used our client’s SES service to send spam were able to do so because the access key pair they stole was authorized to have essentially complete control over the entire AWS account. When we researched the issue, we created a new user with extremely limited permissions – basically, all it can do is start and stop existing dev/staging instances, a common cost-savings tactic with Amazon’s pay-as-you-go pricing model. Keys created for this user have these same permissions, and no more.

This is the easiest, best approach for limiting the damage that can be done with compromised access keys. You simply aren’t going to need a script that can do everything Root can. Create multiple, purpose-built user accounts if need be to ensure that access keys present the lowest risk threshold possible. We do a lot of work on AWS services from local command lines without any reduction in convenience.

2. Do not store keys in public, insecure directories

This is how you get into trouble, and it falls mostly on devs and admins to not make this mistake. There are some technical answers: AWS pushes its add-on Secrets Manager service as a lock box for API keys, and browser plug-ins and other tools for scrub GitHub and other directories for possibly sensitive information (this 47-minute video from ExamPro is worth a watch). This “keep it secret” mantra also extends to not including keys (particularly the “secret” signature key) in application code.

Again, this boils down to “just don’t do it”. As I said earlier, access keys should be limited to high-trust team members who get the importance of keeping them secure.

3. Rotate access keys

Amazon recommends this best practice right there in the access key generator. Obviously, rotating keys requires some manual overhead – by design, they do not expire, so you need to schedule a manual task for going into IAM and:

• Expiring all access keys after X period
• Creating new keys to replace them
• Securely distributing new keys to affected parties
• Updating any scripts that use the expired keys

For dev who are using access keys for command line access, the updating part of the process will work itself out. But if you are using keys for automated housekeeping, you may want to bump out the rotation cycle to six months or a little more. It’s all a risk management calculation.

4. Set up CloudWatch alarms for red flags

We used AWS CloudWatch's anomaly detection service to send warning messages if it sees unexpected behaviors. For our use-case customer, these were high volume of mail sent over a time period, new AWS user creation, and some data access points. These red flags can be essentially any behavior or logging pattern you define -- the toolset is incredibly flexible. But, as we said earlier, limiting the actual permissions assigned to your access keys is the best way to mitigate harm. Tailor your alerts to the possible mischief your keys could be used for; CloudWatch and the various AWS services have built-in triggers for wildly anomalous events.

5. Review you IAM users and policies regularly

This is a no-brainer, but review your IAM users about once a month or so, to ensure that a bad actor has not gotten in there and created dummy accounts that can be used for evil.  

6. Turn on AWS multi-factor authentication

This final best practice doesn’t apply directly to access keys, but it’s a common-sense security measure that, candidly, is coming a bit late to AWS. As you’ll recall, my first thought when I learned that our customer had been hacked was that someone simply got their hands on a ID/Password combo. That seemed like the most obvious scenario, since AWS currently does not require multi-factor authentication (MFA) to log into its management console.

Which is kind of a surprise -- even office productivity suites have been pushing MFA for a few years now. Amazon is moving in this direction; in the next few month, MFA will be required for Root logins, with that requirement rolling out to other roles later.

Turning on MFA is not as turn-key as you might expect (you’ll find that’s true of a lot of AWS features).  But it’s a step you should go ahead and implement for all users. The default setting for AWS MFA is to allow users to “remember me” on a specific device. You may want to consider forcing authentication on every log-in if you work in a high-compliance environment.

Manage access keys smartly to avoid security headaches

As you can see, managing AWS access keys basically boils down to common sense. They are potentially quite risky if they are compromised, but are perfectly secure if handled with a little care. I hope our best practices help you map out a plan for using access keys for repetitive AWS management tasks.

How to create AWS Keys

AWS handles programmatic and command-line access through a paired key system based on its own authentication system.

Technically speaking, the “access key” is the account identifier that appears in logs and other semi-public areas. The “secret key” is effectively the signature for the account, and absolutely should never be exposed publicly. If a user forgets a secret key, they have to generate a new one – similar to most password management systems.

Creating access keys is a fairly straightforward process. We will step through the process as the Root account, although individual users can create their own access keys if they have been set to programmatic access by an IAM admin.

1. Create an AWS user account that has only the specific permissions you will need for the script you plan to run. In our example, we are just going to start/stop dev instances, so that’s the scope of the user’s authority. If more than one human admin is going to execute the same programmatic function, create a specific user account per person: e.g., CarlosStartStop, ErnestoStartStop, and so forth. This facilitates precise security audits.

2. Navigate to the IAM dashboard and click Users in the left-hand pane.

3. Select the user for which you want to create the key.

4. In the “User Details” page, click the Security Credentials tab.

5. Click Create Access Key. In the Access Keys section, you will see any associated keys that have already been created.

6. Select the Command Line Interface (CLI) option

7. Add a description tag. In this case, we will use “start/stop instance for username”

IAM will generate and show you the paired keys. You can also choose to download the keys, as a CSV file.