Aka ‘DARK WEB HACKER COST ME $1600 SHOCK HORROR !’
After I set up my Jekyll site and uploaded the content to Amazon S3
using ‘s3website’, I remember thinking ‘I must re-read that section
about securing the configuration file with AWS credentials in plain
If the source code of your website is publicly available, ensure
that the s3website.yml file is in the list of ignored files. For git
users this means that the file .gitignore should mention the
So, I duly added ‘s3website.yml’ to .gitignore and pushed to GitHub. I
wasn’t sure whether this file exclusion only took effect from now so I
checked if the file was still in the repository. Unsurprisingly, it
was but GitHub provided detailed instructions on how to resolve
So, job done and as my AWS credentials were only exposed for 57
minutes, no harm done.
I went for lunch and returned to a voicemail from Amazon customer
services asking me to contact them urgently about a ‘security
issue’. I also had an email and an AWS support case titled ‘Your AWS
account is compromised’ describing, in detail, what corrective action
I should take to promptly resolve the situation.
My heart sank a little as I followed the instructions and examined the
list of EC instances running. ‘Hmm, that’s strange, I don’t remember
setting up 10 instances called “Ghost” in every region…’
I quickly terminated each instance and went to check my billing
information. Phew. Usage for today was $0.00. Then I remembered a
possible reason; in the dim and distant past, I experimented with a
pre-built EC instance running Ghost. Maybe that was the reason but,
deep down, I knew this wasn’t the case as they had all been started
today and I don’t think ‘ghost’ was referring to the blogging
Next I had to lockdown my AWS setup. First, although I already had a
user account, I deleted the access keys associated with the ‘root’
account and changed my Amazon password. I also deleted the existing
user and group, re-created them with new keys and followed the
guidelines and best practice recommendations in the Identity and
Access Management user guide.
Then I enabled multi-factor authentication (MFA) for the AWS root
account. This means that access is secured by the requirement to enter
a 6 digit code from my mobile phone using Google Authenticator.
The following morning, I logged onto AWS and checked my bill. In a
short period of time, the imposter had clocked up $1600 worth of
charges despite Amazon locking down the account once they detected the
compromise. I contacted Amazon customer support who said they would
refund the excess charges due to this ‘mishap’ and would notify me
once this was ‘approved’. A little ambiguous but hopefully, I will get
reimbursed although strictly speaking, this ‘mishap’ was down to my
Finally, I did what I should have done in the first place and move the
s3website configuration file elsewhere completely outside of the
project directory and specify the location when sync’ing the site.
$ s3_website push --config-dir ~/.s3_website
Otherwise, I can anticipate that if I change themes or platforms, I
will repeat this idiotic error and Amazon may not be as understanding
Now, that it looks like the episode might be over, I am struck at how
quickly Amazon detected the appearance of my AWS keys on GitHub. I
presume they have a automated bot looking for this type of data so
maybe it’s not uncommon. Secondly, what benefit did the hacker gain ?
He ran 40 EC instances for a while before being detected and
shutdown. Why ? Just because he could ? In a way, I wish I’d more time
to investigate precisely what was running on the instances.