How to search for Open Amazon s3 Buckets and their contents — https://buckets.grayhatwarfare.com
Intro
TLTR — This is the story of why I created https://buckets.grayhatwarfare.com/ a free tool that lists open s3 buckets and helps you search for interesting files.
For an intro on what Amazon open buckets is you can read this https://blog.rapid7.com/2013/03/27/open-s3-buckets/ . In essence, many files are publicly accessible, some by design, some by incompetence of the admins. These files sometimes include very sensitive data. https://github.com/nagwww/s3-leaks has a list of the biggest leaks recorded.
Since this was exposed, many projects have been created that can enumerate s3 buckets:
- https://github.com/jordanpotti/AWSBucketDump — takes an argument a wordlist and can check if buckets with that name exist. Also it can download the files from the first page of results.
- https://buckhacker.com/ — The original buckhacker as I am informed by him personally. Medium buckethacker and twitter buckethacker is the real, below is copycat.
- https://www.thebuckhacker.com/ — Search engine for some buckets. It include (according to their posts) ~100 000 buckets, and they index the first page of each — copy of buckhacker.com
- https://github.com/sa7mon/S3Scanner —
- https://github.com/kromtech/s3-inspector — Uses amazon keys to check if some of you buckets are open.
All these tools/projects have some common problems:
- The real problem (at least for me) is where to find the list to bruteforce for buckets, not actually doing the bruteforce.
- All tools/projects only scan the first page for results.
- thebuckhacker.com includes uninteresting files and useful results tend to be lost in the noise. Also the first 1000 results of each bucket is fairly limited.
- The process is slow and not productive. Its not very useful for pentesters to run a tool to run for days, save the exports somewhere and then grep them whenever they want to search for something. What is better is a useful tool in front of a large database.
buckets.grayhatwarfare.com
And like that I have created https://buckets.grayhatwarfare.com/. I took ideas from the tools/projects above, but I mostly rewrote them myself, and ran them on my infrastructure. The project’s features are:
- It is a searchable database of open buckets.
- I have included up to million results of each bucket. (In the future might be more).
- I removed all uninteresting (in my opinion) files like images. Most images names are auto-generated.
- I currently have ~180.000.000 files. In I included all images that number would go up to a few billion, which is a completely different system.
- As of today, 70 000 buckets are listed (not all of them have “interesting” files)
- Full text search with binary logic (can search for keywords and also stopwords)
- I include the list of the buckets.
- The user can browse the contents of the bucket.
- Excluded a lot of other things that are not interesting like cloudwatch logs.
- Found a solution the problem on how to generate possible names for buckets. My process reviles some hundreds new buckets per day.
- I have mostly automated the process.
Why create this ?
Although I consider my self a software engineer, I was always fascinated with security. I have on many occasions created multiple tools that mass scan for vulnerabilities, just to see if is possible. Although I created the tools, I never publish anything it always felt like I was wasting my time. grayhatwarfare.com is my attempt to present some of my work, even if it is anonymous.
The project is currently free and running on servers paid by me. There are some limitations in place to protect resources, but otherwise pentesters can use this on their daily tasks.
Whats to come in grayhatwarfare.com
Lots of cool things, if I have the time:
- Subdomains pointing to expired buckets which can lead to something like this: https://www.reddit.com/r/netsec/comments/8t0pb2/how_i_hacked_applecom/
- Exposed cameras/IOT devices.
- Huge resources like extremely large (actual) cracked password lists.