How to search URLs exposed by Shortener services
Almost two years after the release of our Buckets Search tool, we are happy to announce the release of http://shorteners.grayhatwarfare.com, the second tool in the GrayhatWarfare arsenal, that allows you to search exposed Shortened Urls.
TinyURL, bit.ly, and other similar services allow long URLs to be converted to smaller ones hosted on their service; the small URL is visited by a consumer and their web browser is redirected to the long URL. Common use-case is Twitter where the number of characters is limited but also emails and DMs.
There is always the risk, that a Shortener service, for whatever reason might stop delivering urls. Teams like 301Work and URLTeam are trying to mitigate the risk. URLTeam is periodically visiting the urls and archiving the results. The design of Shortener services made this process easier, because they use the minimum characters to accommodate the number of links they host, meaning the number of possible urls is really low and can be brute forced.
There are some security risks associated with Shortener services. First, you never know where you are redirected when you click a shortened url — there could be a malicious url on the other end. Second Shortener services can track users and their behavior. Also there are other risks when shortening urls that should remain private. Those include things like:
- Links to shared/unlocked files in Cloud services (Dropbox, Google Drive, OneDrive). OneDrive in particular used to generate short URLs for documents and folders using the 1drv.ms domain. This is a “branded short domain” operated by Bitly and uses the same tokens as bit.ly. Searching by any cloud service domain (dropbox.com, drive.google.com), reveals a lot of downloadable files. This reveals files and directories and some of them even have write access.
- Links to locations (Google Maps or other maps/location services), which often contain personal information, or routes to sensitive areas.
- Links to internal/hidden but accessible urls with sensitive information. Those could be, Admin panels, Hidden directories, or private documents
- Accidentally public trello.com boards or other similar services.
It’s a common practice for pentesters to search for sensitive urls, or gather information for a domain name, through Shortened urls. Utku Sen did an excellent job creating https://github.com/utkusen/urlhunter, which is the most known tool that tackles the problem of searching through the urls. However there are some limitations:
- The tool needs to download the files from Archive.org before it’s operational. This process is rather slow because Archive.org limits the bandwidth.
- The user must have 500GB of free disk space available to save all data.
- Filtering the results is relatively slow.
- It’s a command line tool which might be intimidating for some people.
- Most of the urls are inactive (404 links, expired domains etc).
In order to improve the search experience we:
- Removed urls from expired domains.
- Removed all non existing urls (404)
- Removed urls from parked domains, that falsely respond with 200 to all requests.
- Deduced content-type from the contents of all urls and guessed the file extension.
The functionality we offer is:
- Filter by keywords
- Filter by extension, calculated from the contents of the url and not its url extension.
- Allow to filter by domain and keywords.
- Sort by size when known.
- Sort by discovery date, based on URLTeam release date.
Our primary goal for the first release was to clean up as much junk as possible and create a robust filtering mechanism while keeping the tool as minimal as possible. As always we want your input on what functionality you want to see in the tool. shorteners.grayhatwarfare.com is the second tool we released and we have a lot more on the way.
Follow us on twitter to get announcements for new tools and updates first:
Thanks for your support, it makes delivering new tools feasible.