Discover Hidden URLs with urlhunter: Guide for Bug Bounty Pros

In cybersecurity, discovering URLs that are hidden or sensitive can be a turning point for security researchers, bug bounty hunters, and penetration testers. urlhunter, which is an open-source tool created by utkusen, simplifies this task by searching for previously crawled URLs indexed by search engines. This blog will guide you on how urlhunter operates, its uses in real life, and how to utilize it for your security research.

What is urlhunter?

urlhunter is a command-line utility aimed at retrieving URLs from Wayback Machine records. Wayback Machine, owned by the Internet Archive, caches web page historical snapshots, such as URLs no longer publicly visible. urlhunter quickly searches such archived records to uncover hidden, forgotten, or sensitive endpoints.

The utility will prove useful to security professionals wishing to:

• Find hidden or unlisted URLs.
• Find old endpoints that may still be vulnerable.
• Identify sensitive information that was once public but later removed.
• Enhance reconnaissance during bug bounty hunting.

How Does urlhunter Work?

urlhunter works by fetching URL data from the Wayback Machine and filtering it based on user-defined keywords or patterns. Here's a breakdown of the process:

• Retrieve Data: The tool queries the Wayback Machine for all URLs belonging to a specified domain.
• Filter Results: The user is able to keyword-filter URLs of interest.
• Export & Analyze: The results are saved by the tool so that the user can analyze and dig deeper into them.

This enables security researchers to easily pick useful endpoints without having to sift through thousands of URLs manually.

Installation and Setup

Prerequisites

Before running urlhunter, make sure you have:

• A Linux or macOS (Windows users can use WSL)
• Go installed (because urlhunter is a Go program)

Installing urlhunter

To install urlhunter, open a terminal and execute the following commands:

• Clone the repository

git clone https://github.com/utkusen/urlhunter.git
cd urlhunter

• Compile the tool

go build urlhunter.go

• Move the binary to /usr/local/bin for global use

mv urlhunter /usr/local/bin/

• After installation, you can check it by executing:

urlhunter -h

This should print the help menu, indicating that the tool is ready to use.

Using urlhunter: Basic to Advanced

Basic Usage

To begin with, execute urlhunter with a simple command:

urlhunter -d example.com -o output.txt

• -d is used to specify the target domain.
• -o is used to specify the output file where results will be stored.

This will retrieve all URLs archived for example.com and store them in output.txt.

Filtering URLs with Keywords

If you want to search for specific endpoints (such as admin, login, or api), you can filter using keywords:

urlhunter -d example.com -k admin,login,api -o filtered_urls.txt

• -k allows you to specify multiple keywords.
• Only URLs that have these words will be pulled.

This is helpful to identify login pages, admin panels, or API endpoints quickly.

Finding File Extensions

Security researchers might at times need to find certain file extensions such as .js, .json, .php files that could contain API keys or credentials. To achieve this, one should use:

urlhunter -d example.com -k .js,.json,.php -o sensitive_files.txt

This will give you URLs with these extensions, aiding in the identification of potentially sensitive files.

urlhunter integration with Other Tools

To achieve maximum efficiency, you can integrate urlhunter with other security tools such as grep, httpx, and nuclei.

Example: Identifying Live Endpoints

cat output.txt | httpx -silent

This makes HTTP requests against the found URLs using urlhunter and outputs only live endpoints.

Example: Performing Security Scans

cat output.txt | nuclei -t vulnerabilities/ -o vulnerabilities.txt

This identifies known vulnerabilities within the extracted URLs using Nuclei.

Practical Applications

• Bug Bounty Hunting: Bug hunters can utilize urlhunter to look for unlisted or abandoned endpoints that could contain security flaws. This raises the possibility of capturing valid security threats and receiving payment.
• Penetration Testing: Pentesters can apply urlhunter in order to list out abandoned attack surfaces that organisations might have failed to notice. Abandoned APIs, admin controls, and secure documents are examples of targets.
• Digital Forensics: Security researchers probing data breaches can employ urlhunter to identify historical public releases, misconfigured files, or API endpoints which exposed sensitive information.
• Competitive Intelligence: Competitors may monitor prior incarnations of a website in order to compare alterations in functionality, services, or internal APIs.

Limitations of urlhunter

Although urlhunter is an advanced tool, there are certain limitations:

• Relys on Wayback Machine: urlhunter will be unable to capture a URL that was never crawled.
• Data may be stale: Web pages may no longer exist or be accessible.
• Rate Limiting: Too many queries to the Wayback Machine may be limited.

Conclusion

urlhunter is an essential tool for security researchers interested in scraping useful URLs from archived web content. Its keyword filtering capability makes it effective and convenient for bug bounty hunting, penetration testing, and forensic analysis.

If you're interested in cybersecurity, becoming an expert in urlhunter can help you discover concealed attack surfaces and improve your chances of detecting security vulnerabilities.