Browse Exploit & Auxiliary Modules
The Metasploit Project hosts the world's largest database of quality assured exploits, including hundreds of remote exploits, auxiliary modules, and payloads. You can even review the Metasploit Framework source code of any module - or write your own.
Search for modules
Web Site Crawler
Crawl a web site and store information about what was found
Rank
- Normal
Authors
- hdm < hdm [at] metasploit.com >
Development
Similar Modules
- auxiliary/scanner/http/adobe_xml_inject
- auxiliary/scanner/http/apache_userdir_enum
- auxiliary/scanner/http/axis_local_file_include
- auxiliary/scanner/http/axis_login
- auxiliary/scanner/http/backup_file
- auxiliary/scanner/http/barracuda_directory_traversal
- auxiliary/scanner/http/blind_sql_query
- auxiliary/scanner/http/brute_dirs
- auxiliary/scanner/http/cert
- auxiliary/scanner/http/cisco_device_manager
Usage Information
$ msfconsole
## ### ## ##
## ## #### ###### #### ##### ##### ## #### ######
####### ## ## ## ## ## ## ## ## ## ## ### ##
####### ###### ## ##### #### ## ## ## ## ## ## ##
## # ## ## ## ## ## ## ##### ## ## ## ## ##
## ## #### ### ##### ##### ## #### #### #### ###
##
msf > use auxiliary/scanner/http/crawler
msf auxiliary(crawler) > set RHOST [TARGET IP]
msf auxiliary(crawler) > run
## ### ## ##
## ## #### ###### #### ##### ##### ## #### ######
####### ## ## ## ## ## ## ## ## ## ## ### ##
####### ###### ## ##### #### ## ## ## ## ## ## ##
## # ## ## ## ## ## ## ##### ## ## ## ## ##
## ## #### ### ##### ##### ## #### #### #### ###
##
msf > use auxiliary/scanner/http/crawler
msf auxiliary(crawler) > set RHOST [TARGET IP]
msf auxiliary(crawler) > run
Module Options
| MAX_MINUTES | The maximum number of minutes to spend on each URL (default: 5) |
| MAX_PAGES | The maximum number of pages to crawl per URL (default: 500) |
| MAX_THREADS | The maximum number of concurrent requests (default: 4) |
| Proxies | Use a proxy chain |
| RHOST | The target address |
| RPORT | The target port (default: 80) |
| URI | The starting page to crawl (default: /) |
| VHOST | HTTP server virtual host |
| BasicAuthPass | The HTTP password to specify for basic authentication |
| BasicAuthUser | The HTTP username to specify for basic authentication |
| HTTPAdditionalHeaders | A list of additional headers to send (separated by \x01) |
| HTTPCookie | A HTTP cookie header to send with each request |
| RedirectLimit | The maximum number of redirects for a single request |
| RequestTimeout | The maximum number of seconds to wait for a reply |
| RetryLimit | The maximum number of attempts for a single request |
| SSL | Negotiate SSL for outgoing connections |
| SSLVersion | Specify the version of SSL that should be used (accepted: SSL2, SSL23, SSL3, TLS1) |
| UserAgent | The User-Agent header to use for all requests |
| VERBOSE | Enable detailed status messages |
| WORKSPACE | Specify the workspace for this module |