1.$$$ $. .$$$ $. 2$$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$. 3$ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$. 4$ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$' 5$. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$. 6$::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$ 7$;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$ 8$$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$' 9 10~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11 12Readme for WhatWeb - Next generation web scanner. 13Developed by Andrew Horton aka urbanadventurer and Brendan Coles 14Version: 0.4.8. Unreleased 15License: GPLv2 16 17~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 18 19This product is subject to the terms detailed in the license agreement. For more information about WhatWeb visit: 20 21 Homepage: http://www.morningstarsecurity.com/research/whatweb 22 Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/ 23 24If you have any questions, comments or concerns regarding WhatWeb, please consult the documentation prior to contacting one of the developers. Your feedback is always welcome. 25 26 27 Contents 28 ======================================================================== 29 1. About WhatWeb 30 2. Example Usage 31 3. Usage 32 4. Logging & Output 33 5. Plugins 34 6. Aggression 35 7. Performance & Stability 36 8. Optional Dependencies 37 9. Release History 38 10. Credits 39 11. Updates & Additional Information 40 ======================================================================== 41 42 43 441. About WhatWeb 45================================================================================ 46 47WhatWeb identifies websites. Its goal is to answer the question, "What is that Website?". WhatWeb recognises web technologies including content management systems (CMS), blogging platforms, statistic/analytics packages, JavaScript libraries, web servers, and embedded devices. WhatWeb has over 1700 plugins, each to recognise something different. WhatWeb also identifies version numbers, email addresses, account IDs, web framework modules, SQL errors, and more. 48 49WhatWeb can be stealthy and fast, or thorough but slow. WhatWeb supports an aggression level to control the trade off between speed and reliability. When you visit a website in your browser, the transaction includes many hints of what web technologies are powering that website. Sometimes a single webpage visit contains enough information to identify a website but when it does not, WhatWeb can interrogate the website further. The default level of aggression, called 'stealthy', is the fastest and requires only one HTTP request of a website. This is suitable for scanning public websites. More aggressive modes were developed for use in penetration tests. 50 51Most WhatWeb plugins are thorough and recognise a range of cues from subtle to obvious. For example, most WordPress websites can be identified by the meta HTML tag, e.g. '<meta name="generator" content="WordPress 2.6.5">', but a minority of WordPress websites remove this identifying tag but this does not thwart WhatWeb. The WordPress WhatWeb plugin has over 15 tests, which include checking the favicon, default installation files, login pages, and checking for "/wp-content/" within relative links. 52 53 54Features: 55 * Over 1700 plugins 56 * Control the trade off between speed/stealth and reliability 57 * Performance tuning. Control how many websites to scan concurrently. 58 * Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB. 59 * Proxy support including TOR 60 * Custom HTTP headers 61 * Basic HTTP authentication 62 * Control over webpage redirection 63 * Nmap-style IP ranges 64 * Fuzzy matching 65 * Result certainty awareness 66 * Custom plugins defined on the command line 67 68 69 702. Example Usage 71================================================================================ 72 73Using WhatWeb on a couple of websites (standard WhatWeb output is in colour): 74 75$ ./whatweb slashdot.org reddit.com 76http://reddit.com [302] HTTPServer[AkamaiGHost], RedirectLocation[http://www.reddit.com/], Via-Proxy[1.1 bc1], IP[173.223.232.64], Akamai-Global-Host, Country[UNITED STATES][US] 77http://slashdot.org [200] Script, HTTPServer[Unix][Apache/1.3.42 (Unix) mod_perl/1.31], Google-Analytics[GA][32013], Via-Proxy[1.1 bc5], UncommonHeaders[x-fry,x-varnish,x-xrds-location,slash_log_data], Apache[1.3.42][mod_perl/1.31], HTML5, IP[216.34.181.45], OpenGraphProtocol[100000696822412], X-Powered-By[Slash 2.005001], Title[Slashdot: News for nerds, stuff that matters], Email[canadaboy@nOspam.gmail.com,jbort@nww.com], Country[UNITED STATES][US] 78http://www.reddit.com/ [200] Frame, PasswordField[passwd,passwd2], Script, HTTPServer['; DROP TABLE servertypes; --], IP[203.97.86.202], JQuery, Cookies[reddit_first], Title[reddit: the voice of the internet -- news before it happens], Country[NEW ZEALAND][NZ] 79 80 81 82 833. Usage 84================================================================================ 85 86.$$$ $. .$$$ $. 87$$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$. 88$ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$. 89$ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$' 90$. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$. 91$::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$ 92$;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$ 93$$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$' 94 95WhatWeb - Next generation web scanner version 0.4.8-dev. 96Developed by Andrew Horton aka urbanadventurer and Brendan Coles. 97Homepage: http://www.morningstarsecurity.com/research/whatweb 98 99Usage: whatweb [options] <URLs> 100 101TARGET SELECTION: 102 <TARGETs> Enter URLs, hostnames, IP adddresses, filenames, 103 or nmap-format IP address ranges. 104 --input-file=FILE, -i Read targets from a file. You can pipe 105 hostnames or URLs directly with -i /dev/stdin. 106 107TARGET MODIFICATION: 108 --url-prefix Add a prefix to target URLs. 109 --url-suffix Add a suffix to target URLs. 110 --url-pattern Insert the targets into a URL. Requires --input-file, 111 eg. www.example.com/%insert%/robots.txt 112 113AGGRESSION: 114 The aggression level controls the trade-off between speed/stealth and 115 reliability. 116 --aggression, -a=LEVEL Set the aggression level. Default: 1. 117 Aggression levels are: 118 1. Stealthy Makes one HTTP request per target. Also follows redirects. 119 2. Unused 120 3. Aggressive If a level 1 plugin is matched, additional requests will be 121 made. 122 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from 123 all plugins are used for all URLs. 124 125HTTP OPTIONS: 126 --user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.4.8-dev. 127 --header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default 128 header will replace it. Specifying an empty value, eg. 129 "User-Agent:" will remove the header. 130 --follow-redirect=WHEN Control when to follow redirects. WHEN may be `never', 131 `http-only', `meta-only', `same-site', `same-domain' 132 or `always'. Default: always. 133 --max-redirects=NUM Maximum number of contiguous redirects. Default: 10. 134 135AUTHENTICATION: 136 --user, -u=<user:password> HTTP basic authentication. 137 --cookie, -c=COOKIES Provide cookies, e.g. 'name=value; name2=value2'. 138 139PROXY: 140 --proxy <hostname[:port]> Set proxy hostname and port. 141 Default: 8080. 142 --proxy-user <username:password> Set proxy user and password. 143 144PLUGINS: 145 --list-plugins, -l List all plugins. 146 --info-plugins, -I=[SEARCH] List all plugins with detailed information. 147 Optionally search with keywords in a comma 148 delimited list. 149 --search-plugins=STRING Search plugins for a keyword. 150 --plugins, -p=LIST Select plugins. LIST is a comma delimited set of 151 selected plugins. Default is all. 152 Each element can be a directory, file or plugin name and 153 can optionally have a modifier, eg. + or - 154 Examples: +/tmp/moo.rb,+/tmp/foo.rb 155 title,md5,+./plugins-disabled/ 156 ./plugins-disabled,-md5 157 -p + is a shortcut for -p +plugins-disabled. 158 159 --grep, -g=STRING Search for STRING in HTTP responses. Reports with a 160 plugin named Grep. 161 --custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin, 162 Examples: ":text=>'powered by abc'" 163 ":version=>/powered[ ]?by ab[0-9]/" 164 ":ghdb=>'intitle:abc \"powered by abc\"'" 165 ":md5=>'8666257030b94d3bdb46e05945f60b42'" 166 --dorks=PLUGIN List Google dorks for the selected plugin. 167 168OUTPUT: 169 --verbose, -v Verbose output includes plugin descriptions. Use twice 170 for debugging. 171 --colour,--color=WHEN control whether colour is used. WHEN may be `never', 172 `always', or `auto'. 173 --quiet, -q Do not display brief logging to STDOUT. 174 --no-errors Suppress error messages. 175 176LOGGING: 177 --log-brief=FILE Log brief, one-line output. 178 --log-verbose=FILE Log verbose output. 179 --log-errors=FILE Log errors. 180 --log-xml=FILE Log XML format. 181 --log-json=FILE Log JSON format. 182 --log-sql=FILE Log SQL INSERT statements. 183 --log-sql-create=FILE Create SQL database tables. 184 --log-json-verbose=FILE Log JSON Verbose format. 185 --log-magictree=FILE Log MagicTree XML format. 186 --log-object=FILE Log Ruby object inspection format. 187 --log-mongo-database Name of the MongoDB database. 188 --log-mongo-collection Name of the MongoDB collection. Default: whatweb. 189 --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0. 190 --log-mongo-username MongoDB username. Default: nil. 191 --log-mongo-password MongoDB password. Default: nil. 192 193PERFORMANCE & STABILITY: 194 --max-threads, -t Number of simultaneous threads. Default: 25. 195 --open-timeout Time in seconds. Default: 15. 196 --read-timeout Time in seconds. Default: 30. 197 --wait=SECONDS Wait SECONDS between connections. 198 This is useful when using a single thread. 199 200HELP & MISCELLANEOUS: 201 --short-help Short usage help. 202 --help, -h Complete usage help. 203 --debug Raise errors in plugins. 204 --version Display version information. (WhatWeb 0.4.8-dev). 205 206EXAMPLE USAGE: 207* Scan example.com. 208 ./whatweb example.com 209* Scan reddit.com slashdot.org with verbose plugin descriptions. 210 ./whatweb -v reddit.com slashdot.org 211* An aggressive scan of wired.com detects the exact version of WordPress. 212 ./whatweb -a 3 www.wired.com 213* Scan the local network quickly and suppress errors. 214 whatweb --no-errors 192.168.0.0/24 215* Scan the local network for https websites. 216 whatweb --no-errors --url-prefix https:// 192.168.0.0/24 217* Scan for crossdomain policies in the Alexa Top 1000. 218 ./whatweb -i plugin-development/alexa-top-100.txt \ 219 --url-suffix /crossdomain.xml -p crossdomain_xml 220 221OPTIONAL DEPENDENCIES 222-------------------------------------------------------------------------------- 223To enable MongoDB logging install the mongo gem. 224To enable character set detection and MongoDB logging install the rchardet gem. 225 226 227 228 229 2304. Logging & Output 231================================================================================ 232 233The following types of logging are supported: 234 --log-brief=FILE Brief, one-line, greppable format 235 --log-verbose=FILE Verbose 236 --log-xml=FILE XML format. XSL stylesheet is provided 237 --log-json=FILE JSON format 238 --log-json-verbose=FILE JSON verbose format 239 --log-magictree=FILE MagicTree XML format 240 --log-object=FILE Ruby object inspection format 241 --log-mongo-database Name of the MongoDB database 242 --log-mongo-collection Name of the MongoDB collection. Default: whatweb 243 --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0 244 --log-mongo-username MongoDB username. Default: nil 245 --log-mongo-password MongoDB password. Default: nil 246 --log-errors=FILE Log errors. This is usually printed to the screen in red. 247 248You can output to multiple logs simultaneously by specifying multiple command line logging options. Advanced users who want SQL output should read the source code to see unsupported features. 249 250 2515. Plugins 252================================================================================ 253 254Matches are made with: 255 * Text strings (case sensitive) 256 * Regular expressions 257 * Google Hack Database queries (limited set of keywords) 258 * MD5 hashes 259 * URL recognition 260 * HTML tag patterns 261 * Custom ruby code for passive and aggressive operations 262 263To list the plugins supported: 264 265$ ./whatweb -l 266 267WhatWeb Plugin List 268 269Plugin Name - Description 270-------------------------------------------------------------------------------- 2711024-CMS - 1024 is one of a few CMS's leading the way with the implementation... 272360-Web-Manager - 360-Web-Manager 2733COM-NBX - 3COM NBX phone system. The NBX NetSet utility is a web interface i... 2743dcart - 3dcart - The 3dcart Shopping Cart Software is a complete ecommerce s... 2754D - 4D web application deployment server 2764images - 4images is a powerful web-based image gallery management system. Fe... 277... (truncated) 278 279 280To view more detail about a plugin or search plugins for a keyword: 281 282$ ./whatweb -I phpBB 283WhatWeb Detailed Plugin List 284Searching for phpBB 285================================================================================ 286Plugin: phpBB 287-------------------------------------------------------------------------------- 288Description: phpBB is a free forum 289Website: http://phpbb.org/ 290 291Author: Andrew Horton 292Version: 0.3 293 294Features: [Yes] Pattern Matching (7) 295 [Yes] Version detection from pattern matching 296 [Yes] Function for passive matches 297 [Yes] Function for aggressive matches 298 [Yes] Google Dorks (1) 299 300Google Dorks: 301[1] "Powered by phpBB" 302================================================================================ 303 304 305All plugins are loaded by default. 306 307Plugins can be selected by directories, files or plugin names as a comma delimited list with the -p or --plugin command line option. 308 309Each list item may have a modifier: + adds to the full set, - removes from the full set and no modifier overrides the defaults. 310 311Examples : 312 313 --plugins +plugins-disabled,-foobar 314 --plugins +/tmp/moo.rb 315 --plugins foobar (only select foobar) 316 -p title,md5,+./plugins-disabled/ 317 -p ./plugins-disabled,-md5 318 319 320The --dorks <plugin name> command line option returns google dorks for the selected plugin. 321 322For example, --dorks wordpress returns "is proudly powered by WordPress" 323 324The --grep, -g command line option searches the target page for the selected string and returns a match in a plugin called Grep if it is found. 325 326 3276. Aggression 328================================================================================ 329 330WhatWeb features several levels of aggression. By default the aggression level is set to 1 (stealthy) which sends a single HTTP GET request and also follows redirects. 331 332--aggression, -a 333 334 1. Stealthy Makes one HTTP request per target. Also follows redirects. 335 2. Unused 336 3. Aggressive Can make a handful of HTTP requests per target. This triggers 337 aggressive plugins for targets only when those plugins are 338 identified with a level 1 request first. 339 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from 340 all plugins are used for all URLs. 341 342Level 3 aggressive plugins will guess more URLs and perform actions that are potentially unsuitable without permission. WhatWeb currently does not support any intrusion/exploit level tests in plugins. 343 344An example of the different results between level 1 and level 3: 345----------------------------------------------------------------- 346A level 1, stealthy scan identifes that smartor.is-root.com/forum/ uses phpBB version 2: 347 348$ ./whatweb smartor.is-root.com/forum/ 349http://smartor.is-root.com/forum/ [200] PasswordField[password], HTTPServer[Apache/2.2.15], PoweredBy[phpBB], Apache[2.2.15], IP[88.198.177.36], phpBB[2], PHP[5.2.13], X-Powered-By[PHP/5.2.13], Cookies[phpbb2mysql_data,phpbb2mysql_sid], Title[Smartors Mods Forums - Reloaded], Country[GERMANY][DE] 350 351A level 3, aggressive scan triggers additional tests in the phpBB plugin which identifies that the website uses phpBB version 2.0.20 or higher: 352 353$ ./whatweb -p plugins/phpbb.rb -a 3 smartor.is-root.com/forum/ 354http://smartor.is-root.com/forum/ [200] phpBB[2,>2.0.20] 355 356Note the use of the -p argument to select only the phpBB plugin. It is advisable, but not mandatory, to select a specific plugin when attempting to fingerprint software versions in aggressive mode. This approach is far more stealthy as it will limit the number of requests. 357 358WhatWeb has no caching so if you use aggressive plugins on redirecting URLs you may fetch the same files multiple times. 359 360 3617. Performance & Stability 362================================================================================ 363 364WhatWeb features several options to increase performance and stability. 365 366 --max-threads, -t Number of simultaneous threads. Default: 25. 367 --open-timeout Time in seconds. Default: 15 368 --read-timeout Time in seconds. Default: 30 369 --wait=SECONDS Wait SECONDS between connections 370 This is useful when using a single thread. 371 372The --wait and --max-threads commands can be used to assist in IDS evasion. 373 374Changing the user-agent using the -U or --user-agent command line option will avoid the Snort IDS rule for WhatWeb. 375 376If you are scanning ranges of IP addresses, it is much more efficient to use a port scanner like nmap to discover which have port 80 open before scanning with WhatWeb. 377 378Character set detection, with the Charset plugin dramatically decreases performance by requiring more CPU. This is required by JSON and MongoDB logging. 379 380 381 3828. Optional Dependencies 383================================================================================ 384 385To enable JSON logging install the json gem. 386 gem install json 387 388To enable MongoDB logging install the mongo gem. 389 gem install mongo 390 391To enable character set detection and MongoDB logging install the rchardet gem. 392 gem install rchardet 393 cp plugins-disabled/charset.rb my-plugins/ 394 395 396 3979. Release History 398================================================================================ 399 400Version 0.3 Released at Kiwicon III (kiwicon.org), November 2nd, 2009 401Version 0.4 Released March 14th, 2010 402Version 0.4.1 Released April 28th, 2010 403Version 0.4.2 Released April 30th, 2010 404Version 0.4.3 Released May 24th, 2010 405Version 0.4.4 Released June 29th, 2010 406Version 0.4.5 Released August 17th, 2010 407Version 0.4.6 Released March 25th, 2011 408Version 0.4.7 Released April 5th, 2011 409Version 0.4.8-dev Unreleased 410 41110. Credits 412================================================================================ 413 414Written by urbanadventurer aka Andrew Horton and Brendan Coles 415Homepage: http://www.morningstarsecurity.com/research/whatweb 416License: GPLv2 417 418 419DEVELOPERS 420 421Andrew Horton 422Brendan Coles 423 424 425CONTRIBUTORS 426 427Thank you to the following people who have contributed to WhatWeb. 428 429Emilio Casbas 430Louis Nyffenegger 431Patrik Wallström (pawal) 432Caleb Anderson (alhazred) 433Tonmoy Saikia 434Aung Khant (yehgdotnet) 435Erik Inge Bolsø 436nk@dsigned.gr 437Steve Milner (ashcrow) 438Michal Ambroz 439Gremwell 440Sagar Prakash Junnarkar (sagarjunnarkar) 441GertBerger 442Quintin Poirier 443Eric Sesterhenn 444dengjw (jawa) 445Pedro Worcel (droop) 446Matthieu Keller (maggick) 447Peter (pvdl) 448Napz (RootCon) 449nilx042 450Fabian Affolter (fabaff) 451Andrew Silvernail (buff3r) 452Andre Ricardo (andrericardo) 453nikosk 454Patrick Thomas (coffeetocode) 455Guillaume Delcaour (guikcd) 456Sean (wiifm69) 457Matthieu Keller (maggick) 458Raul (raurodse) 459Andrew Petro (apetro) 460Artem Taranyuk (610) 461Matti Paksula (matti) 462Tim Smith (tas50) 463Sarthak Munshi (saru95) 464 465Please let me know if I need to add any more names. 466 467 46811. Updates & Additional Information 469================================================================================ 470 471The WhatWeb development build features regular updates. 472 473 * WhatWeb-dev: https://github.com/urbanadventurer/WhatWeb/ 474 * WhatWeb-dev-unstable: https://github.com/bcoles/WhatWeb/ 475 476Browse the wiki for more documentation and advanced usage techniques. 477 478 * Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/ 479 480 481