1.$$$ $. .$$$ $.
2$$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$.
3$ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$.
4$ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$'
5$. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$.
6$::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$
7$;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$
8$$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$'
9
10~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11
12Readme for WhatWeb - Next generation web scanner.
13Developed by Andrew Horton aka urbanadventurer and Brendan Coles
14Version: 0.4.8. Unreleased
15License: GPLv2
16
17~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
18
19This product is subject to the terms detailed in the license agreement. For more information about WhatWeb visit:
20
21 Homepage: http://www.morningstarsecurity.com/research/whatweb
22 Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/
23
24If you have any questions, comments or concerns regarding WhatWeb, please consult the documentation prior to contacting one of the developers. Your feedback is always welcome.
25
26
27 Contents
28 ========================================================================
29 1. About WhatWeb
30 2. Example Usage
31 3. Usage
32 4. Logging & Output
33 5. Plugins
34 6. Aggression
35 7. Performance & Stability
36 8. Optional Dependencies
37 9. Release History
38 10. Credits
39 11. Updates & Additional Information
40 ========================================================================
41
42
43
441. About WhatWeb
45================================================================================
46
47WhatWeb identifies websites. Its goal is to answer the question, "What is that Website?". WhatWeb recognises web technologies including content management systems (CMS), blogging platforms, statistic/analytics packages, JavaScript libraries, web servers, and embedded devices. WhatWeb has over 1700 plugins, each to recognise something different. WhatWeb also identifies version numbers, email addresses, account IDs, web framework modules, SQL errors, and more.
48
49WhatWeb can be stealthy and fast, or thorough but slow. WhatWeb supports an aggression level to control the trade off between speed and reliability. When you visit a website in your browser, the transaction includes many hints of what web technologies are powering that website. Sometimes a single webpage visit contains enough information to identify a website but when it does not, WhatWeb can interrogate the website further. The default level of aggression, called 'stealthy', is the fastest and requires only one HTTP request of a website. This is suitable for scanning public websites. More aggressive modes were developed for use in penetration tests.
50
51Most WhatWeb plugins are thorough and recognise a range of cues from subtle to obvious. For example, most WordPress websites can be identified by the meta HTML tag, e.g. '<meta name="generator" content="WordPress 2.6.5">', but a minority of WordPress websites remove this identifying tag but this does not thwart WhatWeb. The WordPress WhatWeb plugin has over 15 tests, which include checking the favicon, default installation files, login pages, and checking for "/wp-content/" within relative links.
52
53
54Features:
55 * Over 1700 plugins
56 * Control the trade off between speed/stealth and reliability
57 * Performance tuning. Control how many websites to scan concurrently.
58 * Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB.
59 * Proxy support including TOR
60 * Custom HTTP headers
61 * Basic HTTP authentication
62 * Control over webpage redirection
63 * Nmap-style IP ranges
64 * Fuzzy matching
65 * Result certainty awareness
66 * Custom plugins defined on the command line
67
68
69
702. Example Usage
71================================================================================
72
73Using WhatWeb on a couple of websites (standard WhatWeb output is in colour):
74
75$ ./whatweb slashdot.org reddit.com
76http://reddit.com [302] HTTPServer[AkamaiGHost], RedirectLocation[http://www.reddit.com/], Via-Proxy[1.1 bc1], IP[173.223.232.64], Akamai-Global-Host, Country[UNITED STATES][US]
77http://slashdot.org [200] Script, HTTPServer[Unix][Apache/1.3.42 (Unix) mod_perl/1.31], Google-Analytics[GA][32013], Via-Proxy[1.1 bc5], UncommonHeaders[x-fry,x-varnish,x-xrds-location,slash_log_data], Apache[1.3.42][mod_perl/1.31], HTML5, IP[216.34.181.45], OpenGraphProtocol[100000696822412], X-Powered-By[Slash 2.005001], Title[Slashdot: News for nerds, stuff that matters], Email[canadaboy@nOspam.gmail.com,jbort@nww.com], Country[UNITED STATES][US]
78http://www.reddit.com/ [200] Frame, PasswordField[passwd,passwd2], Script, HTTPServer['; DROP TABLE servertypes; --], IP[203.97.86.202], JQuery, Cookies[reddit_first], Title[reddit: the voice of the internet -- news before it happens], Country[NEW ZEALAND][NZ]
79
80
81
82
833. Usage
84================================================================================
85
86.$$$ $. .$$$ $.
87$$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$.
88$ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$.
89$ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$'
90$. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$.
91$::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$
92$;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$
93$$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$'
94
95WhatWeb - Next generation web scanner version 0.4.8-dev.
96Developed by Andrew Horton aka urbanadventurer and Brendan Coles.
97Homepage: http://www.morningstarsecurity.com/research/whatweb
98
99Usage: whatweb [options] <URLs>
100
101TARGET SELECTION:
102 <TARGETs> Enter URLs, hostnames, IP adddresses, filenames,
103 or nmap-format IP address ranges.
104 --input-file=FILE, -i Read targets from a file. You can pipe
105 hostnames or URLs directly with -i /dev/stdin.
106
107TARGET MODIFICATION:
108 --url-prefix Add a prefix to target URLs.
109 --url-suffix Add a suffix to target URLs.
110 --url-pattern Insert the targets into a URL. Requires --input-file,
111 eg. www.example.com/%insert%/robots.txt
112
113AGGRESSION:
114 The aggression level controls the trade-off between speed/stealth and
115 reliability.
116 --aggression, -a=LEVEL Set the aggression level. Default: 1.
117 Aggression levels are:
118 1. Stealthy Makes one HTTP request per target. Also follows redirects.
119 2. Unused
120 3. Aggressive If a level 1 plugin is matched, additional requests will be
121 made.
122 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from
123 all plugins are used for all URLs.
124
125HTTP OPTIONS:
126 --user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.4.8-dev.
127 --header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default
128 header will replace it. Specifying an empty value, eg.
129 "User-Agent:" will remove the header.
130 --follow-redirect=WHEN Control when to follow redirects. WHEN may be `never',
131 `http-only', `meta-only', `same-site', `same-domain'
132 or `always'. Default: always.
133 --max-redirects=NUM Maximum number of contiguous redirects. Default: 10.
134
135AUTHENTICATION:
136 --user, -u=<user:password> HTTP basic authentication.
137 --cookie, -c=COOKIES Provide cookies, e.g. 'name=value; name2=value2'.
138
139PROXY:
140 --proxy <hostname[:port]> Set proxy hostname and port.
141 Default: 8080.
142 --proxy-user <username:password> Set proxy user and password.
143
144PLUGINS:
145 --list-plugins, -l List all plugins.
146 --info-plugins, -I=[SEARCH] List all plugins with detailed information.
147 Optionally search with keywords in a comma
148 delimited list.
149 --search-plugins=STRING Search plugins for a keyword.
150 --plugins, -p=LIST Select plugins. LIST is a comma delimited set of
151 selected plugins. Default is all.
152 Each element can be a directory, file or plugin name and
153 can optionally have a modifier, eg. + or -
154 Examples: +/tmp/moo.rb,+/tmp/foo.rb
155 title,md5,+./plugins-disabled/
156 ./plugins-disabled,-md5
157 -p + is a shortcut for -p +plugins-disabled.
158
159 --grep, -g=STRING Search for STRING in HTTP responses. Reports with a
160 plugin named Grep.
161 --custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin,
162 Examples: ":text=>'powered by abc'"
163 ":version=>/powered[ ]?by ab[0-9]/"
164 ":ghdb=>'intitle:abc \"powered by abc\"'"
165 ":md5=>'8666257030b94d3bdb46e05945f60b42'"
166 --dorks=PLUGIN List Google dorks for the selected plugin.
167
168OUTPUT:
169 --verbose, -v Verbose output includes plugin descriptions. Use twice
170 for debugging.
171 --colour,--color=WHEN control whether colour is used. WHEN may be `never',
172 `always', or `auto'.
173 --quiet, -q Do not display brief logging to STDOUT.
174 --no-errors Suppress error messages.
175
176LOGGING:
177 --log-brief=FILE Log brief, one-line output.
178 --log-verbose=FILE Log verbose output.
179 --log-errors=FILE Log errors.
180 --log-xml=FILE Log XML format.
181 --log-json=FILE Log JSON format.
182 --log-sql=FILE Log SQL INSERT statements.
183 --log-sql-create=FILE Create SQL database tables.
184 --log-json-verbose=FILE Log JSON Verbose format.
185 --log-magictree=FILE Log MagicTree XML format.
186 --log-object=FILE Log Ruby object inspection format.
187 --log-mongo-database Name of the MongoDB database.
188 --log-mongo-collection Name of the MongoDB collection. Default: whatweb.
189 --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0.
190 --log-mongo-username MongoDB username. Default: nil.
191 --log-mongo-password MongoDB password. Default: nil.
192
193PERFORMANCE & STABILITY:
194 --max-threads, -t Number of simultaneous threads. Default: 25.
195 --open-timeout Time in seconds. Default: 15.
196 --read-timeout Time in seconds. Default: 30.
197 --wait=SECONDS Wait SECONDS between connections.
198 This is useful when using a single thread.
199
200HELP & MISCELLANEOUS:
201 --short-help Short usage help.
202 --help, -h Complete usage help.
203 --debug Raise errors in plugins.
204 --version Display version information. (WhatWeb 0.4.8-dev).
205
206EXAMPLE USAGE:
207* Scan example.com.
208 ./whatweb example.com
209* Scan reddit.com slashdot.org with verbose plugin descriptions.
210 ./whatweb -v reddit.com slashdot.org
211* An aggressive scan of wired.com detects the exact version of WordPress.
212 ./whatweb -a 3 www.wired.com
213* Scan the local network quickly and suppress errors.
214 whatweb --no-errors 192.168.0.0/24
215* Scan the local network for https websites.
216 whatweb --no-errors --url-prefix https:// 192.168.0.0/24
217* Scan for crossdomain policies in the Alexa Top 1000.
218 ./whatweb -i plugin-development/alexa-top-100.txt \
219 --url-suffix /crossdomain.xml -p crossdomain_xml
220
221OPTIONAL DEPENDENCIES
222--------------------------------------------------------------------------------
223To enable MongoDB logging install the mongo gem.
224To enable character set detection and MongoDB logging install the rchardet gem.
225
226
227
228
229
2304. Logging & Output
231================================================================================
232
233The following types of logging are supported:
234 --log-brief=FILE Brief, one-line, greppable format
235 --log-verbose=FILE Verbose
236 --log-xml=FILE XML format. XSL stylesheet is provided
237 --log-json=FILE JSON format
238 --log-json-verbose=FILE JSON verbose format
239 --log-magictree=FILE MagicTree XML format
240 --log-object=FILE Ruby object inspection format
241 --log-mongo-database Name of the MongoDB database
242 --log-mongo-collection Name of the MongoDB collection. Default: whatweb
243 --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0
244 --log-mongo-username MongoDB username. Default: nil
245 --log-mongo-password MongoDB password. Default: nil
246 --log-errors=FILE Log errors. This is usually printed to the screen in red.
247
248You can output to multiple logs simultaneously by specifying multiple command line logging options. Advanced users who want SQL output should read the source code to see unsupported features.
249
250
2515. Plugins
252================================================================================
253
254Matches are made with:
255 * Text strings (case sensitive)
256 * Regular expressions
257 * Google Hack Database queries (limited set of keywords)
258 * MD5 hashes
259 * URL recognition
260 * HTML tag patterns
261 * Custom ruby code for passive and aggressive operations
262
263To list the plugins supported:
264
265$ ./whatweb -l
266
267WhatWeb Plugin List
268
269Plugin Name - Description
270--------------------------------------------------------------------------------
2711024-CMS - 1024 is one of a few CMS's leading the way with the implementation...
272360-Web-Manager - 360-Web-Manager
2733COM-NBX - 3COM NBX phone system. The NBX NetSet utility is a web interface i...
2743dcart - 3dcart - The 3dcart Shopping Cart Software is a complete ecommerce s...
2754D - 4D web application deployment server
2764images - 4images is a powerful web-based image gallery management system. Fe...
277... (truncated)
278
279
280To view more detail about a plugin or search plugins for a keyword:
281
282$ ./whatweb -I phpBB
283WhatWeb Detailed Plugin List
284Searching for phpBB
285================================================================================
286Plugin: phpBB
287--------------------------------------------------------------------------------
288Description: phpBB is a free forum
289Website: http://phpbb.org/
290
291Author: Andrew Horton
292Version: 0.3
293
294Features: [Yes] Pattern Matching (7)
295 [Yes] Version detection from pattern matching
296 [Yes] Function for passive matches
297 [Yes] Function for aggressive matches
298 [Yes] Google Dorks (1)
299
300Google Dorks:
301[1] "Powered by phpBB"
302================================================================================
303
304
305All plugins are loaded by default.
306
307Plugins can be selected by directories, files or plugin names as a comma delimited list with the -p or --plugin command line option.
308
309Each list item may have a modifier: + adds to the full set, - removes from the full set and no modifier overrides the defaults.
310
311Examples :
312
313 --plugins +plugins-disabled,-foobar
314 --plugins +/tmp/moo.rb
315 --plugins foobar (only select foobar)
316 -p title,md5,+./plugins-disabled/
317 -p ./plugins-disabled,-md5
318
319
320The --dorks <plugin name> command line option returns google dorks for the selected plugin.
321
322For example, --dorks wordpress returns "is proudly powered by WordPress"
323
324The --grep, -g command line option searches the target page for the selected string and returns a match in a plugin called Grep if it is found.
325
326
3276. Aggression
328================================================================================
329
330WhatWeb features several levels of aggression. By default the aggression level is set to 1 (stealthy) which sends a single HTTP GET request and also follows redirects.
331
332--aggression, -a
333
334 1. Stealthy Makes one HTTP request per target. Also follows redirects.
335 2. Unused
336 3. Aggressive Can make a handful of HTTP requests per target. This triggers
337 aggressive plugins for targets only when those plugins are
338 identified with a level 1 request first.
339 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from
340 all plugins are used for all URLs.
341
342Level 3 aggressive plugins will guess more URLs and perform actions that are potentially unsuitable without permission. WhatWeb currently does not support any intrusion/exploit level tests in plugins.
343
344An example of the different results between level 1 and level 3:
345-----------------------------------------------------------------
346A level 1, stealthy scan identifes that smartor.is-root.com/forum/ uses phpBB version 2:
347
348$ ./whatweb smartor.is-root.com/forum/
349http://smartor.is-root.com/forum/ [200] PasswordField[password], HTTPServer[Apache/2.2.15], PoweredBy[phpBB], Apache[2.2.15], IP[88.198.177.36], phpBB[2], PHP[5.2.13], X-Powered-By[PHP/5.2.13], Cookies[phpbb2mysql_data,phpbb2mysql_sid], Title[Smartors Mods Forums - Reloaded], Country[GERMANY][DE]
350
351A level 3, aggressive scan triggers additional tests in the phpBB plugin which identifies that the website uses phpBB version 2.0.20 or higher:
352
353$ ./whatweb -p plugins/phpbb.rb -a 3 smartor.is-root.com/forum/
354http://smartor.is-root.com/forum/ [200] phpBB[2,>2.0.20]
355
356Note the use of the -p argument to select only the phpBB plugin. It is advisable, but not mandatory, to select a specific plugin when attempting to fingerprint software versions in aggressive mode. This approach is far more stealthy as it will limit the number of requests.
357
358WhatWeb has no caching so if you use aggressive plugins on redirecting URLs you may fetch the same files multiple times.
359
360
3617. Performance & Stability
362================================================================================
363
364WhatWeb features several options to increase performance and stability.
365
366 --max-threads, -t Number of simultaneous threads. Default: 25.
367 --open-timeout Time in seconds. Default: 15
368 --read-timeout Time in seconds. Default: 30
369 --wait=SECONDS Wait SECONDS between connections
370 This is useful when using a single thread.
371
372The --wait and --max-threads commands can be used to assist in IDS evasion.
373
374Changing the user-agent using the -U or --user-agent command line option will avoid the Snort IDS rule for WhatWeb.
375
376If you are scanning ranges of IP addresses, it is much more efficient to use a port scanner like nmap to discover which have port 80 open before scanning with WhatWeb.
377
378Character set detection, with the Charset plugin dramatically decreases performance by requiring more CPU. This is required by JSON and MongoDB logging.
379
380
381
3828. Optional Dependencies
383================================================================================
384
385To enable JSON logging install the json gem.
386 gem install json
387
388To enable MongoDB logging install the mongo gem.
389 gem install mongo
390
391To enable character set detection and MongoDB logging install the rchardet gem.
392 gem install rchardet
393 cp plugins-disabled/charset.rb my-plugins/
394
395
396
3979. Release History
398================================================================================
399
400Version 0.3 Released at Kiwicon III (kiwicon.org), November 2nd, 2009
401Version 0.4 Released March 14th, 2010
402Version 0.4.1 Released April 28th, 2010
403Version 0.4.2 Released April 30th, 2010
404Version 0.4.3 Released May 24th, 2010
405Version 0.4.4 Released June 29th, 2010
406Version 0.4.5 Released August 17th, 2010
407Version 0.4.6 Released March 25th, 2011
408Version 0.4.7 Released April 5th, 2011
409Version 0.4.8-dev Unreleased
410
41110. Credits
412================================================================================
413
414Written by urbanadventurer aka Andrew Horton and Brendan Coles
415Homepage: http://www.morningstarsecurity.com/research/whatweb
416License: GPLv2
417
418
419DEVELOPERS
420
421Andrew Horton
422Brendan Coles
423
424
425CONTRIBUTORS
426
427Thank you to the following people who have contributed to WhatWeb.
428
429Emilio Casbas
430Louis Nyffenegger
431Patrik Wallström (pawal)
432Caleb Anderson (alhazred)
433Tonmoy Saikia
434Aung Khant (yehgdotnet)
435Erik Inge Bolsø
436nk@dsigned.gr
437Steve Milner (ashcrow)
438Michal Ambroz
439Gremwell
440Sagar Prakash Junnarkar (sagarjunnarkar)
441GertBerger
442Quintin Poirier
443Eric Sesterhenn
444dengjw (jawa)
445Pedro Worcel (droop)
446Matthieu Keller (maggick)
447Peter (pvdl)
448Napz (RootCon)
449nilx042
450Fabian Affolter (fabaff)
451Andrew Silvernail (buff3r)
452Andre Ricardo (andrericardo)
453nikosk
454Patrick Thomas (coffeetocode)
455Guillaume Delcaour (guikcd)
456Sean (wiifm69)
457Matthieu Keller (maggick)
458Raul (raurodse)
459Andrew Petro (apetro)
460Artem Taranyuk (610)
461Matti Paksula (matti)
462Tim Smith (tas50)
463Sarthak Munshi (saru95)
464
465Please let me know if I need to add any more names.
466
467
46811. Updates & Additional Information
469================================================================================
470
471The WhatWeb development build features regular updates.
472
473 * WhatWeb-dev: https://github.com/urbanadventurer/WhatWeb/
474 * WhatWeb-dev-unstable: https://github.com/bcoles/WhatWeb/
475
476Browse the wiki for more documentation and advanced usage techniques.
477
478 * Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/
479
480
481