1.. Licensed to the Apache Software Foundation (ASF) under one
2   or more contributor license agreements.  See the NOTICE file
3   distributed with this work for additional information
4   regarding copyright ownership.  The ASF licenses this file
5   to you under the Apache License, Version 2.0 (the
6   "License"); you may not use this file except in compliance
7   with the License.  You may obtain a copy of the License at
8
9   http://www.apache.org/licenses/LICENSE-2.0
10
11   Unless required by applicable law or agreed to in writing,
12   software distributed under the License is distributed on an
13   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14   KIND, either express or implied.  See the License for the
15   specific language governing permissions and limitations
16   under the License.
17
18.. include:: ../../common.defs
19
20.. _traffic_top:
21
22traffic_top
23***********
24
25Description
26===========
27
28The :program:`traffic_top` program provides a simple CLI view of your |TS|
29statistics, reminiscent of programs like :manpage:`top(1)` and :manpage:`nmon(1)`
30for system processes and statistics.
31
32Options
33=======
34
35.. program:: traffic_top
36
37.. option:: -s COUNT
38
39   Number of seconds in between each polling of the |TS| statistics API. The
40   default is 5 seconds.
41
42.. option:: URL|hostname|hostname:port
43
44   Location at which the JSON output of |TS| statistics are accessible.
45
46Requirements
47============
48
49The :program:`traffic_top` program requires that your |TS| nodes are running
50the :ref:`Stats Over HTTP plugin <admin-plugins-stats-over-http>` and that the
51machine from which you run :program:`traffic_top` is able to access the HTTP(S)
52end point for the plugin.
53
54If, for example, you have the following entry in your :file:`plugin.config` to
55enable the plugin::
56
57    stats_over_http.so statistics
58
59And your |TS| node is accessible using the hostname ``ats.example.tld``
60(listening on port 80), then you would run :program:`traffic_top` as follows::
61
62    traffic_top http://ats.example.tld/statistics
63
64The hostname, port, and path to the :ref:`Stats Over HTTP plugin
65<admin-plugins-stats-over-http>` JSON output should be adjusted as necessary to
66match your environment.
67
68.. important::
69
70   The statistics provided by |TS| (through :program:`traffic_ctl` and
71   :ref:`admin-plugins-stats-over-http`) expose quite a bit about the inner
72   workings of your |TS| cache. While consumers of the JSON statistics endpoint
73   won't be able to see or modify the raw contents of your cache, it is still
74   very strongly advised to limit access to this URL.
75
76Interface
77=========
78
79Upon startup (and successful connection to the
80:ref:`admin-plugins-stats-over-http` endpoint), you will be presented with a
81curses interface that looks like:
82
83.. figure:: ../../static/images/appendix/traffic_top_tiny.png
84   :alt: Main interface for the traffic_top command line program.
85
86   Main interface for the traffic_top command line program.
87
88Each area of the main interface is explained in the following sections.
89
90Cache Information
91-----------------
92
93Disk Used
94~~~~~~~~~
95
96The amount of disk space currently in use by the |TS| cache. This number will
97never exceed `Disk Total`_.
98
99Statistic: :ts:stat:`proxy.process.cache.bytes_used`.
100
101Disk Total
102~~~~~~~~~~
103
104Total disk space allocated for |TS| cache.
105
106Statistic: :ts:stat:`proxy.process.cache.bytes_total`.
107
108Ram Used
109~~~~~~~~
110
111Current amount of RAM Cache occupied by objects. Objects located and served
112from the |TS| RAM Cache avoid the much slower disk I/O necessary to serve from
113spinning rust.
114
115Statistic: :ts:stat:`proxy.process.cache.ram_cache.bytes_used`
116
117Ram Total
118~~~~~~~~~
119
120Total space allocated for used by the |TS| RAM Cache.
121
122Statistic: :ts:stat:`proxy.process.cache.ram_cache.total_bytes`
123
124Lookups
125~~~~~~~
126
127Total cache object lookups performed, including disk and RAM caches.
128
129Statistic: :ts:stat:`proxy.process.http.cache_lookups`
130
131Writes
132~~~~~~
133
134Total number of object writes to the |TS| cache.
135
136Statistic: :ts:stat:`proxy.process.http.cache_writes`.
137
138Updates
139~~~~~~~
140
141Total number of existing cache objects which have been updated with new content
142from the origin server (i.e. existing cache object was stale, so |TS| revalidated
143against the origin and received a new object).
144
145Statistic: :ts:stat:`proxy.process.http.cache_updates`.
146
147Deletes
148~~~~~~~
149
150Total number of |TS| cache objects which have been deleted (generally through a
151PURGE request).
152
153Statistic: :ts:stat:`proxy.process.http.cache_deletes`.
154
155Read Active
156~~~~~~~~~~~
157
158Current number of active cache reads.
159
160Statistic: :ts:stat:`proxy.process.cache.read.active`.
161
162Writes Act
163~~~~~~~~~~
164
165Current number of active cache writes.
166
167Statistic: :ts:stat:`proxy.process.cache.write.active`.
168
169Update Act
170~~~~~~~~~~
171
172Current number of active cache updates.
173
174Statistic: :ts:stat:`proxy.process.cache.update.active`.
175
176Entries
177~~~~~~~
178
179The current number of cache directory entries in use.
180
181Statistic: :ts:stat:`proxy.process.cache.direntries.used`.
182
183Avg Size
184~~~~~~~~
185
186The average size of current in the cache directory. This is calculated by
187dividing `Entries`_ into `Disk Used`_.
188
189Statistics: :ts:stat:`proxy.process.cache.bytes_used`,
190:ts:stat:`proxy.process.cache.direntries.used`.
191
192DNS Lookup
193~~~~~~~~~~
194
195Total number of DNS lookups performed by |TS|, regardless of whether they were
196full DNS queries or were satisfied by entries in the HostDB cache. If you are
197not operating a forward proxy and if none of your origin servers are mapped by
198hostnames, then it is normal for your HostDB cache to be empty.
199
200Statistic: :ts:stat:`proxy.process.hostdb.total_lookups`.
201
202DNS Hits
203~~~~~~~~
204
205Total number of DNS lookups which were successfully served from the HostDB
206cache.
207
208Statistic: :ts:stat:`proxy.process.hostdb.total_hits`.
209
210Ram Hit
211~~~~~~~
212
213The percentage of cache lookups which were served successfully from the RAM
214Cache (thus avoiding slower I/O from the disk cache, or even slower network I/O
215to the origin server). This is calculated as a ratio from the two RAM Cache
216statistics for hits and misses.
217
218Statistics: :ts:stat:`proxy.process.cache.ram_cache.hits`,
219:ts:stat:`proxy.process.cache.ram_cache.misses`.
220
221Fresh
222~~~~~
223
224The percentage of cache lookups which located a fresh cache object (i.e. an
225object not in need of any revalidation). These transactions are served directly
226from the cache to the client without any need to contact origin servers or
227spend time updating the cache. An effective |TS| cache will have a very high
228Fresh percentage, as these are the fastest responses to clients.
229
230Statistic: :ts:stat:`proxy.process.http.transaction_counts.hit_fresh`.
231
232Revalidate
233~~~~~~~~~~
234
235The percentage of cache lookups which located a stale cache object, but for
236which the origin server did not return new data when |TS| revalidated the
237object. Revalidated objects don't incur cache update performance hits, but they
238do still lead to (what ends up being unnecessary) network traffic with origin
239servers.
240
241A high percentage of revalidated cache lookups may indicate that |TS| is being
242too aggressive with its object staleness heuristics.
243
244Statistic: :ts:stat:`proxy.process.http.transaction_counts.hit_revalidated`.
245
246Cold
247~~~~
248
249The percentage of cache lookups which located an expired cache object. These
250were requests which located a matching object in the cache, but it had already
251been expired fully and a new copy was retrieved from the origin server.
252Unfortunately, the new copy from the origin server ended up being the unchanged
253from what had been marked expired in the cache.
254
255A high percentage of cold misses indicates that your origin servers are setting
256expirations on their responses which are too short, as compared to the actual
257lifetime of the content in those responses.
258
259Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_cold`.
260
261Changed
262~~~~~~~
263
264The percentage of cache lookups which located an expired cache object and which
265resulted in new data being retrieved from the origin server.
266
267Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_changed`.
268
269Not Cache
270~~~~~~~~~
271
272The percentage of requests for which the requested object was marked not
273cacheable by the origin server. A high percentage of uncacheable objects may
274indicate that either your origin servers simply contain a large amount of
275dynamic, uncacheable data, or that they are not properly setting cache control
276headers in their responses.
277
278Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_not_cacheable`.
279
280No Cache
281~~~~~~~~
282
283The percentage of requests for which the client indicated that the cache should
284not be used (e.g. a ``Cache-Control: no-cache`` request header was present).
285
286A high percentage of these requests may indicate possible client-side abuse of
287your proxy, in which a disproportionate number of client connections are
288attempting to force their way past your |TS| cache.
289
290Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_client_no_cache`.
291
292Fresh (ms)
293~~~~~~~~~~
294
295The average amount of time per lookup (in milliseconds) spent serving requests
296which were served by fresh cache lookups. Note that the underlying statistic is
297the total amount of time |TS| has spent serving these requests since startup,
298whereas :program:`traffic_top` is displaying the number averaged by the total
299`Fresh`_ requests.
300
301Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.hit_fresh`.
302
303Reval (ms)
304~~~~~~~~~~
305
306The average amount of time per lookup (in milliseconds) spent serving requests
307which involved revalidating a stale object for which the origin server did not
308return new data. Note that the underlying statistic is the total amount of time
309|TS| has spent serving these requests since startup, whereas
310:program:`traffic_top` is displaying the number averaged by the total
311`Revalidate`_ requests.
312
313Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.hit_revalidated`.
314
315Cold (ms)
316~~~~~~~~~
317
318The average amount of time per lookup (in milliseconds) spent serving requests
319which involved expired cache objects. Note that the underlying statistic is the
320total amount of time |TS| has spent serving these requests since startup,
321whereas :program:`traffic_top` is displaying the number averaged by the total
322`Cold`_ requests.
323
324Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_cold`.
325
326Chang (ms)
327~~~~~~~~~~
328
329The average amount of time per lookup (in milliseconds) spent serving requests
330which were served by |TS| with new data obtained from an origin server. Note
331that the underlying statistic is the total amount of time |TS| has spent
332serving these requests since startup, whereas :program:`traffic_top` is
333displaying the number averaged by the total `Changed`_ requests.
334
335Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_changed`.
336
337Not (ms)
338~~~~~~~~
339
340The average amount of time per lookup (in milliseconds) spent serving requests
341which were served from the origin server because it had marked the object as
342uncacheable. Note that the underlying statistic is the total amount of time
343|TS| has spent serving these requests since startup, whereas
344:program:`traffic_top` is displaying the number averaged by the total `Not
345Cache`_ requests.
346
347Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_not_cacheable`.
348
349No (ms)
350~~~~~~~
351
352The average amount of time per lookup (in milliseconds) spent serving requests
353which were served by the origin server because the client had requested that
354the |TS| cache be bypassed. Note that the underlying statistic is the total
355amount of time |TS| has spent serving these requests since startup, whereas
356:program:`traffic_top` is displaying the number averaged by the total `No
357Cache`_ requests.
358
359Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_client_no_cache`.
360
361DNS Hit
362~~~~~~~
363
364The percentage of DNS lookups which were served from the HostDB cache, rather
365than requiring full DNS queries.
366
367Statistics: :ts:stat:`proxy.process.hostdb.total_hits`,
368:ts:stat:`proxy.process.hostdb.total_lookups`.
369
370DNS Entry
371~~~~~~~~~
372
373The total number of entries in the HostDB lookup cache. If you are not operating
374a forward proxy and if none of your origin servers are mapped by hostnames,
375then it is normal for your HostDB cache to be empty.
376
377Statistic: :ts:stat:`proxy.process.hostdb.cache.current_items`.
378
379Client Request & Response
380-------------------------
381
382GET, HEAD, POST
383~~~~~~~~~~~~~~~
384
385Each of these display the percentage of total requests by clients to |TS| which
386used the given HTTP method.
387
388Statistics: :ts:stat:`proxy.process.http.get_requests`,
389:ts:stat:`proxy.process.http.head_requests`,
390:ts:stat:`proxy.process.http.post_requests`.
391
3922xx
393~~~
394
395Percentage of all requests made to |TS| which resulted in an HTTP response code
396between 200 and 299 (inclusive) being sent back to the client. 2xx response
397codes all indicate some form of successful transaction with content delivered.
398
399Statistic: :ts:stat:`proxy.process.http.2xx_responses`.
400
4013xx
402~~~
403
404Percentage of all requests made to |TS| which resulted in an HTTP response code
405between 300 and 399 (inclusive) being sent back to the client. 3xx response
406codes indicate non-error conditions; most commonly redirects or IMS not-modified
407responses that did not deliver content (because they did not need to).
408
409Statistic: :ts:stat:`proxy.process.http.3xx_responses`.
410
4114xx
412~~~
413
414Percentage of all requests made to |TS| which resulted in an HTTP response code
415between 400 and 499 (inclusive) being sent back to the client. 4xx response
416codes are used for requests which included a client-side error. Most frequently
417these responses are for invalid URLs (e.g. 404 Not Found), but also include
418authentication failures (e.g. 403 Forbidden). In short, |TS| refused to fulfill
419the request because the client sent invalid or incorrect information.
420
421Statistic: :ts:stat:`proxy.process.http.4xx_responses`.
422
4235xx
424~~~
425
426Percentage of all requests made to |TS| which resulted in an HTTP response code
427between 500 and 599 (inclusive) being sent back to the client. 5xx response
428code indicate a server-side error. For a caching proxy like |TS|, these are
429likely to be most often returned for gateway errors; e.g. 502 Bad Gateway (the
430origin server address was invalid or a connection could not be established at
431all due to system or network failures) and 504 Gateway Timeout (the origin
432server was contacted, but did not return data in the time allowed).
433
434Statistic: :ts:stat:`proxy.process.http.5xx_responses`.
435
436Conn Fail
437~~~~~~~~~
438
439The total number of connections to |TS| which failed to connect properly.
440
441Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.connect_failed`.
442
443Other Err
444~~~~~~~~~
445
446The total number of |TS| transactions which experienced an error not covered by
447`Conn Fail`_ and `Abort`_.
448
449Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.other`.
450
451Abort
452~~~~~
453
454The total number of |TS| transactions which were prematurely ended (connection
455was closed before all data had been received and/or sent).
456
457Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.aborts`.
458
459200, 206, 301, 302, 304, 404, 502
460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
461
462The percentage of total |TS| transactions which resulted in the specified HTTP
463response code. For details on the meaning of individual status codes, please
464refer to :ref:`appendix-http-status-codes`.
465
466Statistics:
467:ts:stat:`proxy.process.http.200_responses`,
468:ts:stat:`proxy.process.http.206_responses`,
469:ts:stat:`proxy.process.http.301_responses`,
470:ts:stat:`proxy.process.http.302_responses`,
471:ts:stat:`proxy.process.http.304_responses`,
472:ts:stat:`proxy.process.http.404_responses`,
473:ts:stat:`proxy.process.http.502_responses`.
474
475.. note::
476
477   |TS| also provides statistics for every other response code. The keen observer
478   will have hopefully already recognized the pattern in statistic names.
479
480100 B, 1 KB, 3 KB, 5 KB, 10 KB, 1 MB, > 1 MB
481~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
482
483Each of these fields indicates the percentage of responses from |TS| which fell
484within a particular size (document body size, excluding response headers). The
485individual fields are the upper bounds of exclusive *buckets*, meaning that a
486response with a document body of 4,500 bytes will be counted in the ``5 KB``
487field, but not in any of the smaller sizes.
488
489Statistics:
490:ts:stat:`proxy.process.http.response_document_size_100`,
491:ts:stat:`proxy.process.http.response_document_size_1K`,
492:ts:stat:`proxy.process.http.response_document_size_3K`,
493:ts:stat:`proxy.process.http.response_document_size_5K`,
494:ts:stat:`proxy.process.http.response_document_size_10K`,
495:ts:stat:`proxy.process.http.response_document_size_1M`,
496:ts:stat:`proxy.process.http.response_document_size_inf`.
497
498Client
499------
500
501Requests
502~~~~~~~~
503
504Total number of client requests serviced by |TS|. This includes both successful
505content-bearing responses as well as errors, redirects, and not-modified IMS
506responses. Additionally, this number includes requests which were serviced from
507the |TS| cache as well as through proxied origin connections.
508
509Statistic: :ts:stat:`proxy.process.http.incoming_requests`.
510
511Req/Conn
512~~~~~~~~
513
514The average number of requests made per client connection. When Keep-Alive
515support is enabled in |TS| and clients make use of it, they are able to submit
516multiple document requests over a single connection in some situations. This
517number is calculated from the total number of client requests divided by the
518total number of new connections that have been created.
519
520Statistics:
521:ts:stat:`proxy.process.http.incoming_requests`,
522:ts:stat:`proxy.process.http.total_client_connections`.
523
524New Conn
525~~~~~~~~
526
527The total number of new HTTP connections from clients which have been created
528over the lifetime of the |TS| process.
529
530Statistic: :ts:stat:`proxy.process.http.total_client_connections`.
531
532Curr Conn
533~~~~~~~~~
534
535The number of currently-open HTTP connections from clients with |TS|.
536
537Statistic: :ts:stat:`proxy.process.http.current_client_connections`.
538
539Active Con
540~~~~~~~~~~
541
542The number of currently active client connection threads (requests in the
543process of being fulfilled when the statistics snapshot was taken).
544
545Statistic: :ts:stat:`proxy.process.http.current_active_client_connections`.
546
547Dynamic KA
548~~~~~~~~~~
549
550Statistics:
551:ts:stat:`proxy.process.net.dynamic_keep_alive_timeout_in_total`,
552:ts:stat:`proxy.process.net.dynamic_keep_alive_timeout_in_count`.
553
554Head Bytes
555~~~~~~~~~~
556
557The total bytes consumed by outgoing server response headers from |TS| to
558clients.
559
560Statistic: :ts:stat:`proxy.process.http.user_agent_response_header_total_size`.
561
562Body Bytes
563~~~~~~~~~~
564
565The total bytes consumed by outgoing document bodies in responses from |TS| to
566clients.
567
568Statistic: :ts:stat:`proxy.process.http.user_agent_response_document_total_size`.
569
570Avg Size
571~~~~~~~~
572
573Average size in bytes of combined headers and document bodies for |TS|
574responses to clients.
575
576Statistics:
577:ts:stat:`proxy.process.http.user_agent_response_header_total_size`,
578:ts:stat:`proxy.process.http.user_agent_response_document_total_size`,
579:ts:stat:`proxy.process.http.incoming_requests`.
580
581Net (bits)
582~~~~~~~~~~
583
584The summed bits (not bytes) of all |TS| responses to clients, whether served
585from the |TS| or through a proxied connection to an origin server.
586
587Statistics:
588:ts:stat:`proxy.process.http.user_agent_response_header_total_size`,
589:ts:stat:`proxy.process.http.user_agent_response_document_total_size`.
590
591Resp (ms)
592~~~~~~~~~
593
594The average response time by |TS| across all client requests. Response time is
595measured from the moment a client connection is established, until the moment
596the last byte of the response is delivered. This field is simply the result of
597dividing the total time spent by |TS| servicing client requests by the total
598number of those requests.
599
600Statistics: :ts:stat:`proxy.process.http.total_transactions_time`,
601:ts:stat:`proxy.process.http.incoming_requests`.
602
603Origin Server
604-------------
605
606Requests
607~~~~~~~~
608
609The total number of requests made by |TS| to origin servers, because client
610requests could not be fulfilled by the |TS| cache (for any reason, whether it
611was not present in the cache, was stale or expired, or not cacheable).
612
613Statistic: :ts:stat:`proxy.process.http.outgoing_requests`.
614
615Req/Conn
616~~~~~~~~
617
618The average number of requests made to origin servers by |TS| per connection.
619This field is simply the result of dividing the total number of requests made
620by the total number of connections that have ever been opened.
621
622Statistics:
623:ts:stat:`proxy.process.http.outgoing_requests`,
624:ts:stat:`proxy.process.http.total_server_connections`.
625
626New Conn
627~~~~~~~~
628
629The total number of new HTTP connections from |TS| to origin servers that have
630been created during the lifetime of the |TS| process.
631
632Statistic: :ts:stat:`proxy.process.http.total_server_connections`.
633
634Curr Conn
635~~~~~~~~~
636
637The number of HTTP connections currently open from |TS| to origin servers. Note
638that |TS| maintains a configurable number of origin server connections open at
639all times, whether there are requests being proxied or not, when configured as
640a reverse proxy to a known list of origin servers. This is not the case for
641forward proxy configurations, however, as |TS| has no foreknowledge of the
642servers to which clients may try to connect.
643
644Statistic: :ts:stat:`proxy.process.http.current_server_connections`.
645
646Head Bytes
647~~~~~~~~~~
648
649The total bytes delivered as headers in responses from origin servers to |TS|.
650
651Statistic: :ts:stat:`proxy.process.http.origin_server_response_header_total_size`.
652
653Body Bytes
654~~~~~~~~~~
655
656The total bytes delivered as document bodies in responses from origin servers
657to |TS|.
658
659Statistic: :ts:stat:`proxy.process.http.origin_server_response_document_total_size`.
660
661Avg Size
662~~~~~~~~
663
664The average size of the combined header and document body responses from origin
665servers to |TS|.
666
667Statistics:
668:ts:stat:`proxy.process.http.origin_server_response_header_total_size`,
669:ts:stat:`proxy.process.http.origin_server_response_document_total_size`,
670:ts:stat:`proxy.process.http.outgoing_requests`.
671
672Net (bits)
673~~~~~~~~~~
674
675The total bits (not bytes) transferred from origin servers to |TS| for proxied
676requests not fulfilled by the |TS| cache.
677
678Statistics:
679:ts:stat:`proxy.process.http.origin_server_response_header_total_size`,
680:ts:stat:`proxy.process.http.origin_server_response_document_total_size`.
681