1.. Licensed to the Apache Software Foundation (ASF) under one 2 or more contributor license agreements. See the NOTICE file 3 distributed with this work for additional information 4 regarding copyright ownership. The ASF licenses this file 5 to you under the Apache License, Version 2.0 (the 6 "License"); you may not use this file except in compliance 7 with the License. You may obtain a copy of the License at 8 9 http://www.apache.org/licenses/LICENSE-2.0 10 11 Unless required by applicable law or agreed to in writing, 12 software distributed under the License is distributed on an 13 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 14 KIND, either express or implied. See the License for the 15 specific language governing permissions and limitations 16 under the License. 17 18.. include:: ../../common.defs 19 20.. _traffic_top: 21 22traffic_top 23*********** 24 25Description 26=========== 27 28The :program:`traffic_top` program provides a simple CLI view of your |TS| 29statistics, reminiscent of programs like :manpage:`top(1)` and :manpage:`nmon(1)` 30for system processes and statistics. 31 32Options 33======= 34 35.. program:: traffic_top 36 37.. option:: -s COUNT 38 39 Number of seconds in between each polling of the |TS| statistics API. The 40 default is 5 seconds. 41 42.. option:: URL|hostname|hostname:port 43 44 Location at which the JSON output of |TS| statistics are accessible. 45 46Requirements 47============ 48 49The :program:`traffic_top` program requires that your |TS| nodes are running 50the :ref:`Stats Over HTTP plugin <admin-plugins-stats-over-http>` and that the 51machine from which you run :program:`traffic_top` is able to access the HTTP(S) 52end point for the plugin. 53 54If, for example, you have the following entry in your :file:`plugin.config` to 55enable the plugin:: 56 57 stats_over_http.so statistics 58 59And your |TS| node is accessible using the hostname ``ats.example.tld`` 60(listening on port 80), then you would run :program:`traffic_top` as follows:: 61 62 traffic_top http://ats.example.tld/statistics 63 64The hostname, port, and path to the :ref:`Stats Over HTTP plugin 65<admin-plugins-stats-over-http>` JSON output should be adjusted as necessary to 66match your environment. 67 68.. important:: 69 70 The statistics provided by |TS| (through :program:`traffic_ctl` and 71 :ref:`admin-plugins-stats-over-http`) expose quite a bit about the inner 72 workings of your |TS| cache. While consumers of the JSON statistics endpoint 73 won't be able to see or modify the raw contents of your cache, it is still 74 very strongly advised to limit access to this URL. 75 76Interface 77========= 78 79Upon startup (and successful connection to the 80:ref:`admin-plugins-stats-over-http` endpoint), you will be presented with a 81curses interface that looks like: 82 83.. figure:: ../../static/images/appendix/traffic_top_tiny.png 84 :alt: Main interface for the traffic_top command line program. 85 86 Main interface for the traffic_top command line program. 87 88Each area of the main interface is explained in the following sections. 89 90Cache Information 91----------------- 92 93Disk Used 94~~~~~~~~~ 95 96The amount of disk space currently in use by the |TS| cache. This number will 97never exceed `Disk Total`_. 98 99Statistic: :ts:stat:`proxy.process.cache.bytes_used`. 100 101Disk Total 102~~~~~~~~~~ 103 104Total disk space allocated for |TS| cache. 105 106Statistic: :ts:stat:`proxy.process.cache.bytes_total`. 107 108Ram Used 109~~~~~~~~ 110 111Current amount of RAM Cache occupied by objects. Objects located and served 112from the |TS| RAM Cache avoid the much slower disk I/O necessary to serve from 113spinning rust. 114 115Statistic: :ts:stat:`proxy.process.cache.ram_cache.bytes_used` 116 117Ram Total 118~~~~~~~~~ 119 120Total space allocated for used by the |TS| RAM Cache. 121 122Statistic: :ts:stat:`proxy.process.cache.ram_cache.total_bytes` 123 124Lookups 125~~~~~~~ 126 127Total cache object lookups performed, including disk and RAM caches. 128 129Statistic: :ts:stat:`proxy.process.http.cache_lookups` 130 131Writes 132~~~~~~ 133 134Total number of object writes to the |TS| cache. 135 136Statistic: :ts:stat:`proxy.process.http.cache_writes`. 137 138Updates 139~~~~~~~ 140 141Total number of existing cache objects which have been updated with new content 142from the origin server (i.e. existing cache object was stale, so |TS| revalidated 143against the origin and received a new object). 144 145Statistic: :ts:stat:`proxy.process.http.cache_updates`. 146 147Deletes 148~~~~~~~ 149 150Total number of |TS| cache objects which have been deleted (generally through a 151PURGE request). 152 153Statistic: :ts:stat:`proxy.process.http.cache_deletes`. 154 155Read Active 156~~~~~~~~~~~ 157 158Current number of active cache reads. 159 160Statistic: :ts:stat:`proxy.process.cache.read.active`. 161 162Writes Act 163~~~~~~~~~~ 164 165Current number of active cache writes. 166 167Statistic: :ts:stat:`proxy.process.cache.write.active`. 168 169Update Act 170~~~~~~~~~~ 171 172Current number of active cache updates. 173 174Statistic: :ts:stat:`proxy.process.cache.update.active`. 175 176Entries 177~~~~~~~ 178 179The current number of cache directory entries in use. 180 181Statistic: :ts:stat:`proxy.process.cache.direntries.used`. 182 183Avg Size 184~~~~~~~~ 185 186The average size of current in the cache directory. This is calculated by 187dividing `Entries`_ into `Disk Used`_. 188 189Statistics: :ts:stat:`proxy.process.cache.bytes_used`, 190:ts:stat:`proxy.process.cache.direntries.used`. 191 192DNS Lookup 193~~~~~~~~~~ 194 195Total number of DNS lookups performed by |TS|, regardless of whether they were 196full DNS queries or were satisfied by entries in the HostDB cache. If you are 197not operating a forward proxy and if none of your origin servers are mapped by 198hostnames, then it is normal for your HostDB cache to be empty. 199 200Statistic: :ts:stat:`proxy.process.hostdb.total_lookups`. 201 202DNS Hits 203~~~~~~~~ 204 205Total number of DNS lookups which were successfully served from the HostDB 206cache. 207 208Statistic: :ts:stat:`proxy.process.hostdb.total_hits`. 209 210Ram Hit 211~~~~~~~ 212 213The percentage of cache lookups which were served successfully from the RAM 214Cache (thus avoiding slower I/O from the disk cache, or even slower network I/O 215to the origin server). This is calculated as a ratio from the two RAM Cache 216statistics for hits and misses. 217 218Statistics: :ts:stat:`proxy.process.cache.ram_cache.hits`, 219:ts:stat:`proxy.process.cache.ram_cache.misses`. 220 221Fresh 222~~~~~ 223 224The percentage of cache lookups which located a fresh cache object (i.e. an 225object not in need of any revalidation). These transactions are served directly 226from the cache to the client without any need to contact origin servers or 227spend time updating the cache. An effective |TS| cache will have a very high 228Fresh percentage, as these are the fastest responses to clients. 229 230Statistic: :ts:stat:`proxy.process.http.transaction_counts.hit_fresh`. 231 232Revalidate 233~~~~~~~~~~ 234 235The percentage of cache lookups which located a stale cache object, but for 236which the origin server did not return new data when |TS| revalidated the 237object. Revalidated objects don't incur cache update performance hits, but they 238do still lead to (what ends up being unnecessary) network traffic with origin 239servers. 240 241A high percentage of revalidated cache lookups may indicate that |TS| is being 242too aggressive with its object staleness heuristics. 243 244Statistic: :ts:stat:`proxy.process.http.transaction_counts.hit_revalidated`. 245 246Cold 247~~~~ 248 249The percentage of cache lookups which located an expired cache object. These 250were requests which located a matching object in the cache, but it had already 251been expired fully and a new copy was retrieved from the origin server. 252Unfortunately, the new copy from the origin server ended up being the unchanged 253from what had been marked expired in the cache. 254 255A high percentage of cold misses indicates that your origin servers are setting 256expirations on their responses which are too short, as compared to the actual 257lifetime of the content in those responses. 258 259Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_cold`. 260 261Changed 262~~~~~~~ 263 264The percentage of cache lookups which located an expired cache object and which 265resulted in new data being retrieved from the origin server. 266 267Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_changed`. 268 269Not Cache 270~~~~~~~~~ 271 272The percentage of requests for which the requested object was marked not 273cacheable by the origin server. A high percentage of uncacheable objects may 274indicate that either your origin servers simply contain a large amount of 275dynamic, uncacheable data, or that they are not properly setting cache control 276headers in their responses. 277 278Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_not_cacheable`. 279 280No Cache 281~~~~~~~~ 282 283The percentage of requests for which the client indicated that the cache should 284not be used (e.g. a ``Cache-Control: no-cache`` request header was present). 285 286A high percentage of these requests may indicate possible client-side abuse of 287your proxy, in which a disproportionate number of client connections are 288attempting to force their way past your |TS| cache. 289 290Statistic: :ts:stat:`proxy.process.http.transaction_counts.miss_client_no_cache`. 291 292Fresh (ms) 293~~~~~~~~~~ 294 295The average amount of time per lookup (in milliseconds) spent serving requests 296which were served by fresh cache lookups. Note that the underlying statistic is 297the total amount of time |TS| has spent serving these requests since startup, 298whereas :program:`traffic_top` is displaying the number averaged by the total 299`Fresh`_ requests. 300 301Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.hit_fresh`. 302 303Reval (ms) 304~~~~~~~~~~ 305 306The average amount of time per lookup (in milliseconds) spent serving requests 307which involved revalidating a stale object for which the origin server did not 308return new data. Note that the underlying statistic is the total amount of time 309|TS| has spent serving these requests since startup, whereas 310:program:`traffic_top` is displaying the number averaged by the total 311`Revalidate`_ requests. 312 313Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.hit_revalidated`. 314 315Cold (ms) 316~~~~~~~~~ 317 318The average amount of time per lookup (in milliseconds) spent serving requests 319which involved expired cache objects. Note that the underlying statistic is the 320total amount of time |TS| has spent serving these requests since startup, 321whereas :program:`traffic_top` is displaying the number averaged by the total 322`Cold`_ requests. 323 324Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_cold`. 325 326Chang (ms) 327~~~~~~~~~~ 328 329The average amount of time per lookup (in milliseconds) spent serving requests 330which were served by |TS| with new data obtained from an origin server. Note 331that the underlying statistic is the total amount of time |TS| has spent 332serving these requests since startup, whereas :program:`traffic_top` is 333displaying the number averaged by the total `Changed`_ requests. 334 335Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_changed`. 336 337Not (ms) 338~~~~~~~~ 339 340The average amount of time per lookup (in milliseconds) spent serving requests 341which were served from the origin server because it had marked the object as 342uncacheable. Note that the underlying statistic is the total amount of time 343|TS| has spent serving these requests since startup, whereas 344:program:`traffic_top` is displaying the number averaged by the total `Not 345Cache`_ requests. 346 347Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_not_cacheable`. 348 349No (ms) 350~~~~~~~ 351 352The average amount of time per lookup (in milliseconds) spent serving requests 353which were served by the origin server because the client had requested that 354the |TS| cache be bypassed. Note that the underlying statistic is the total 355amount of time |TS| has spent serving these requests since startup, whereas 356:program:`traffic_top` is displaying the number averaged by the total `No 357Cache`_ requests. 358 359Statistic: :ts:stat:`proxy.process.http.transaction_totaltime.miss_client_no_cache`. 360 361DNS Hit 362~~~~~~~ 363 364The percentage of DNS lookups which were served from the HostDB cache, rather 365than requiring full DNS queries. 366 367Statistics: :ts:stat:`proxy.process.hostdb.total_hits`, 368:ts:stat:`proxy.process.hostdb.total_lookups`. 369 370DNS Entry 371~~~~~~~~~ 372 373The total number of entries in the HostDB lookup cache. If you are not operating 374a forward proxy and if none of your origin servers are mapped by hostnames, 375then it is normal for your HostDB cache to be empty. 376 377Statistic: :ts:stat:`proxy.process.hostdb.cache.current_items`. 378 379Client Request & Response 380------------------------- 381 382GET, HEAD, POST 383~~~~~~~~~~~~~~~ 384 385Each of these display the percentage of total requests by clients to |TS| which 386used the given HTTP method. 387 388Statistics: :ts:stat:`proxy.process.http.get_requests`, 389:ts:stat:`proxy.process.http.head_requests`, 390:ts:stat:`proxy.process.http.post_requests`. 391 3922xx 393~~~ 394 395Percentage of all requests made to |TS| which resulted in an HTTP response code 396between 200 and 299 (inclusive) being sent back to the client. 2xx response 397codes all indicate some form of successful transaction with content delivered. 398 399Statistic: :ts:stat:`proxy.process.http.2xx_responses`. 400 4013xx 402~~~ 403 404Percentage of all requests made to |TS| which resulted in an HTTP response code 405between 300 and 399 (inclusive) being sent back to the client. 3xx response 406codes indicate non-error conditions; most commonly redirects or IMS not-modified 407responses that did not deliver content (because they did not need to). 408 409Statistic: :ts:stat:`proxy.process.http.3xx_responses`. 410 4114xx 412~~~ 413 414Percentage of all requests made to |TS| which resulted in an HTTP response code 415between 400 and 499 (inclusive) being sent back to the client. 4xx response 416codes are used for requests which included a client-side error. Most frequently 417these responses are for invalid URLs (e.g. 404 Not Found), but also include 418authentication failures (e.g. 403 Forbidden). In short, |TS| refused to fulfill 419the request because the client sent invalid or incorrect information. 420 421Statistic: :ts:stat:`proxy.process.http.4xx_responses`. 422 4235xx 424~~~ 425 426Percentage of all requests made to |TS| which resulted in an HTTP response code 427between 500 and 599 (inclusive) being sent back to the client. 5xx response 428code indicate a server-side error. For a caching proxy like |TS|, these are 429likely to be most often returned for gateway errors; e.g. 502 Bad Gateway (the 430origin server address was invalid or a connection could not be established at 431all due to system or network failures) and 504 Gateway Timeout (the origin 432server was contacted, but did not return data in the time allowed). 433 434Statistic: :ts:stat:`proxy.process.http.5xx_responses`. 435 436Conn Fail 437~~~~~~~~~ 438 439The total number of connections to |TS| which failed to connect properly. 440 441Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.connect_failed`. 442 443Other Err 444~~~~~~~~~ 445 446The total number of |TS| transactions which experienced an error not covered by 447`Conn Fail`_ and `Abort`_. 448 449Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.other`. 450 451Abort 452~~~~~ 453 454The total number of |TS| transactions which were prematurely ended (connection 455was closed before all data had been received and/or sent). 456 457Statistic: :ts:stat:`proxy.process.http.transaction_counts.errors.aborts`. 458 459200, 206, 301, 302, 304, 404, 502 460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 461 462The percentage of total |TS| transactions which resulted in the specified HTTP 463response code. For details on the meaning of individual status codes, please 464refer to :ref:`appendix-http-status-codes`. 465 466Statistics: 467:ts:stat:`proxy.process.http.200_responses`, 468:ts:stat:`proxy.process.http.206_responses`, 469:ts:stat:`proxy.process.http.301_responses`, 470:ts:stat:`proxy.process.http.302_responses`, 471:ts:stat:`proxy.process.http.304_responses`, 472:ts:stat:`proxy.process.http.404_responses`, 473:ts:stat:`proxy.process.http.502_responses`. 474 475.. note:: 476 477 |TS| also provides statistics for every other response code. The keen observer 478 will have hopefully already recognized the pattern in statistic names. 479 480100 B, 1 KB, 3 KB, 5 KB, 10 KB, 1 MB, > 1 MB 481~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 482 483Each of these fields indicates the percentage of responses from |TS| which fell 484within a particular size (document body size, excluding response headers). The 485individual fields are the upper bounds of exclusive *buckets*, meaning that a 486response with a document body of 4,500 bytes will be counted in the ``5 KB`` 487field, but not in any of the smaller sizes. 488 489Statistics: 490:ts:stat:`proxy.process.http.response_document_size_100`, 491:ts:stat:`proxy.process.http.response_document_size_1K`, 492:ts:stat:`proxy.process.http.response_document_size_3K`, 493:ts:stat:`proxy.process.http.response_document_size_5K`, 494:ts:stat:`proxy.process.http.response_document_size_10K`, 495:ts:stat:`proxy.process.http.response_document_size_1M`, 496:ts:stat:`proxy.process.http.response_document_size_inf`. 497 498Client 499------ 500 501Requests 502~~~~~~~~ 503 504Total number of client requests serviced by |TS|. This includes both successful 505content-bearing responses as well as errors, redirects, and not-modified IMS 506responses. Additionally, this number includes requests which were serviced from 507the |TS| cache as well as through proxied origin connections. 508 509Statistic: :ts:stat:`proxy.process.http.incoming_requests`. 510 511Req/Conn 512~~~~~~~~ 513 514The average number of requests made per client connection. When Keep-Alive 515support is enabled in |TS| and clients make use of it, they are able to submit 516multiple document requests over a single connection in some situations. This 517number is calculated from the total number of client requests divided by the 518total number of new connections that have been created. 519 520Statistics: 521:ts:stat:`proxy.process.http.incoming_requests`, 522:ts:stat:`proxy.process.http.total_client_connections`. 523 524New Conn 525~~~~~~~~ 526 527The total number of new HTTP connections from clients which have been created 528over the lifetime of the |TS| process. 529 530Statistic: :ts:stat:`proxy.process.http.total_client_connections`. 531 532Curr Conn 533~~~~~~~~~ 534 535The number of currently-open HTTP connections from clients with |TS|. 536 537Statistic: :ts:stat:`proxy.process.http.current_client_connections`. 538 539Active Con 540~~~~~~~~~~ 541 542The number of currently active client connection threads (requests in the 543process of being fulfilled when the statistics snapshot was taken). 544 545Statistic: :ts:stat:`proxy.process.http.current_active_client_connections`. 546 547Dynamic KA 548~~~~~~~~~~ 549 550Statistics: 551:ts:stat:`proxy.process.net.dynamic_keep_alive_timeout_in_total`, 552:ts:stat:`proxy.process.net.dynamic_keep_alive_timeout_in_count`. 553 554Head Bytes 555~~~~~~~~~~ 556 557The total bytes consumed by outgoing server response headers from |TS| to 558clients. 559 560Statistic: :ts:stat:`proxy.process.http.user_agent_response_header_total_size`. 561 562Body Bytes 563~~~~~~~~~~ 564 565The total bytes consumed by outgoing document bodies in responses from |TS| to 566clients. 567 568Statistic: :ts:stat:`proxy.process.http.user_agent_response_document_total_size`. 569 570Avg Size 571~~~~~~~~ 572 573Average size in bytes of combined headers and document bodies for |TS| 574responses to clients. 575 576Statistics: 577:ts:stat:`proxy.process.http.user_agent_response_header_total_size`, 578:ts:stat:`proxy.process.http.user_agent_response_document_total_size`, 579:ts:stat:`proxy.process.http.incoming_requests`. 580 581Net (bits) 582~~~~~~~~~~ 583 584The summed bits (not bytes) of all |TS| responses to clients, whether served 585from the |TS| or through a proxied connection to an origin server. 586 587Statistics: 588:ts:stat:`proxy.process.http.user_agent_response_header_total_size`, 589:ts:stat:`proxy.process.http.user_agent_response_document_total_size`. 590 591Resp (ms) 592~~~~~~~~~ 593 594The average response time by |TS| across all client requests. Response time is 595measured from the moment a client connection is established, until the moment 596the last byte of the response is delivered. This field is simply the result of 597dividing the total time spent by |TS| servicing client requests by the total 598number of those requests. 599 600Statistics: :ts:stat:`proxy.process.http.total_transactions_time`, 601:ts:stat:`proxy.process.http.incoming_requests`. 602 603Origin Server 604------------- 605 606Requests 607~~~~~~~~ 608 609The total number of requests made by |TS| to origin servers, because client 610requests could not be fulfilled by the |TS| cache (for any reason, whether it 611was not present in the cache, was stale or expired, or not cacheable). 612 613Statistic: :ts:stat:`proxy.process.http.outgoing_requests`. 614 615Req/Conn 616~~~~~~~~ 617 618The average number of requests made to origin servers by |TS| per connection. 619This field is simply the result of dividing the total number of requests made 620by the total number of connections that have ever been opened. 621 622Statistics: 623:ts:stat:`proxy.process.http.outgoing_requests`, 624:ts:stat:`proxy.process.http.total_server_connections`. 625 626New Conn 627~~~~~~~~ 628 629The total number of new HTTP connections from |TS| to origin servers that have 630been created during the lifetime of the |TS| process. 631 632Statistic: :ts:stat:`proxy.process.http.total_server_connections`. 633 634Curr Conn 635~~~~~~~~~ 636 637The number of HTTP connections currently open from |TS| to origin servers. Note 638that |TS| maintains a configurable number of origin server connections open at 639all times, whether there are requests being proxied or not, when configured as 640a reverse proxy to a known list of origin servers. This is not the case for 641forward proxy configurations, however, as |TS| has no foreknowledge of the 642servers to which clients may try to connect. 643 644Statistic: :ts:stat:`proxy.process.http.current_server_connections`. 645 646Head Bytes 647~~~~~~~~~~ 648 649The total bytes delivered as headers in responses from origin servers to |TS|. 650 651Statistic: :ts:stat:`proxy.process.http.origin_server_response_header_total_size`. 652 653Body Bytes 654~~~~~~~~~~ 655 656The total bytes delivered as document bodies in responses from origin servers 657to |TS|. 658 659Statistic: :ts:stat:`proxy.process.http.origin_server_response_document_total_size`. 660 661Avg Size 662~~~~~~~~ 663 664The average size of the combined header and document body responses from origin 665servers to |TS|. 666 667Statistics: 668:ts:stat:`proxy.process.http.origin_server_response_header_total_size`, 669:ts:stat:`proxy.process.http.origin_server_response_document_total_size`, 670:ts:stat:`proxy.process.http.outgoing_requests`. 671 672Net (bits) 673~~~~~~~~~~ 674 675The total bits (not bytes) transferred from origin servers to |TS| for proxied 676requests not fulfilled by the |TS| cache. 677 678Statistics: 679:ts:stat:`proxy.process.http.origin_server_response_header_total_size`, 680:ts:stat:`proxy.process.http.origin_server_response_document_total_size`. 681