1.. Licensed to the Apache Software Foundation (ASF) under one 2 or more contributor license agreements. See the NOTICE file 3 distributed with this work for additional information 4 regarding copyright ownership. The ASF licenses this file 5 to you under the Apache License, Version 2.0 (the 6 "License"); you may not use this file except in compliance 7 with the License. You may obtain a copy of the License at 8 9 http://www.apache.org/licenses/LICENSE-2.0 10 11 Unless required by applicable law or agreed to in writing, 12 software distributed under the License is distributed on an 13 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 14 KIND, either express or implied. See the License for the 15 specific language governing permissions and limitations 16 under the License. 17 18.. include:: ../../common.defs 19 20.. _admin-logging-rotation-retention: 21 22Log Rotation and Retention 23************************** 24 25Logging is a nearly indispensable part of any networked service, but especially 26with high traffic installations care needs to be taken to ensure that log files 27don't exhaust storage space and cause maintenance or outage nightmares. 28 29|TS| provides a two-pronged solution: log rotation (also called log rolling) to 30keep individual logs as manageable in size as possible for easier ingestion and 31analysis by humans and other programs, and log retention to keep logs from 32using more space than available and necessary. 33 34This section covers both features. 35 36.. _admin-logging-rotation: 37 38Rotation Options 39================ 40 41|TS| provides automatic log file rolling. At specific intervals during the day 42or when log files reach a certain size, |TS| closes its current set of log 43files and opens new log files. Depending on the amount of traffic your servers 44are exposed to, you may find that increasing the frequency of log rolling is 45beneficial, or even necessary, to maintain manageable log file sets. |TS| nodes 46processing moderately high levels of traffic may want to start by rolling logs 47every six hours, and adjusting from there. 48 49Log file rolling offers the following benefits: 50 51- It defines a consistent interval over which log analysis can be performed. 52 53- It keeps any single log file from becoming too large and helps to 54 keep the logging system within the specified space limits. 55 56- It provides an easy way to identify files that are no longer being 57 used so that an automated script can clean the logging directory and 58 run log analysis programs. 59 60Rolled Log Filename Format 61-------------------------- 62 63|TS| provides a consistent naming scheme for rolled log files that enables you 64to easily identify log files. When |TS| rolls a log file, it saves and closes 65the old file before it starts a new file. |TS| renames the old file to include 66the following information: 67 68- The original log file's name (such as ``access.log``). 69 70- The hostname of the |TS| node that generated the log file. 71 72- Two timestamps separated by a hyphen (``-``). The first timestamp is 73 a *lower bound* for the timestamp of the first record in the log 74 file. The lower bound is the time when the new buffer for log records 75 is created. Under low load, the first timestamp in the filename can 76 be different from the timestamp of the first entry. Under normal 77 load, the first timestamp in the filename and the timestamp of the 78 first entry are similar. The second timestamp is an *upper bound* 79 for the timestamp of the last record in the log file (this is 80 normally the rolling time). 81 82- The suffix ``.old``, which makes it easy for automated scripts to 83 find rolled log files. 84 85Timestamps have the following format: :: 86 87 %Y%M%D.%Hh%Mm%Ss-%Y%M%D.%Hh%Mm%Ss 88 89The following table describes the format: 90 91====== ============================================ ====== 92Format Description Sample 93====== ============================================ ====== 94``%Y`` The year in four-digit format. 2000 95``%M`` The month in two-digit format, from 01-12. 07 96``%D`` The day in two-digit format, from 01-31. 19 97``%H`` The hour in two-digit format, from 00-23. 21 98``%M`` The minute in two-digit format, from 00-59. 52 99``%S`` The second in two-digit format, from 00-59. 36 100====== ============================================ ====== 101 102The following is an example of a rolled log filename: :: 103 104 access.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old 105 106The logging system buffers log records before writing them to disk. When 107a log file is rolled, the log buffer might be partially full. If it is, 108then the first entry in the new log file will have a timestamp earlier 109than the time of rolling. When the new log file is rolled, its first 110timestamp will be a lower bound for the timestamp of the first entry. 111 112For example, suppose logs are rolled every three hours, and the first 113rolled log file is: :: 114 115 access.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old 116 117If the lower bound for the first entry in the log buffer at 3:00:00 is 1182:59:47, then the next log file will have the following timestamp when 119rolled: :: 120 121 access.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old 122 123The contents of a log file are always between the two timestamps. Log 124files do not contain overlapping entries, even if successive timestamps 125appear to overlap. 126 127Rolling Intervals 128----------------- 129 130Log files are rolled at specific intervals relative to a given hour of the day. 131Three options may be used to control when log files are rolled: 132 133- A file size threshold, which will prevent any individual log from growing 134 too large. 135 136- The offset hour, which is an hour between ``0`` (midnight) and ``23``. 137 138- The rolling interval. 139 140Both the offset hour and the rolling interval determine when log file rolling 141starts. Rolling occurs every *rolling interval* and at the *offset* hour. For 142example, if the rolling interval is six hours and the offset hour is ``0`` 143(midnight), then the logs will roll at midnight (00:00), 06:00, 12:00, and 14418:00 each day. If the rolling interval is 12 hours and the offset hour is 145``3``, then logs will roll at 03:00 and 15:00 each day. 146 147To set log file rolling options and/or configure |TS| to roll log files when 148they reach a certain size, adjust the following settings in 149:file:`records.config`: 150 151#. Enable log rolling with :ts:cv:`proxy.config.log.rolling_enabled`. :: 152 153 CONFIG proxy.config.log.rolling_enabled INT 1 154 155#. Configure the upper limit on log file size with 156 :ts:cv:`proxy.config.log.rolling_size_mb`. :: 157 158 CONFIG proxy.config.log.rolling_size_mb INT 1024 159 160#. Set the offset hour with :ts:cv:`proxy.config.log.rolling_offset_hr`. :: 161 162 CONFIG proxy.config.log.rolling_offset_hr INT 0 163 164#. Set the interval (in seconds) with 165 :ts:cv:`proxy.config.log.rolling_interval_sec`. :: 166 167 CONFIG proxy.config.log.rolling_interval_sec INT 21600 168 169#. Set the minimum number of rolled files with 170 :ts:cv:`proxy.config.log.rolling_min_count`. :: 171 172 CONFIG proxy.config.log.rolling_min_count INT 0 173 174#. Run the command :option:`traffic_ctl config reload` to apply the configuration 175 changes. 176 177You can fine-tune log file rolling settings for individual log files in the 178``log.*`` specification in :file:`logging.yaml`. The custom log file uses the 179rolling settings provided in the relevant ``log`` function call, which override 180the default settings you specify in Traffic Manager or :file:`records.config` 181described above. 182 183.. _admin-logging-retention: 184 185Retention Options 186================= 187 188|TS| enables you to control the amount of disk space that the logging directory 189can consume. This allows the system to operate smoothly within a specified 190space window for a long period of time. After you establish a space limit, 191|TS| continues to monitor the space in the logging directory. When the free 192space dwindles to the headroom limit, it enters a low space state and takes the 193following actions: 194 195- If the autodelete option is enabled, then |TS| identifies previously-rolled 196 log files (log files with the ``.old`` extension). It starts deleting files 197 one by one, beginning with the oldest file with largest ratio between current 198 number of files and the minimum rolling count, until it emerges from the low 199 state. The default minimum rolling count of 0 is treated as INT_MAX during 200 ratio calculation. Hence the `rolling_min_count` is not guaranteed to be 201 reserved; instead, it is used as a reference to decide the priority of log 202 files to delete. In low space state, even when all log files are below minimum 203 count, |TS| still tries to delete files until it emerges from the low state. 204 |TS| logs a record of all deleted files in the system error log. 205 206- If the autodelete option is disabled or there are not enough old log files 207 to delete for the system to emerge from its low space state, then |TS| 208 issues a warning and continues logging until the allocated log space is 209 exhausted (which, if configured appropriately, will be well before your 210 actual filesystem space is fully consumed and causes additional problems). 211 At this point, event logging stops even though proxy traffic is still served 212 without client-visible interruption. |TS| resumes event logging when enough 213 space becomes available for it to exit the low space state. To make space 214 available, either explicitly increase the logging space limit or remove 215 files from the logging directory manually. 216 217You can run a :manpage:`cron(8)` script in conjunction with |TS| to 218automatically remove old log files from the logging directory before |TS| 219enters the low space state. Relocate the old log files to a temporary 220partition, where you can run a variety of log analysis scripts. Following 221analysis, either compress the logs and move to an archive location, or simply 222delete them. 223 224|TS| periodically checks the amount of log space used against both the log 225space allocation configured by :ts:cv:`proxy.config.log.max_space_mb_for_logs` 226and the actual amount of space available on the disk partition. The used log 227space is calculated by summing the size of all files present in the logging 228directory and is published in the 229:ts:stat:`proxy.process.log.log_files_space_used` metric. The 230:ts:cv:`proxy.config.log.max_space_mb_headroom` configuration variable 231specifies an amount of headroom that is subtracted from the log space 232allocation. This can be tuned to reduce the risk of completely filling the disk 233partition. 234 235Setting Log File Management Options 236----------------------------------- 237 238To set log management options, follow the steps below: 239 240#. In the :file:`records.config` file, edit the following variables 241 242 - :ts:cv:`proxy.config.log.max_space_mb_for_logs` 243 - :ts:cv:`proxy.config.log.max_space_mb_headroom` 244 245#. Run the command :option:`traffic_ctl config reload` to apply the configuration 246 changes. 247 248 249Retaining Logs For No More Than a Specified Period 250-------------------------------------------------- 251 252If for security reasons logs need to be purged to make sure no log entry remains on the box 253for more then a specified period of time, we could achieve this by setting the rolling interval, 254the maximum number of rolled log files, and forcing |TS| to roll even when there is no traffic. 255 256Let us say we wanted the oldest log entry to be kept on the box to be no older than 2-hour old. 257 258Set :ts:cv:`proxy.config.log.rolling_interval_sec` (yaml: `rolling_interval_sec`) to 3600 (1h) 259which will lead to rolling every 1h. 260 261Set :ts:cv:`proxy.config.log.rolling_max_count` (yaml: `rolling_max_count`) to 1 262which will lead to keeping only one rolled log file at any moment (rolled will be trimmed on every roll). 263 264Set :ts:cv:`proxy.config.log.rolling_allow_empty` (yaml: `rolling_allow_empty`) to 1 (default: 0) 265which will allow logs to be open and rolled even if there was nothing to be logged during the previous period 266(i.e. no requests to |TS|). 267 268The above will ensure logs are rolled every 1h hour, only 1 rolled log file to be kept 269(rest will be trimmed/removed) and logs will be rolling ("moving") even if nothing is logged 270(i.e. no traffic to |TS|). 271 272