1.\" Copyright (c) 2001 Matthew Dillon. Terms and conditions are those of 2.\" the BSD Copyright as specified in the file "/usr/src/COPYRIGHT" in 3.\" the source tree. 4.\" 5.\" $FreeBSD: src/share/man/man7/tuning.7,v 1.1.2.30 2002/12/17 19:32:08 dillon Exp $ 6.\" $DragonFly: src/share/man/man7/tuning.7,v 1.7 2006/02/28 02:25:11 swildner Exp $ 7.\" 8.Dd May 25, 2001 9.Dt TUNING 7 10.Os 11.Sh NAME 12.Nm tuning 13.Nd performance tuning under 14.Dx 15.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP 16When using 17.Xr disklabel 8 18or 19.Xr sysinstall 8 20to lay out your filesystems on a hard disk it is important to remember 21that hard drives can transfer data much more quickly from outer tracks 22than they can from inner tracks. 23To take advantage of this you should 24try to pack your smaller filesystems and swap closer to the outer tracks, 25follow with the larger filesystems, and end with the largest filesystems. 26It is also important to size system standard filesystems such that you 27will not be forced to resize them later as you scale the machine up. 28I usually create, in order, a 128M root, 1G swap, 128M 29.Pa /var , 30128M 31.Pa /var/tmp , 323G 33.Pa /usr , 34and use any remaining space for 35.Pa /home . 36.Pp 37You should typically size your swap space to approximately 2x main memory. 38If you do not have a lot of RAM, though, you will generally want a lot 39more swap. 40It is not recommended that you configure any less than 41256M of swap on a system and you should keep in mind future memory 42expansion when sizing the swap partition. 43The kernel's VM paging algorithms are tuned to perform best when there is 44at least 2x swap versus main memory. 45Configuring too little swap can lead 46to inefficiencies in the VM page scanning code as well as create issues 47later on if you add more memory to your machine. 48Finally, on larger systems 49with multiple SCSI disks (or multiple IDE disks operating on different 50controllers), we strongly recommend that you configure swap on each drive 51(up to four drives). 52The swap partitions on the drives should be approximately the same size. 53The kernel can handle arbitrary sizes but 54internal data structures scale to 4 times the largest swap partition. 55Keeping 56the swap partitions near the same size will allow the kernel to optimally 57stripe swap space across the N disks. 58Do not worry about overdoing it a 59little, swap space is the saving grace of 60.Ux 61and even if you do not normally use much swap, it can give you more time to 62recover from a runaway program before being forced to reboot. 63.Pp 64How you size your 65.Pa /var 66partition depends heavily on what you intend to use the machine for. 67This 68partition is primarily used to hold mailboxes, the print spool, and log 69files. 70Some people even make 71.Pa /var/log 72its own partition (but except for extreme cases it is not worth the waste 73of a partition ID). 74If your machine is intended to act as a mail 75or print server, 76or you are running a heavily visited web server, you should consider 77creating a much larger partition \(en perhaps a gig or more. 78It is very easy 79to underestimate log file storage requirements. 80.Pp 81Sizing 82.Pa /var/tmp 83depends on the kind of temporary file usage you think you will need. 84128M is 85the minimum we recommend. 86Also note that sysinstall will create a 87.Pa /tmp 88directory. 89Dedicating a partition for temporary file storage is important for 90two reasons: first, it reduces the possibility of filesystem corruption 91in a crash, and second it reduces the chance of a runaway process that 92fills up 93.Oo Pa /var Oc Ns Pa /tmp 94from blowing up more critical subsystems (mail, 95logging, etc). 96Filling up 97.Oo Pa /var Oc Ns Pa /tmp 98is a very common problem to have. 99.Pp 100In the old days there were differences between 101.Pa /tmp 102and 103.Pa /var/tmp , 104but the introduction of 105.Pa /var 106(and 107.Pa /var/tmp ) 108led to massive confusion 109by program writers so today programs haphazardly use one or the 110other and thus no real distinction can be made between the two. 111So it makes sense to have just one temporary directory and 112softlink to it from the other tmp directory locations. 113However you handle 114.Pa /tmp , 115the one thing you do not want to do is leave it sitting 116on the root partition where it might cause root to fill up or possibly 117corrupt root in a crash/reboot situation. 118.Pp 119The 120.Pa /usr 121partition holds the bulk of the files required to support the system and 122a subdirectory within it called 123.Pa /usr/local 124holds the bulk of the files installed from the 125.Xr ports 7 126hierarchy. 127If you do not use ports all that much and do not intend to keep 128system source 129.Pq Pa /usr/src 130on the machine, you can get away with 131a 1 gigabyte 132.Pa /usr 133partition. 134However, if you install a lot of ports 135(especially window managers and Linux-emulated binaries), we recommend 136at least a 2 gigabyte 137.Pa /usr 138and if you also intend to keep system source 139on the machine, we recommend a 3 gigabyte 140.Pa /usr . 141Do not underestimate the 142amount of space you will need in this partition, it can creep up and 143surprise you! 144.Pp 145The 146.Pa /home 147partition is typically used to hold user-specific data. 148I usually size it to the remainder of the disk. 149.Pp 150Why partition at all? 151Why not create one big 152.Pa / 153partition and be done with it? 154Then I do not have to worry about undersizing things! 155Well, there are several reasons this is not a good idea. 156First, 157each partition has different operational characteristics and separating them 158allows the filesystem to tune itself to those characteristics. 159For example, 160the root and 161.Pa /usr 162partitions are read-mostly, with very little writing, while 163a lot of reading and writing could occur in 164.Pa /var 165and 166.Pa /var/tmp . 167By properly 168partitioning your system fragmentation introduced in the smaller more 169heavily write-loaded partitions will not bleed over into the mostly-read 170partitions. 171Additionally, keeping the write-loaded partitions closer to 172the edge of the disk (i.e. before the really big partitions instead of after 173in the partition table) will increase I/O performance in the partitions 174where you need it the most. 175Now it is true that you might also need I/O 176performance in the larger partitions, but they are so large that shifting 177them more towards the edge of the disk will not lead to a significant 178performance improvement whereas moving 179.Pa /var 180to the edge can have a huge impact. 181Finally, there are safety concerns. 182Having a small neat root partition that 183is essentially read-only gives it a greater chance of surviving a bad crash 184intact. 185.Pp 186Properly partitioning your system also allows you to tune 187.Xr newfs 8 , 188and 189.Xr tunefs 8 190parameters. 191Tuning 192.Xr newfs 8 193requires more experience but can lead to significant improvements in 194performance. 195There are three parameters that are relatively safe to tune: 196.Em blocksize , bytes/i-node , 197and 198.Em cylinders/group . 199.Pp 200.Dx 201performs best when using 8K or 16K filesystem block sizes. 202The default filesystem block size is 16K, 203which provides best performance for most applications, 204with the exception of those that perform random access on large files 205(such as database server software). 206Such applications tend to perform better with a smaller block size, 207although modern disk characteristics are such that the performance 208gain from using a smaller block size may not be worth consideration. 209Using a block size larger than 16K 210can cause fragmentation of the buffer cache and 211lead to lower performance. 212.Pp 213The defaults may be unsuitable 214for a filesystem that requires a very large number of i-nodes 215or is intended to hold a large number of very small files. 216Such a filesystem should be created with an 8K or 4K block size. 217This also requires you to specify a smaller 218fragment size. 219We recommend always using a fragment size that is \(18 220the block size (less testing has been done on other fragment size factors). 221The 222.Xr newfs 8 223options for this would be 224.Dq Li "newfs -f 1024 -b 8192 ..." . 225.Pp 226If a large partition is intended to be used to hold fewer, larger files, such 227as database files, you can increase the 228.Em bytes/i-node 229ratio which reduces the number of i-nodes (maximum number of files and 230directories that can be created) for that partition. 231Decreasing the number 232of i-nodes in a filesystem can greatly reduce 233.Xr fsck 8 234recovery times after a crash. 235Do not use this option 236unless you are actually storing large files on the partition, because if you 237overcompensate you can wind up with a filesystem that has lots of free 238space remaining but cannot accommodate any more files. 239Using 32768, 65536, or 262144 bytes/i-node is recommended. 240You can go higher but 241it will have only incremental effects on 242.Xr fsck 8 243recovery times. 244For example, 245.Dq Li "newfs -i 32768 ..." . 246.Pp 247.Xr tunefs 8 248may be used to further tune a filesystem. 249This command can be run in 250single-user mode without having to reformat the filesystem. 251However, this is possibly the most abused program in the system. 252Many people attempt to 253increase available filesystem space by setting the min-free percentage to 0. 254This can lead to severe filesystem fragmentation and we do not recommend 255that you do this. 256Really the only 257.Xr tunefs 8 258option worthwhile here is turning on 259.Em softupdates 260with 261.Dq Li "tunefs -n enable /filesystem" . 262(Note: in 263.Fx 4.5 264and later, softupdates can be turned on using the 265.Fl U 266option to 267.Xr newfs 8 , 268and 269.Xr sysinstall 8 270will typically enable softupdates automatically for non-root filesystems). 271Softupdates drastically improves meta-data performance, mainly file 272creation and deletion. 273We recommend enabling softupdates on most filesystems; however, there 274are two limitations to softupdates that you should be aware of when 275determining whether to use it on a filesystem. 276First, softupdates guarantees filesystem consistency in the 277case of a crash but could very easily be several seconds (even a minute!) 278behind on pending writes to the physical disk. 279If you crash you may lose more work 280than otherwise. 281Secondly, softupdates delays the freeing of filesystem 282blocks. 283If you have a filesystem (such as the root filesystem) which is 284close to full, doing a major update of it, e.g.\& 285.Dq Li "make installworld" , 286can run it out of space and cause the update to fail. 287For this reason, softupdates will not be enabled on the root filesystem 288during a typical install. There is no loss of performance since the root 289filesystem is rarely written to. 290.Pp 291A number of run-time 292.Xr mount 8 293options exist that can help you tune the system. 294The most obvious and most dangerous one is 295.Cm async . 296Do not ever use it; it is far too dangerous. 297A less dangerous and more 298useful 299.Xr mount 8 300option is called 301.Cm noatime . 302.Ux 303filesystems normally update the last-accessed time of a file or 304directory whenever it is accessed. 305This operation is handled in 306.Dx 307with a delayed write and normally does not create a burden on the system. 308However, if your system is accessing a huge number of files on a continuing 309basis the buffer cache can wind up getting polluted with atime updates, 310creating a burden on the system. 311For example, if you are running a heavily 312loaded web site, or a news server with lots of readers, you might want to 313consider turning off atime updates on your larger partitions with this 314.Xr mount 8 315option. 316However, you should not gratuitously turn off atime 317updates everywhere. 318For example, the 319.Pa /var 320filesystem customarily 321holds mailboxes, and atime (in combination with mtime) is used to 322determine whether a mailbox has new mail. 323You might as well leave 324atime turned on for mostly read-only partitions such as 325.Pa / 326and 327.Pa /usr 328as well. 329This is especially useful for 330.Pa / 331since some system utilities 332use the atime field for reporting. 333.Sh STRIPING DISKS 334In larger systems you can stripe partitions from several drives together 335to create a much larger overall partition. 336Striping can also improve 337the performance of a filesystem by splitting I/O operations across two 338or more disks. 339The 340.Xr vinum 8 341and 342.Xr ccdconfig 8 343utilities may be used to create simple striped filesystems. 344Generally 345speaking, striping smaller partitions such as the root and 346.Pa /var/tmp , 347or essentially read-only partitions such as 348.Pa /usr 349is a complete waste of time. 350You should only stripe partitions that require serious I/O performance, 351typically 352.Pa /var , /home , 353or custom partitions used to hold databases and web pages. 354Choosing the proper stripe size is also 355important. 356Filesystems tend to store meta-data on power-of-2 boundaries 357and you usually want to reduce seeking rather than increase seeking. 358This 359means you want to use a large off-center stripe size such as 1152 sectors 360so sequential I/O does not seek both disks and so meta-data is distributed 361across both disks rather than concentrated on a single disk. 362If 363you really need to get sophisticated, we recommend using a real hardware 364RAID controller from the list of 365.Dx 366supported controllers. 367.Sh SYSCTL TUNING 368.Xr sysctl 8 369variables permit system behavior to be monitored and controlled at 370run-time. 371Some sysctls simply report on the behavior of the system; others allow 372the system behavior to be modified; 373some may be set at boot time using 374.Xr rc.conf 5 , 375but most will be set via 376.Xr sysctl.conf 5 . 377There are several hundred sysctls in the system, including many that appear 378to be candidates for tuning but actually are not. 379In this document we will only cover the ones that have the greatest effect 380on the system. 381.Pp 382The 383.Va kern.ipc.shm_use_phys 384sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on). 385Setting 386this parameter to 1 will cause all System V shared memory segments to be 387mapped to unpageable physical RAM. 388This feature only has an effect if you 389are either (A) mapping small amounts of shared memory across many (hundreds) 390of processes, or (B) mapping large amounts of shared memory across any 391number of processes. 392This feature allows the kernel to remove a great deal 393of internal memory management page-tracking overhead at the cost of wiring 394the shared memory into core, making it unswappable. 395.Pp 396The 397.Va vfs.vmiodirenable 398sysctl defaults to 1 (on). 399This parameter controls how directories are cached 400by the system. 401Most directories are small and use but a single fragment 402(typically 1K) in the filesystem and even less (typically 512 bytes) in 403the buffer cache. 404However, when operating in the default mode the buffer 405cache will only cache a fixed number of directories even if you have a huge 406amount of memory. 407Turning on this sysctl allows the buffer cache to use 408the VM Page Cache to cache the directories. 409The advantage is that all of 410memory is now available for caching directories. 411The disadvantage is that 412the minimum in-core memory used to cache a directory is the physical page 413size (typically 4K) rather than 512 bytes. 414We recommend turning this option off in memory-constrained environments; 415however, when on, it will substantially improve the performance of services 416that manipulate a large number of files. 417Such services can include web caches, large mail systems, and news systems. 418Turning on this option will generally not reduce performance even with the 419wasted memory but you should experiment to find out. 420.Pp 421The 422.Va vfs.write_behind 423sysctl defaults to 1 (on). This tells the filesystem to issue media 424writes as full clusters are collected, which typically occurs when writing 425large sequential files. The idea is to avoid saturating the buffer 426cache with dirty buffers when it would not benefit I/O performance. However, 427this may stall processes and under certain circumstances you may wish to turn 428it off. 429.Pp 430The 431.Va vfs.hirunningspace 432sysctl determines how much outstanding write I/O may be queued to 433disk controllers system wide at any given instance. The default is 434usually sufficient but on machines with lots of disks you may want to bump 435it up to four or five megabytes. Note that setting too high a value 436(exceeding the buffer cache's write threshold) can lead to extremely 437bad clustering performance. Do not set this value arbitrarily high! Also, 438higher write queueing values may add latency to reads occuring at the same 439time. 440.Pp 441There are various other buffer-cache and VM page cache related sysctls. 442We do not recommend modifying these values. 443As of 444.Fx 4.3 , 445the VM system does an extremely good job tuning itself. 446.Pp 447The 448.Va net.inet.tcp.sendspace 449and 450.Va net.inet.tcp.recvspace 451sysctls are of particular interest if you are running network intensive 452applications. 453They control the amount of send and receive buffer space 454allowed for any given TCP connection. 455The default sending buffer is 32K; the default receiving buffer 456is 64K. 457You can often 458improve bandwidth utilization by increasing the default at the cost of 459eating up more kernel memory for each connection. 460We do not recommend 461increasing the defaults if you are serving hundreds or thousands of 462simultaneous connections because it is possible to quickly run the system 463out of memory due to stalled connections building up. 464But if you need 465high bandwidth over a fewer number of connections, especially if you have 466gigabit Ethernet, increasing these defaults can make a huge difference. 467You can adjust the buffer size for incoming and outgoing data separately. 468For example, if your machine is primarily doing web serving you may want 469to decrease the recvspace in order to be able to increase the 470sendspace without eating too much kernel memory. 471Note that the routing table (see 472.Xr route 8 ) 473can be used to introduce route-specific send and receive buffer size 474defaults. 475.Pp 476As an additional management tool you can use pipes in your 477firewall rules (see 478.Xr ipfw 8 ) 479to limit the bandwidth going to or from particular IP blocks or ports. 480For example, if you have a T1 you might want to limit your web traffic 481to 70% of the T1's bandwidth in order to leave the remainder available 482for mail and interactive use. 483Normally a heavily loaded web server 484will not introduce significant latencies into other services even if 485the network link is maxed out, but enforcing a limit can smooth things 486out and lead to longer term stability. 487Many people also enforce artificial 488bandwidth limitations in order to ensure that they are not charged for 489using too much bandwidth. 490.Pp 491Setting the send or receive TCP buffer to values larger then 65535 will result 492in a marginal performance improvement unless both hosts support the window 493scaling extension of the TCP protocol, which is controlled by the 494.Va net.inet.tcp.rfc1323 495sysctl. 496These extensions should be enabled and the TCP buffer size should be set 497to a value larger than 65536 in order to obtain good performance from 498certain types of network links; specifically, gigabit WAN links and 499high-latency satellite links. 500RFC1323 support is enabled by default. 501.Pp 502The 503.Va net.inet.tcp.always_keepalive 504sysctl determines whether or not the TCP implementation should attempt 505to detect dead TCP connections by intermittently delivering 506.Dq keepalives 507on the connection. 508By default, this is enabled for all applications; by setting this 509sysctl to 0, only applications that specifically request keepalives 510will use them. 511In most environments, TCP keepalives will improve the management of 512system state by expiring dead TCP connections, particularly for 513systems serving dialup users who may not always terminate individual 514TCP connections before disconnecting from the network. 515However, in some environments, temporary network outages may be 516incorrectly identified as dead sessions, resulting in unexpectedly 517terminated TCP connections. 518In such environments, setting the sysctl to 0 may reduce the occurrence of 519TCP session disconnections. 520.Pp 521The 522.Va net.inet.tcp.delayed_ack 523TCP feature is largly misunderstood. Historically speaking this feature 524was designed to allow the acknowledgement to transmitted data to be returned 525along with the response. For example, when you type over a remote shell 526the acknowledgement to the character you send can be returned along with the 527data representing the echo of the character. With delayed acks turned off 528the acknowledgement may be sent in its own packet before the remote service 529has a chance to echo the data it just received. This same concept also 530applies to any interactive protocol (e.g. SMTP, WWW, POP3) and can cut the 531number of tiny packets flowing across the network in half. The 532.Dx 533delayed-ack implementation also follows the TCP protocol rule that 534at least every other packet be acknowledged even if the standard 100ms 535timeout has not yet passed. Normally the worst a delayed ack can do is 536slightly delay the teardown of a connection, or slightly delay the ramp-up 537of a slow-start TCP connection. While we aren't sure we believe that 538the several FAQs related to packages such as SAMBA and SQUID which advise 539turning off delayed acks may be refering to the slow-start issue. In 540.Dx 541it would be more beneficial to increase the slow-start flightsize via 542the 543.Va net.inet.tcp.slowstart_flightsize 544sysctl rather then disable delayed acks. 545.Pp 546The 547.Va net.inet.tcp.inflight_enable 548sysctl turns on bandwidth delay product limiting for all TCP connections. 549The system will attempt to calculate the bandwidth delay product for each 550connection and limit the amount of data queued to the network to just the 551amount required to maintain optimum throughput. This feature is useful 552if you are serving data over modems, GigE, or high speed WAN links (or 553any other link with a high bandwidth*delay product), especially if you are 554also using window scaling or have configured a large send window. If 555you enable this option you should also be sure to set 556.Va net.inet.tcp.inflight_debug 557to 0 (disable debugging), and for production use setting 558.Va net.inet.tcp.inflight_min 559to at least 6144 may be beneficial. Note, however, that setting high 560minimums may effectively disable bandwidth limiting depending on the link. 561The limiting feature reduces the amount of data built up in intermediate 562router and switch packet queues as well as reduces the amount of data built 563up in the local host's interface queue. With fewer packets queued up, 564interactive connections, especially over slow modems, will also be able 565to operate with lower round trip times. However, note that this feature 566only effects data transmission (uploading / server-side). It does not 567effect data reception (downloading). 568.Pp 569Adjusting 570.Va net.inet.tcp.inflight_stab 571is not recommended. 572This parameter defaults to 20, representing 2 maximal packets added 573to the bandwidth delay product window calculation. The additional 574window is required to stabilize the algorithm and improve responsiveness 575to changing conditions, but it can also result in higher ping times 576over slow links (though still much lower then you would get without 577the inflight algorithm). In such cases you may 578wish to try reducing this parameter to 15, 10, or 5, and you may also 579have to reduce 580.Va net.inet.tcp.inflight_min 581(for example, to 3500) to get the desired effect. Reducing these parameters 582should be done as a last resort only. 583.Pp 584The 585.Va net.inet.ip.portrange.* 586sysctls control the port number ranges automatically bound to TCP and UDP 587sockets. There are three ranges: A low range, a default range, and a 588high range, selectable via an IP_PORTRANGE setsockopt() call. Most 589network programs use the default range which is controlled by 590.Va net.inet.ip.portrange.first 591and 592.Va net.inet.ip.portrange.last , 593which defaults to 1024 and 5000 respectively. Bound port ranges are 594used for outgoing connections and it is possible to run the system out 595of ports under certain circumstances. This most commonly occurs when you are 596running a heavily loaded web proxy. The port range is not an issue 597when running serves which handle mainly incoming connections such as a 598normal web server, or has a limited number of outgoing connections such 599as a mail relay. For situations where you may run yourself out of 600ports we recommend increasing 601.Va net.inet.ip.portrange.last 602modestly. A value of 10000 or 20000 or 30000 may be reasonable. You should 603also consider firewall effects when changing the port range. Some firewalls 604may block large ranges of ports (usually low-numbered ports) and expect systems 605to use higher ranges of ports for outgoing connections. For this reason 606we do not recommend that 607.Va net.inet.ip.portrange.first 608be lowered. 609.Pp 610The 611.Va kern.ipc.somaxconn 612sysctl limits the size of the listen queue for accepting new TCP connections. 613The default value of 128 is typically too low for robust handling of new 614connections in a heavily loaded web server environment. 615For such environments, 616we recommend increasing this value to 1024 or higher. 617The service daemon 618may itself limit the listen queue size (e.g.\& 619.Xr sendmail 8 , 620apache) but will 621often have a directive in its configuration file to adjust the queue size up. 622Larger listen queues also do a better job of fending off denial of service 623attacks. 624.Pp 625The 626.Va kern.maxfiles 627sysctl determines how many open files the system supports. 628The default is 629typically a few thousand but you may need to bump this up to ten or twenty 630thousand if you are running databases or large descriptor-heavy daemons. 631The read-only 632.Va kern.openfiles 633sysctl may be interrogated to determine the current number of open files 634on the system. 635.Pp 636The 637.Va vm.swap_idle_enabled 638sysctl is useful in large multi-user systems where you have lots of users 639entering and leaving the system and lots of idle processes. 640Such systems 641tend to generate a great deal of continuous pressure on free memory reserves. 642Turning this feature on and adjusting the swapout hysteresis (in idle 643seconds) via 644.Va vm.swap_idle_threshold1 645and 646.Va vm.swap_idle_threshold2 647allows you to depress the priority of pages associated with idle processes 648more quickly then the normal pageout algorithm. 649This gives a helping hand 650to the pageout daemon. 651Do not turn this option on unless you need it, 652because the tradeoff you are making is to essentially pre-page memory sooner 653rather then later, eating more swap and disk bandwidth. 654In a small system 655this option will have a detrimental effect but in a large system that is 656already doing moderate paging this option allows the VM system to stage 657whole processes into and out of memory more easily. 658.Sh LOADER TUNABLES 659Some aspects of the system behavior may not be tunable at runtime because 660memory allocations they perform must occur early in the boot process. 661To change loader tunables, you must set their values in 662.Xr loader.conf 5 663and reboot the system. 664.Pp 665.Va kern.maxusers 666controls the scaling of a number of static system tables, including defaults 667for the maximum number of open files, sizing of network memory resources, etc. 668As of 669.Fx 4.5 , 670.Va kern.maxusers 671is automatically sized at boot based on the amount of memory available in 672the system, and may be determined at run-time by inspecting the value of the 673read-only 674.Va kern.maxusers 675sysctl. 676Some sites will require larger or smaller values of 677.Va kern.maxusers 678and may set it as a loader tunable; values of 64, 128, and 256 are not 679uncommon. 680We do not recommend going above 256 unless you need a huge number 681of file descriptors; many of the tunable values set to their defaults by 682.Va kern.maxusers 683may be individually overridden at boot-time or run-time as described 684elsewhere in this document. 685Systems older than 686.Fx 4.4 687must set this value via the kernel 688.Xr config 8 689option 690.Cd maxusers 691instead. 692.Pp 693.Va kern.ipc.nmbclusters 694may be adjusted to increase the number of network mbufs the system is 695willing to allocate. 696Each cluster represents approximately 2K of memory, 697so a value of 1024 represents 2M of kernel memory reserved for network 698buffers. 699You can do a simple calculation to figure out how many you need. 700If you have a web server which maxes out at 1000 simultaneous connections, 701and each connection eats a 16K receive and 16K send buffer, you need 702approximately 32MB worth of network buffers to deal with it. 703A good rule of 704thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768. 705So for this case 706you would want to set 707.Va kern.ipc.nmbclusters 708to 32768. 709We recommend values between 7101024 and 4096 for machines with moderates amount of memory, and between 4096 711and 32768 for machines with greater amounts of memory. 712Under no circumstances 713should you specify an arbitrarily high value for this parameter, it could 714lead to a boot-time crash. 715The 716.Fl m 717option to 718.Xr netstat 1 719may be used to observe network cluster use. 720Older versions of 721.Fx 722do not have this tunable and require that the 723kernel 724.Xr config 8 725option 726.Dv NMBCLUSTERS 727be set instead. 728.Pp 729More and more programs are using the 730.Xr sendfile 2 731system call to transmit files over the network. 732The 733.Va kern.ipc.nsfbufs 734sysctl controls the number of filesystem buffers 735.Xr sendfile 2 736is allowed to use to perform its work. 737This parameter nominally scales 738with 739.Va kern.maxusers 740so you should not need to modify this parameter except under extreme 741circumstances. 742.Sh KERNEL CONFIG TUNING 743There are a number of kernel options that you may have to fiddle with in 744a large-scale system. 745In order to change these options you need to be 746able to compile a new kernel from source. 747The 748.Xr config 8 749manual page and the handbook are good starting points for learning how to 750do this. 751Generally the first thing you do when creating your own custom 752kernel is to strip out all the drivers and services you do not use. 753Removing things like 754.Dv INET6 755and drivers you do not have will reduce the size of your kernel, sometimes 756by a megabyte or more, leaving more memory available for applications. 757.Pp 758.Dv SCSI_DELAY 759and 760.Dv IDE_DELAY 761may be used to reduce system boot times. 762The defaults are fairly high and 763can be responsible for 15+ seconds of delay in the boot process. 764Reducing 765.Dv SCSI_DELAY 766to 5 seconds usually works (especially with modern drives). 767Reducing 768.Dv IDE_DELAY 769also works but you have to be a little more careful. 770.Pp 771There are a number of 772.Dv *_CPU 773options that can be commented out. 774If you only want the kernel to run 775on a Pentium class CPU, you can easily remove 776.Dv I386_CPU 777and 778.Dv I486_CPU , 779but only remove 780.Dv I586_CPU 781if you are sure your CPU is being recognized as a Pentium II or better. 782Some clones may be recognized as a Pentium or even a 486 and not be able 783to boot without those options. 784If it works, great! 785The operating system 786will be able to better-use higher-end CPU features for MMU, task switching, 787timebase, and even device operations. 788Additionally, higher-end CPUs support 7894MB MMU pages, which the kernel uses to map the kernel itself into memory, 790increasing its efficiency under heavy syscall loads. 791.Sh IDE WRITE CACHING 792.Fx 4.3 793flirted with turning off IDE write caching. 794This reduced write bandwidth 795to IDE disks but was considered necessary due to serious data consistency 796issues introduced by hard drive vendors. 797Basically the problem is that 798IDE drives lie about when a write completes. 799With IDE write caching turned 800on, IDE hard drives will not only write data to disk out of order, they 801will sometimes delay some of the blocks indefinitely under heavy disk 802load. 803A crash or power failure can result in serious filesystem 804corruption. 805So our default was changed to be safe. 806Unfortunately, the 807result was such a huge loss in performance that we caved in and changed the 808default back to on after the release. 809You should check the default on 810your system by observing the 811.Va hw.ata.wc 812sysctl variable. 813If IDE write caching is turned off, you can turn it back 814on by setting the 815.Va hw.ata.wc 816loader tunable to 1. 817More information on tuning the ATA driver system may be found in the 818.Xr ata 4 819man page. 820.Pp 821There is a new experimental feature for IDE hard drives called 822.Va hw.ata.tags 823(you also set this in the boot loader) which allows write caching to be safely 824turned on. 825This brings SCSI tagging features to IDE drives. 826As of this 827writing only IBM DPTA and DTLA drives support the feature. 828Warning! 829These 830drives apparently have quality control problems and I do not recommend 831purchasing them at this time. 832If you need performance, go with SCSI. 833.Sh CPU, MEMORY, DISK, NETWORK 834The type of tuning you do depends heavily on where your system begins to 835bottleneck as load increases. 836If your system runs out of CPU (idle times 837are perpetually 0%) then you need to consider upgrading the CPU or moving to 838an SMP motherboard (multiple CPU's), or perhaps you need to revisit the 839programs that are causing the load and try to optimize them. 840If your system 841is paging to swap a lot you need to consider adding more memory. 842If your 843system is saturating the disk you typically see high CPU idle times and 844total disk saturation. 845.Xr systat 1 846can be used to monitor this. 847There are many solutions to saturated disks: 848increasing memory for caching, mirroring disks, distributing operations across 849several machines, and so forth. 850If disk performance is an issue and you 851are using IDE drives, switching to SCSI can help a great deal. 852While modern 853IDE drives compare with SCSI in raw sequential bandwidth, the moment you 854start seeking around the disk SCSI drives usually win. 855.Pp 856Finally, you might run out of network suds. 857The first line of defense for 858improving network performance is to make sure you are using switches instead 859of hubs, especially these days where switches are almost as cheap. 860Hubs 861have severe problems under heavy loads due to collision backoff and one bad 862host can severely degrade the entire LAN. 863Second, optimize the network path 864as much as possible. 865For example, in 866.Xr firewall 7 867we describe a firewall protecting internal hosts with a topology where 868the externally visible hosts are not routed through it. 869Use 100BaseT rather 870than 10BaseT, or use 1000BaseT rather then 100BaseT, depending on your needs. 871Most bottlenecks occur at the WAN link (e.g.\& 872modem, T1, DSL, whatever). 873If expanding the link is not an option it may be possible to use the 874.Xr dummynet 4 875feature to implement peak shaving or other forms of traffic shaping to 876prevent the overloaded service (such as web services) from affecting other 877services (such as email), or vice versa. 878In home installations this could 879be used to give interactive traffic (your browser, 880.Xr ssh 1 881logins) priority 882over services you export from your box (web services, email). 883.Sh SEE ALSO 884.Xr netstat 1 , 885.Xr systat 1 , 886.Xr ata 4 , 887.Xr dummynet 4 , 888.Xr login.conf 5 , 889.Xr rc.conf 5 , 890.Xr sysctl.conf 5 , 891.Xr firewall 7 , 892.Xr hier 7 , 893.Xr ports 7 , 894.Xr boot 8 , 895.Xr ccdconfig 8 , 896.Xr config 8 , 897.Xr disklabel 8 , 898.Xr fsck 8 , 899.Xr ifconfig 8 , 900.Xr ipfw 8 , 901.Xr loader 8 , 902.Xr mount 8 , 903.Xr newfs 8 , 904.Xr route 8 , 905.Xr sysctl 8 , 906.Xr sysinstall 8 , 907.Xr tunefs 8 , 908.Xr vinum 8 909.Sh HISTORY 910The 911.Nm 912manual page was originally written by 913.An Matthew Dillon 914and first appeared 915in 916.Fx 4.3 , 917May 2001. 918