1.\" Hey, Emacs, edit this file in -*- nroff-fill -*- mode 2.\"- 3.\" Copyright (c) 1997, 1998 4.\" Nan Yang Computer Services Limited. All rights reserved. 5.\" 6.\" This software is distributed under the so-called ``Berkeley 7.\" License'': 8.\" 9.\" Redistribution and use in source and binary forms, with or without 10.\" modification, are permitted provided that the following conditions 11.\" are met: 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 3. Neither the name of the Company nor the names of its contributors 18.\" may be used to endorse or promote products derived from this software 19.\" without specific prior written permission. 20.\" 21.\" This software is provided ``as is'', and any express or implied 22.\" warranties, including, but not limited to, the implied warranties of 23.\" merchantability and fitness for a particular purpose are disclaimed. 24.\" In no event shall the company or contributors be liable for any 25.\" direct, indirect, incidental, special, exemplary, or consequential 26.\" damages (including, but not limited to, procurement of substitute 27.\" goods or services; loss of use, data, or profits; or business 28.\" interruption) however caused and on any theory of liability, whether 29.\" in contract, strict liability, or tort (including negligence or 30.\" otherwise) arising in any way out of the use of this software, even if 31.\" advised of the possibility of such damage. 32.\" 33.\" $Id: vinum.8,v 1.48 2001/01/15 22:15:05 grog Exp $ 34.\" $FreeBSD: src/sbin/vinum/vinum.8,v 1.33.2.10 2002/12/29 16:35:38 schweikh Exp $ 35.\" 36.Dd August 11, 2007 37.Dt VINUM 8 38.Os 39.Sh NAME 40.Nm vinum 41.Nd Logical Volume Manager control program 42.Sh SYNOPSIS 43.Nm 44.Op Ar command 45.Op Fl options 46.Sh COMMANDS 47.Bl -tag -width indent 48.It Ic attach Ar plex volume Op Cm rename 49.It Xo 50.Ic attach Ar subdisk plex 51.Op Ar offset 52.Op Cm rename 53.Xc 54Attach a plex to a volume, or a subdisk to a plex. 55.It Xo 56.Ic checkparity Ar plex 57.Op Fl f 58.Op Fl v 59.Xc 60Check the parity blocks of a RAID-4 or RAID-5 plex. 61.It Xo 62.Ic concat 63.Op Fl f 64.Op Fl n Ar name 65.Op Fl v 66.Ar drives 67.Xc 68Create a concatenated volume from the specified drives. 69.It Xo 70.Ic create 71.Op Fl f 72.Ar description-file 73.Xc 74Create a volume as described in 75.Ar description-file . 76.It Ic debug 77Cause the volume manager to enter the kernel debugger. 78.It Ic debug Ar flags 79Set debugging flags. 80.It Xo 81.Ic detach 82.Op Fl f 83.Op Ar plex | subdisk 84.Xc 85Detach a plex or subdisk from the volume or plex to which it is attached. 86.It Ic dumpconfig Op Ar drive ... 87List the configuration information stored on the specified drives, or all drives 88in the system if no drive names are specified. 89.It Xo 90.Ic info 91.Op Fl v 92.Op Fl V 93.Xc 94List information about volume manager state. 95.It Xo 96.Ic init 97.Op Fl S Ar size 98.Op Fl w 99.Ar plex | subdisk 100.Xc 101.\" XXX 102Initialize the contents of a subdisk or all the subdisks of a plex to all zeros. 103.It Ic label Ar volume 104Create a volume label. 105.It Xo 106.Ic l | list 107.Op Fl r 108.Op Fl s 109.Op Fl v 110.Op Fl V 111.Op Ar volume | plex | subdisk 112.Xc 113List information about specified objects. 114.It Xo 115.Ic ld 116.Op Fl r 117.Op Fl s 118.Op Fl v 119.Op Fl V 120.Op Ar volume 121.Xc 122List information about drives. 123.It Xo 124.Ic ls 125.Op Fl r 126.Op Fl s 127.Op Fl v 128.Op Fl V 129.Op Ar subdisk 130.Xc 131List information about subdisks. 132.It Xo 133.Ic lp 134.Op Fl r 135.Op Fl s 136.Op Fl v 137.Op Fl V 138.Op Ar plex 139.Xc 140List information about plexes. 141.It Xo 142.Ic lv 143.Op Fl r 144.Op Fl s 145.Op Fl v 146.Op Fl V 147.Op Ar volume 148.Xc 149List information about volumes. 150.It Ic makedev 151Remake the device nodes in 152.Pa /dev/vinum . 153.It Xo 154.Ic mirror 155.Op Fl f 156.Op Fl n Ar name 157.Op Fl s 158.Op Fl v 159.Ar drives 160.Xc 161Create a mirrored volume from the specified drives. 162.It Xo 163.Ic move | mv 164.Fl f 165.Ar drive object ... 166.Xc 167Move the object(s) to the specified drive. 168.It Ic printconfig Op Ar file 169Write a copy of the current configuration to 170.Ar file . 171.It Ic quit 172Exit the 173.Nm 174program when running in interactive mode. Normally this would be done by 175entering the 176.Dv EOF 177character. 178.It Ic read Ar disk ... 179Read the 180.Nm 181configuration from the specified disks. 182.It Xo 183.Ic rename Op Fl r 184.Op Ar drive | subdisk | plex | volume 185.Ar newname 186.Xc 187Change the name of the specified object. 188.\" XXX 189.\".It Ic replace Ar drive newdrive 190.\"Move all the subdisks from the specified drive onto the new drive. 191.It Xo 192.Ic rebuildparity Ar plex Op Fl f 193.Op Fl v 194.Op Fl V 195.Xc 196Rebuild the parity blocks of a RAID-4 or RAID-5 plex. 197.It Ic resetconfig 198Reset the complete 199.Nm 200configuration. 201.It Xo 202.Ic resetstats 203.Op Fl r 204.Op Ar volume | plex | subdisk 205.Xc 206Reset statistics counters for the specified objects, or for all objects if none 207are specified. 208.It Xo 209.Ic rm 210.Op Fl f 211.Op Fl r 212.Ar volume | plex | subdisk 213.Xc 214Remove an object. 215.It Ic saveconfig 216Save 217.Nm 218configuration to disk after configuration failures. 219.\" XXX 220.\".It Xo 221.\".Ic set 222.\".Op Fl f 223.\".Ar state 224.\".Ar volume | plex | subdisk | disk 225.\".Xc 226.\"Set the state of the object to 227.\".Ar state . 228.It Ic setdaemon Op Ar value 229Set daemon configuration. 230.It Xo 231.Ic setstate 232.Ar state 233.Op Ar volume | plex | subdisk | drive 234.Xc 235Set state without influencing other objects, for diagnostic purposes only. 236.It Ic start 237Read configuration from all vinum drives. 238.It Xo 239.Ic start 240.Op Fl i Ar interval 241.Op Fl S Ar size 242.Op Fl w 243.Ar volume | plex | subdisk 244.Xc 245Allow the system to access the objects. 246.It Xo 247.Ic stop 248.Op Fl f 249.Op Ar volume | plex | subdisk 250.Xc 251Terminate access to the objects, or stop 252.Nm 253if no parameters are specified. 254.It Xo 255.Ic stripe 256.Op Fl f 257.Op Fl n Ar name 258.Op Fl v 259.Ar drives 260.Xc 261Create a striped volume from the specified drives. 262.El 263.Sh DESCRIPTION 264.Nm 265is a utility program to communicate with the 266.Xr vinum 4 267logical volume 268manager. 269.Nm 270is designed either for interactive use, when started without command line 271arguments, or to execute a single command if the command is supplied on the 272command line. In interactive mode, 273.Nm 274maintains a command line history. 275.Sh OPTIONS 276.Nm 277commands may optionally be followed by an option. Any of the following options 278may be specified with any command, but in some cases the options are ignored. 279For example, the 280.Ic stop 281command ignores the 282.Fl v 283and 284.Fl V 285options. 286.Bl -tag -width indent 287.It Fl f 288The 289.Fl f 290.Pq Dq force 291option overrides safety checks. Use with extreme care. This option is for 292emergency use only. For example, the command 293.Pp 294.Dl rm -f myvolume 295.Pp 296removes 297.Ar myvolume 298even if it is open. Any subsequent access to the volume will almost certainly 299cause a panic. 300.It Fl i Ar millisecs 301When performing the 302.Ic init 303and 304.Ic start 305commands, wait 306.Ar millisecs 307milliseconds between copying each block. This lowers the load on the system. 308.It Fl n Ar name 309Use the 310.Fl n 311option to specify a volume name to the simplified configuration commands 312.Ic concat , mirror 313and 314.Ic stripe . 315.It Fl r 316The 317.Fl r 318.Pq Dq recursive 319option is used by the list commands to display information not 320only about the specified objects, but also about subordinate objects. For 321example, in conjunction with the 322.Ic lv 323command, the 324.Fl r 325option will also show information about the plexes and subdisks belonging to the 326volume. 327.It Fl s 328The 329.Fl s 330.Pq Dq statistics 331option is used by the list commands to display statistical information. The 332.Ic mirror 333command also uses this option to specify that it should create striped plexes. 334.It Fl S Ar size 335The 336.Fl S 337option specifies the transfer size for the 338.Ic init 339and 340.Ic start 341commands. 342.It Fl v 343The 344.Fl v 345.Pq Dq verbose 346option can be used to request more detailed information. 347.It Fl V 348The 349.Fl V 350.Pq Dq Very verbose 351option can be used to request more detailed information than the 352.Fl v 353option provides. 354.It Fl w 355The 356.Fl w 357.Pq Dq wait 358option tells 359.Nm 360to wait for completion of commands which normally run in the background, such as 361.Ic init . 362.El 363.Sh COMMANDS IN DETAIL 364.Nm 365commands perform the following functions: 366.Pp 367.Bl -tag -width indent -compact 368.It Ic attach Ar plex volume Op Cm rename 369.It Xo 370.Ic attach Ar subdisk plex 371.Op Ar offset 372.Op Cm rename 373.Xc 374.Nm Ic attach 375inserts the specified plex or subdisk in a volume or plex. In the case of a 376subdisk, an offset in the plex may be specified. If it is not, the subdisk will 377be attached at the first possible location. After attaching a plex to a 378non-empty volume, 379.Nm 380reintegrates the plex. 381.Pp 382If the keyword 383.Cm rename 384is specified, 385.Nm 386renames the object (and in the case of a plex, any subordinate subdisks) to fit 387in with the default 388.Nm 389naming convention. To rename the object to any other name, use the 390.Ic rename 391command. 392.Pp 393A number of considerations apply to attaching subdisks: 394.Bl -bullet 395.It 396Subdisks can normally only be attached to concatenated plexes. 397.It 398If a striped or RAID-5 plex is missing a subdisk (for example after drive 399failure), it should be replaced by a subdisk of the same size only. 400.It 401In order to add further subdisks to a striped or RAID-5 plex, use the 402.Fl f 403(force) option. This will corrupt the data in the plex. 404.\"No other attachment of 405.\"subdisks is currently allowed for striped and RAID-5 plexes. 406.It 407For concatenated plexes, the 408.Ar offset 409parameter specifies the offset in blocks from the beginning of the plex. For 410striped and RAID-5 plexes, it specifies the offset of the first block of the 411subdisk: in other words, the offset is the numerical position of the subdisk 412multiplied by the stripe size. For example, in a plex with stripe size 271k, 413the first subdisk will have offset 0, the second offset 271k, the third 542k, 414etc. This calculation ignores parity blocks in RAID-5 plexes. 415.El 416.Pp 417.It Xo 418.Ic checkparity 419.Ar plex 420.Op Fl f 421.Op Fl v 422.Xc 423Check the parity blocks on the specified RAID-4 or RAID-5 plex. This operation 424maintains a pointer in the plex, so it can be stopped and later restarted from 425the same position if desired. In addition, this pointer is used by the 426.Ic rebuildparity 427command, so rebuilding the parity blocks need only start at the location where 428the first parity problem has been detected. 429.Pp 430If the 431.Fl f 432flag is specified, 433.Ic checkparity 434starts checking at the beginning of the plex. If the 435.Fl v 436flag is specified, 437.Ic checkparity 438prints a running progress report. 439.Pp 440.It Xo 441.Ic concat 442.Op Fl f 443.Op Fl n Ar name 444.Op Fl v 445.Ar drives 446.Xc 447The 448.Ic concat 449command provides a simplified alternative to the 450.Ic create 451command for creating volumes with a single concatenated plex. The largest 452contiguous space available on each drive is used to create the subdisks for the 453plexes. 454.Pp 455Normally, the 456.Ic concat 457command creates an arbitrary name for the volume and its components. The name 458is composed of the text 459.Dq Li vinum 460and a small integer, for example 461.Dq Li vinum3 . 462You can override this with the 463.Fl n Ar name 464option, which assigns the name specified to the volume. The plexes and subdisks 465are named after the volume in the default manner. 466.Pp 467There is no choice of name for the drives. If the drives have already been 468initialized as 469.Nm 470drives, the name remains. Otherwise the drives are given names starting with 471the text 472.Dq Li vinumdrive 473and a small integer, for example 474.Dq Li vinumdrive7 . 475As with the 476.Ic create 477command, the 478.Fl f 479option can be used to specify that a previous name should be overwritten. The 480.Fl v 481is used to specify verbose output. 482.Pp 483See the section 484.Sx SIMPLIFIED CONFIGURATION 485below for some examples of this 486command. 487.Pp 488.It Xo 489.Ic create 490.Op Fl f 491.Ar description-file 492.Xc 493.Nm Ic create 494is used to create any object. In view of the relatively complicated 495relationship and the potential dangers involved in creating a 496.Nm 497object, there is no interactive interface to this function. If you do not 498specify a file name, 499.Nm 500starts an editor on a temporary file. If the environment variable 501.Ev EDITOR 502is set, 503.Nm 504starts this editor. If not, it defaults to 505.Nm vi . 506See the section 507.Sx CONFIGURATION FILE 508below for more information on the format of 509this file. 510.Pp 511Note that the 512.Nm Ic create 513function is additive: if you run it multiple times, you will create multiple 514copies of all unnamed objects. 515.Pp 516Normally the 517.Ic create 518command will not change the names of existing 519.Nm 520drives, in order to avoid accidentally erasing them. The correct way to dispose 521of no longer wanted 522.Nm 523drives is to reset the configuration with the 524.Ic resetconfig 525command. In some cases, however, it may be necessary to create new data on 526.Nm 527drives which can no longer be started. In this case, use the 528.Ic create Fl f 529command. 530.Pp 531.It Ic debug 532.Nm Ic debug , 533without any arguments, is used to enter the remote kernel debugger. It is only 534activated if 535.Nm 536is built with the 537.Dv VINUMDEBUG 538option. This option will stop the execution of the operating system until the 539kernel debugger is exited. If remote debugging is set and there is no remote 540connection for a kernel debugger, it will be necessary to reset the system and 541reboot in order to leave the debugger. 542.Pp 543.It Ic debug Ar flags 544Set a bit mask of internal debugging flags. These will change without warning 545as the product matures; to be certain, read the header file 546.Pa /sys/dev/raid/vinum/vinumvar.h . 547The bit mask is composed of the following values: 548.Bl -tag -width indent 549.It Dv DEBUG_ADDRESSES Pq No 1 550Show buffer information during requests 551.\".It Dv DEBUG_NUMOUTPUT Pq No 2 552.\"Show the value of 553.\".Va vp->v_numoutput . 554.It Dv DEBUG_RESID Pq No 4 555Go into debugger in 556.Fn complete_rqe . 557.It Dv DEBUG_LASTREQS Pq No 8 558Keep a circular buffer of last requests. 559.It Dv DEBUG_REVIVECONFLICT Pq No 16 560Print info about revive conflicts. 561.It Dv DEBUG_EOFINFO Pq No 32 562Print information about internal state when returning an 563.Dv EOF 564on a striped plex. 565.It Dv DEBUG_MEMFREE Pq No 64 566Maintain a circular list of the last memory areas freed by the memory allocator. 567.It Dv DEBUG_REMOTEGDB Pq No 256 568Go into remote 569.Nm gdb 570when the 571.Ic debug 572command is issued. 573.It Dv DEBUG_WARNINGS Pq No 512 574Print some warnings about minor problems in the implementation. 575.El 576.Pp 577.It Ic detach Oo Fl f Oc Ar plex 578.It Ic detach Oo Fl f Oc Ar subdisk 579.Nm Ic detach 580removes the specified plex or subdisk from the volume or plex to which it is 581attached. If removing the object would impair the data integrity of the volume, 582the operation will fail unless the 583.Fl f 584option is specified. If the object is named after the object above it (for 585example, subdisk 586.Li vol1.p7.s0 587attached to plex 588.Li vol1.p7 ) , 589the name will be changed 590by prepending the text 591.Dq Li ex- 592(for example, 593.Li ex-vol1.p7.s0 ) . 594If necessary, the name will be truncated in the 595process. 596.Pp 597.Ic detach 598does not reduce the number of subdisks in a striped or RAID-5 plex. Instead, 599the subdisk is marked absent, and can later be replaced with the 600.Ic attach 601command. 602.Pp 603.It Ic dumpconfig Op Ar drive ... 604.Pp 605.Nm Ic dumpconfig 606shows the configuration information stored on the specified drives. If no drive 607names are specified, 608.Ic dumpconfig 609searches all drives on the system for Vinum partitions and dumps the 610information. If configuration updates are disabled, it is possible that this 611information is not the same as the information returned by the 612.Ic list 613command. This command is used primarily for maintenance and debugging. 614.Pp 615.It Ic info 616.Nm Ic info 617displays information about 618.Nm 619memory usage. This is intended primarily for debugging. With the 620.Fl v 621option, it will give detailed information about the memory areas in use. 622.Pp 623With the 624.Fl V 625option, 626.Ic info 627displays information about the last up to 64 I/O requests handled by the 628.Nm 629driver. This information is only collected if debug flag 8 is set. The format 630looks like: 631.Bd -literal 632vinum -> info -V 633Flags: 0x200 1 opens 634Total of 38 blocks malloced, total memory: 16460 635Maximum allocs: 56, malloc table at 0xf0f72dbc 636 637Time Event Buf Dev Offset Bytes SD SDoff Doffset Goffset 638 63914:40:00.637758 1VS Write 0xf2361f40 91.3 0x10 16384 64014:40:00.639280 2LR Write 0xf2361f40 91.3 0x10 16384 64114:40:00.639294 3RQ Read 0xf2361f40 4.39 0x104109 8192 19 0 0 0 64214:40:00.639455 3RQ Read 0xf2361f40 4.23 0xd2109 8192 17 0 0 0 64314:40:00.639529 3RQ Read 0xf2361f40 4.15 0x6e109 8192 16 0 0 0 64414:40:00.652978 4DN Read 0xf2361f40 4.39 0x104109 8192 19 0 0 0 64514:40:00.667040 4DN Read 0xf2361f40 4.15 0x6e109 8192 16 0 0 0 64614:40:00.668556 4DN Read 0xf2361f40 4.23 0xd2109 8192 17 0 0 0 64714:40:00.669777 6RP Write 0xf2361f40 4.39 0x104109 8192 19 0 0 0 64814:40:00.685547 4DN Write 0xf2361f40 4.39 0x104109 8192 19 0 0 0 64911:11:14.975184 Lock 0xc2374210 2 0x1f8001 65011:11:15.018400 7VS Write 0xc2374210 0x7c0 32768 10 65111:11:15.018456 8LR Write 0xc2374210 13.39 0xcc0c9 32768 65211:11:15.046229 Unlock 0xc2374210 2 0x1f8001 653.Ed 654.Pp 655The 656.Ar Buf 657field always contains the address of the user buffer header. This can be used 658to identify the requests associated with a user request, though this is not 100% 659reliable: theoretically two requests in sequence could use the same buffer 660header, though this is not common. The beginning of a request can be identified 661by the event 662.Ar 1VS 663or 664.Ar 7VS . 665The first example above shows the requests involved in a user request. The 666second is a subdisk I/O request with locking. 667.Pp 668The 669.Ar Event 670field contains information related to the sequence of events in the request 671chain. The digit 672.Ar 1 673to 674.Ar 6 675indicates the approximate sequence of events, and the two-letter abbreviation is 676a mnemonic for the location: 677.Bl -tag -width Lockwait 678.It 1VS 679(vinumstrategy) shows information about the user request on entry to 680.Fn vinumstrategy . 681The device number is the 682.Nm 683device, and offset and length are the user parameters. This is always the 684beginning of a request sequence. 685.It 2LR 686(launch_requests) shows the user request just prior to launching the low-level 687.Nm 688requests in the function 689.Fn launch_requests . 690The parameters should be the same as in the 691.Ar 1VS 692information. 693.El 694.Pp 695In the following requests, 696.Ar Dev 697is the device number of the associated disk partition, 698.Ar Offset 699is the offset from the beginning of the partition, 700.Ar SD 701is the subdisk index in 702.Va vinum_conf , 703.Ar SDoff 704is the offset from the beginning of the subdisk, 705.Ar Doffset 706is the offset of the associated data request, and 707.Ar Goffset 708is the offset of the associated group request, where applicable. 709.Bl -tag -width Lockwait 710.It 3RQ 711(request) shows one of possibly several low-level 712.Nm 713requests which are launched to satisfy the high-level request. This information 714is also logged in 715.Fn launch_requests . 716.It 4DN 717(done) is called from 718.Fn complete_rqe , 719showing the completion of a request. This completion should match a request 720launched either at stage 721.Ar 4DN 722from 723.Fn launch_requests , 724or from 725.Fn complete_raid5_write 726at stage 727.Ar 5RD 728or 729.Ar 6RP . 730.It 5RD 731(RAID-5 data) is called from 732.Fn complete_raid5_write 733and represents the data written to a RAID-5 data stripe after calculating 734parity. 735.It 6RP 736(RAID-5 parity) is called from 737.Fn complete_raid5_write 738and represents the data written to a RAID-5 parity stripe after calculating 739parity. 740.It 7VS 741shows a subdisk I/O request. These requests are usually internal to 742.Nm 743for operations like initialization or rebuilding plexes. 744.It 8LR 745shows the low-level operation generated for a subdisk I/O request. 746.It Lockwait 747specifies that the process is waiting for a range lock. The parameters are the 748buffer header associated with the request, the plex number and the block number. 749For internal reasons the block number is one higher than the address of the 750beginning of the stripe. 751.It Lock 752specifies that a range lock has been obtained. The parameters are the same as 753for the range lock. 754.It Unlock 755specifies that a range lock has been released. The parameters are the same as 756for the range lock. 757.El 758.\" XXX 759.Pp 760.It Xo 761.Ic init 762.Op Fl S Ar size 763.Op Fl w 764.Ar plex | subdisk 765.Xc 766.Nm Ic init 767initializes a subdisk by writing zeroes to it. You can initialize all subdisks 768in a plex by specifying the plex name. This is the only way to ensure 769consistent data in a plex. You must perform this initialization before using a 770RAID-5 plex. It is also recommended for other new plexes. 771.Nm 772initializes all subdisks of a plex in parallel. Since this operation can take a 773long time, it is normally performed in the background. If you want to wait for 774completion of the command, use the 775.Fl w 776(wait) option. 777.Pp 778Specify the 779.Fl S 780option if you want to write blocks of a different size from the default value of 78116 kB. 782.Nm 783prints a console message when the initialization is complete. 784.Pp 785.It Ic label Ar volume 786The 787.Ic label 788command writes a 789.Xr UFS 5 790style volume label on a volume. It is a simple alternative to an appropriate 791call to 792.Ic disklabel . 793This is needed because some 794.Xr UFS 5 795commands still read the disk to find the label instead of using the correct 796.Xr ioctl 2 797call to access it. 798.Nm 799maintains a volume label separately from the volume data, so this command is not 800needed for 801.Xr newfs 8 . 802This command is deprecated. 803.Pp 804.It Xo 805.Ic list 806.Op Fl r 807.Op Fl V 808.Op Ar volume | plex | subdisk 809.Xc 810.It Xo 811.Ic l 812.Op Fl r 813.Op Fl V 814.Op Ar volume | plex | subdisk 815.Xc 816.It Xo 817.Ic ld 818.Op Fl r 819.Op Fl s 820.Op Fl v 821.Op Fl V 822.Op Ar volume 823.Xc 824.It Xo 825.Ic ls 826.Op Fl r 827.Op Fl s 828.Op Fl v 829.Op Fl V 830.Op Ar subdisk 831.Xc 832.It Xo 833.Ic lp 834.Op Fl r 835.Op Fl s 836.Op Fl v 837.Op Fl V 838.Op Ar plex 839.Xc 840.It Xo 841.Ic lv 842.Op Fl r 843.Op Fl s 844.Op Fl v 845.Op Fl V 846.Op Ar volume 847.Xc 848.Ic list 849is used to show information about the specified object. If the argument is 850omitted, information is shown about all objects known to 851.Nm . 852The 853.Ic l 854command is a synonym for 855.Ic list . 856.Pp 857The 858.Fl r 859option relates to volumes and plexes: if specified, it recursively lists 860information for the subdisks and (for a volume) plexes subordinate to the 861objects. The commands 862.Ic lv , lp , ls 863and 864.Ic ld 865list only volumes, plexes, subdisks and drives respectively. This is 866particularly useful when used without parameters. 867.Pp 868The 869.Fl s 870option causes 871.Nm 872to output device statistics, the 873.Fl v 874(verbose) option causes some additional information to be output, and the 875.Fl V 876causes considerable additional information to be output. 877.Pp 878.It Ic makedev 879The 880.Ic makedev 881command removes the directory 882.Pa /dev/vinum 883and recreates it with device nodes 884which reflect the current configuration. This command is not intended for 885general use, and is provided for emergency use only. 886.Pp 887.It Xo 888.Ic mirror 889.Op Fl f 890.Op Fl n Ar name 891.Op Fl s 892.Op Fl v 893.Ar drives 894.Xc 895The 896.Ic mirror 897command provides a simplified alternative to the 898.Ic create 899command for creating mirrored volumes. Without any options, it creates a RAID-1 900(mirrored) volume with two concatenated plexes. The largest contiguous space 901available on each drive is used to create the subdisks for the plexes. The 902first plex is built from the odd-numbered drives in the list, and the second 903plex is built from the even-numbered drives. If the drives are of different 904sizes, the plexes will be of different sizes. 905.Pp 906If the 907.Fl s 908option is provided, 909.Ic mirror 910builds striped plexes with a stripe size of 256 kB. The size of the subdisks in 911each plex is the size of the smallest contiguous storage available on any of the 912drives which form the plex. Again, the plexes may differ in size. 913.Pp 914Normally, the 915.Ic mirror 916command creates an arbitrary name for the volume and its components. The name 917is composed of the text 918.Dq Li vinum 919and a small integer, for example 920.Dq Li vinum3 . 921You can override this with the 922.Fl n Ar name 923option, which assigns the name specified to the volume. The plexes and subdisks 924are named after the volume in the default manner. 925.Pp 926There is no choice of name for the drives. If the drives have already been 927initialized as 928.Nm 929drives, the name remains. Otherwise the drives are given names starting with 930the text 931.Dq Li vinumdrive 932and a small integer, for example 933.Dq Li vinumdrive7 . 934As with the 935.Ic create 936command, the 937.Fl f 938option can be used to specify that a previous name should be overwritten. The 939.Fl v 940is used to specify verbose output. 941.Pp 942See the section 943.Sx SIMPLIFIED CONFIGURATION 944below for some examples of this 945command. 946.Pp 947.It Ic mv Fl f Ar drive object ... 948.It Ic move Fl f Ar drive object ... 949Move all the subdisks from the specified objects onto the new drive. The 950objects may be subdisks, drives or plexes. When drives or plexes are specified, 951all subdisks associated with the object are moved. 952.Pp 953The 954.Fl f 955option is required for this function, since it currently does not preserve the 956data in the subdisk. This functionality will be added at a later date. In this 957form, however, it is suited to recovering a failed disk drive. 958.Pp 959.It Ic printconfig Op Ar file 960Write a copy of the current configuration to 961.Ar file 962in a format that can be used to recreate the 963.Nm 964configuration. Unlike the configuration saved on disk, it includes definitions 965of the drives. If you omit 966.Ar file , 967.Nm 968writes the list to 969.Dv stdout . 970.Pp 971.It Ic quit 972Exit the 973.Nm 974program when running in interactive mode. Normally this would be done by 975entering the 976.Dv EOF 977character. 978.Pp 979.It Ic read Ar disk ... 980The 981.Ic read 982command scans the specified disks for 983.Nm 984partitions containing previously created configuration information. It reads 985the configuration in order from the most recently updated to least recently 986updated configuration. 987.Nm 988maintains an up-to-date copy of all configuration information on each disk 989partition. You must specify all of the slices in a configuration as the 990parameter to this command. 991.Pp 992The 993.Ic read 994command is intended to selectively load a 995.Nm 996configuration on a system which has other 997.Nm 998partitions. If you want to start all partitions on the system, it is easier to 999use the 1000.Ic start 1001command. 1002.Pp 1003If 1004.Nm 1005encounters any errors during this command, it will turn off automatic 1006configuration update to avoid corrupting the copies on disk. This will also 1007happen if the configuration on disk indicates a configuration error (for 1008example, subdisks which do not have a valid space specification). You can turn 1009the updates on again with the 1010.Ic setdaemon 1011and 1012.Ic saveconfig 1013commands. Reset bit 2 (numerical value 4) of the daemon options mask to 1014re-enable configuration saves. 1015.Pp 1016.It Xo 1017.Ic rebuildparity 1018.Ar plex 1019.Op Fl f 1020.Op Fl v 1021.Op Fl V 1022.Xc 1023Rebuild the parity blocks on the specified RAID-4 or RAID-5 plex. This 1024operation maintains a pointer in the plex, so it can be stopped and later 1025restarted from the same position if desired. In addition, this pointer is used 1026by the 1027.Ic checkparity 1028command, so rebuilding the parity blocks need only start at the location where 1029the first parity problem has been detected. 1030.Pp 1031If the 1032.Fl f 1033flag is specified, 1034.Ic rebuildparity 1035starts rebuilding at the beginning of the plex. If the 1036.Fl v 1037flag is specified, 1038.Ic rebuildparity 1039first checks the existing parity blocks prints information about those found to 1040be incorrect before rebuilding. If the 1041.Fl V 1042flag is specified, 1043.Ic rebuildparity 1044prints a running progress report. 1045.Pp 1046.It Xo 1047.Ic rename 1048.Op Fl r 1049.Op Ar drive | subdisk | plex | volume 1050.Ar newname 1051.Xc 1052Change the name of the specified object. If the 1053.Fl r 1054option is specified, subordinate objects will be named by the default rules: 1055plex names will be formed by appending 1056.Li .p Ns Ar number 1057to the volume name, and 1058subdisk names will be formed by appending 1059.Li .s Ns Ar number 1060to the plex name. 1061.\".Pp 1062.\".It Xo 1063.\".Ic replace 1064.\".Ar drive newdrive 1065.\"Move all the subdisks from the specified drive onto the new drive. This will 1066.\"attempt to recover those subdisks that can be recovered, and create the others 1067.\"from scratch. If the new drive lacks the space for this operation, as many 1068.\"subdisks as possible will be fitted onto the drive, and the rest will be left on 1069.\"the original drive. 1070.Pp 1071.It Ic resetconfig 1072The 1073.Ic resetconfig 1074command completely obliterates the 1075.Nm 1076configuration on a system. Use this command only when you want to completely 1077delete the configuration. 1078.Nm 1079will ask for confirmation; you must type in the words 1080.Li "NO FUTURE" 1081exactly as shown: 1082.Bd -unfilled -offset indent 1083.No # Nm Ic resetconfig 1084 1085WARNING! This command will completely wipe out your vinum 1086configuration. All data will be lost. If you really want 1087to do this, enter the text 1088 1089NO FUTURE 1090.No "Enter text ->" Sy "NO FUTURE" 1091Vinum configuration obliterated 1092.Ed 1093.Pp 1094As the message suggests, this is a last-ditch command. Don't use it unless you 1095have an existing configuration which you never want to see again. 1096.Pp 1097.It Xo 1098.Ic resetstats 1099.Op Fl r 1100.Op Ar volume | plex | subdisk 1101.Xc 1102.Nm 1103maintains a number of statistical counters for each object. See the header file 1104.Pa /sys/dev/raid/vinum/vinumvar.h 1105for more information. 1106.\" XXX put it in here when it's finalized 1107Use the 1108.Ic resetstats 1109command to reset these counters. In conjunction with the 1110.Fl r 1111option, 1112.Nm 1113also resets the counters of subordinate objects. 1114.Pp 1115.It Xo 1116.Ic rm 1117.Op Fl f 1118.Op Fl r 1119.Ar volume | plex | subdisk 1120.Xc 1121.Ic rm 1122removes an object from the 1123.Nm 1124configuration. Once an object has been removed, there is no way to recover it. 1125Normally 1126.Nm 1127performs a large amount of consistency checking before removing an object. The 1128.Fl f 1129option tells 1130.Nm 1131to omit this checking and remove the object anyway. Use this option with great 1132care: it can result in total loss of data on a volume. 1133.Pp 1134Normally, 1135.Nm 1136refuses to remove a volume or plex if it has subordinate plexes or subdisks 1137respectively. You can tell 1138.Nm 1139to remove the object anyway by using the 1140.Fl f 1141option, or you can cause 1142.Nm 1143to remove the subordinate objects as well by using the 1144.Fl r 1145(recursive) option. If you remove a volume with the 1146.Fl r 1147option, it will remove both the plexes and the subdisks which belong to the 1148plexes. 1149.Pp 1150.It Ic saveconfig 1151Save the current configuration to disk. Normally this is not necessary, since 1152.Nm 1153automatically saves any change in configuration. If an error occurs on startup, 1154updates will be disabled. When you reenable them with the 1155.Ic setdaemon 1156command, 1157.Nm 1158does not automatically save the configuration to disk. Use this command to save 1159the configuration. 1160.\".Pp 1161.\".It Xo 1162.\".Ic set 1163.\".Op Fl f 1164.\".Ar state 1165.\".Ar volume | plex | subdisk | disk 1166.\".Xc 1167.\".Ic set 1168.\"sets the state of the specified object to one of the valid states (see 1169.\".Sx OBJECT STATES 1170.\"below). Normally 1171.\".Nm 1172.\"performs a large amount of consistency checking before making the change. The 1173.\".Fl f 1174.\"option tells 1175.\".Nm 1176.\"to omit this checking and perform the change anyway. Use this option with great 1177.\"care: it can result in total loss of data on a volume. 1178.Pp 1179.It Ic setdaemon Op Ar value 1180.Ic setdaemon 1181sets a variable bitmask for the 1182.Nm 1183daemon. This command is temporary and will be replaced. Currently, the bit mask 1184may contain the bits 1 (log every action to syslog) and 4 (don't update 1185configuration). Option bit 4 can be useful for error recovery. 1186.Pp 1187.It Xo 1188.Ic setstate Ar state 1189.Op Ar volume | plex | subdisk | drive 1190.Xc 1191.Ic setstate 1192sets the state of the specified objects to the specified state. This bypasses 1193the usual consistency mechanism of 1194.Nm 1195and should be used only for recovery purposes. It is possible to crash the 1196system by incorrect use of this command. 1197.Pp 1198.It Xo 1199.Ic start 1200.Op Fl i Ar interval 1201.Op Fl S Ar size 1202.Op Fl w 1203.Op Ar plex | subdisk 1204.Xc 1205.Ic start 1206starts (brings into the 1207.Em up 1208state) one or more 1209.Nm 1210objects. 1211.Pp 1212If no object names are specified, 1213.Nm 1214scans the disks known to the system for 1215.Nm 1216drives and then reads in the configuration as described under the 1217.Ic read 1218commands. The 1219.Nm 1220drive contains a header with all information about the data stored on the drive, 1221including the names of the other drives which are required in order to represent 1222plexes and volumes. 1223.Pp 1224If 1225.Nm 1226encounters any errors during this command, it will turn off automatic 1227configuration update to avoid corrupting the copies on disk. This will also 1228happen if the configuration on disk indicates a configuration error (for 1229example, subdisks which do not have a valid space specification). You can turn 1230the updates on again with the 1231.Ic setdaemon 1232and 1233.Ic saveconfig 1234command. Reset bit 4 of the daemon options mask to re-enable configuration 1235saves. 1236.Pp 1237If object names are specified, 1238.Nm 1239starts them. Normally this operation is only of use with subdisks. The action 1240depends on the current state of the object: 1241.Bl -bullet 1242.It 1243If the object is already in the 1244.Em up 1245state, 1246.Nm 1247does nothing. 1248.It 1249If the object is a subdisk in the 1250.Em down 1251or 1252.Em reborn 1253states, 1254.Nm 1255changes it to the 1256.Em up 1257state. 1258.It 1259If the object is a subdisk in the 1260.Em empty 1261state, the change depends on the subdisk. If it is part of a plex which is part 1262of a volume which contains other plexes, 1263.Nm 1264places the subdisk in the 1265.Em reviving 1266state and attempts to copy the data from the volume. When the operation 1267completes, the subdisk is set into the 1268.Em up 1269state. If it is part of a plex which is part of a volume which contains no 1270other plexes, or if it is not part of a plex, 1271.Nm 1272brings it into the 1273.Em up 1274state immediately. 1275.It 1276If the object is a subdisk in the 1277.Em reviving 1278state, 1279.Nm 1280continues the revive 1281operation offline. When the operation completes, the subdisk is set into the 1282.Em up 1283state. 1284.El 1285.Pp 1286When a subdisk comes into the 1287.Em up 1288state, 1289.Nm 1290automatically checks the state of any plex and volume to which it may belong and 1291changes their state where appropriate. 1292.Pp 1293If the object is a plex, 1294.Ic start 1295checks the state of the subordinate subdisks (and plexes in the case of a 1296volume) and starts any subdisks which can be started. 1297.Pp 1298To start a plex in a multi-plex volume, the data must be copied from another 1299plex in the volume. Since this frequently takes a long time, it is normally 1300done in the background. If you want to wait for this operation to complete (for 1301example, if you are performing this operation in a script), use the 1302.Fl w 1303option. 1304.Pp 1305Copying data doesn't just take a long time, it can also place a significant load 1306on the system. You can specify the transfer size in bytes or sectors with the 1307.Fl S 1308option, and an interval (in milliseconds) to wait between copying each block with 1309the 1310.Fl i 1311option. Both of these options lessen the load on the system. 1312.Pp 1313.It Xo 1314.Ic stop 1315.Op Fl f 1316.Op Ar volume | plex | subdisk 1317.Xc 1318If no parameters are specified, 1319.Ic stop 1320removes the 1321.Nm 1322KLD and stops 1323.Xr vinum 4 . 1324This can only be done if no objects are active. In particular, the 1325.Fl f 1326option does not override this requirement. Normally, the 1327.Ic stop 1328command writes the current configuration back to the drives before terminating. 1329This will not be possible if configuration updates are disabled, so 1330.Nm 1331will not stop if configuration updates are disabled. You can override this by 1332specifying the 1333.Fl f 1334option. 1335.Pp 1336The 1337.Ic stop 1338command can only work if 1339.Nm 1340has been loaded as a KLD, since it is not possible to unload a statically 1341configured driver. 1342.Nm Ic stop 1343will fail if 1344.Nm 1345is statically configured. 1346.Pp 1347If object names are specified, 1348.Ic stop 1349disables access to the objects. If the objects have subordinate objects, they 1350subordinate objects must either already be inactive (stopped or in error), or 1351the 1352.Fl r 1353and 1354.Fl f 1355options must be specified. This command does not remove the objects from the 1356configuration. They can be accessed again after a 1357.Ic start 1358command. 1359.Pp 1360By default, 1361.Nm 1362does not stop active objects. For example, you cannot stop a plex which is 1363attached to an active volume, and you cannot stop a volume which is open. The 1364.Fl f 1365option tells 1366.Nm 1367to omit this checking and remove the object anyway. Use this option with great 1368care and understanding: used incorrectly, it can result in serious data 1369corruption. 1370.Pp 1371.It Xo 1372.Ic stripe 1373.Op Fl f 1374.Op Fl n Ar name 1375.Op Fl v 1376.Ar drives 1377.Xc 1378The 1379.Ic stripe 1380command provides a simplified alternative to the 1381.Ic create 1382command for creating volumes with a single striped plex. The size of the 1383subdisks is the size of the largest contiguous space available on all the 1384specified drives. The stripe size is fixed at 256 kB. 1385.Pp 1386Normally, the 1387.Ic stripe 1388command creates an arbitrary name for the volume and its components. The name 1389is composed of the text 1390.Dq Li vinum 1391and a small integer, for example 1392.Dq Li vinum3 . 1393You can override this with the 1394.Fl n Ar name 1395option, which assigns the name specified to the volume. The plexes and subdisks 1396are named after the volume in the default manner. 1397.Pp 1398There is no choice of name for the drives. If the drives have already been 1399initialized as 1400.Nm 1401drives, the name remains. Otherwise the drives are given names starting with 1402the text 1403.Dq Li vinumdrive 1404and a small integer, for example 1405.Dq Li vinumdrive7 . 1406As with the 1407.Ic create 1408command, the 1409.Fl f 1410option can be used to specify that a previous name should be overwritten. The 1411.Fl v 1412is used to specify verbose output. 1413.Pp 1414See the section 1415.Sx SIMPLIFIED CONFIGURATION 1416below for some examples of this 1417command. 1418.El 1419.Sh SIMPLIFIED CONFIGURATION 1420This section describes a simplified interface to 1421.Nm 1422configuration using the 1423.Ic concat , 1424.Ic mirror 1425and 1426.Ic stripe 1427commands. These commands create convenient configurations for some more normal 1428situations, but they are not as flexible as the 1429.Ic create 1430command. 1431.Pp 1432See above for the description of the commands. Here are some examples, all 1433performed with the same collection of disks. Note that the first drive, 1434.Pa /dev/da1s0h , 1435is smaller than the others. This has an effect on the sizes chosen for each 1436kind of subdisk. 1437.Pp 1438The following examples all use the 1439.Fl v 1440option to show the commands passed to the system, and also to list the structure 1441of the volume. Without the 1442.Fl v 1443option, these commands produce no output. 1444.Ss Volume with a single concatenated plex 1445Use a volume with a single concatenated plex for the largest possible storage 1446without resilience to drive failures: 1447.Bd -literal 1448vinum -> concat -v /dev/da1s0h /dev/da2s0h /dev/da3s0h /dev/da4s0h 1449volume vinum0 1450 plex name vinum0.p0 org concat 1451drive vinumdrive0 device /dev/da1s0h 1452 sd name vinum0.p0.s0 drive vinumdrive0 size 0 1453drive vinumdrive1 device /dev/da2s0h 1454 sd name vinum0.p0.s1 drive vinumdrive1 size 0 1455drive vinumdrive2 device /dev/da3s0h 1456 sd name vinum0.p0.s2 drive vinumdrive2 size 0 1457drive vinumdrive3 device /dev/da4s0h 1458 sd name vinum0.p0.s3 drive vinumdrive3 size 0 1459V vinum0 State: up Plexes: 1 Size: 2134 MB 1460P vinum0.p0 C State: up Subdisks: 4 Size: 2134 MB 1461S vinum0.p0.s0 State: up PO: 0 B Size: 414 MB 1462S vinum0.p0.s1 State: up PO: 414 MB Size: 573 MB 1463S vinum0.p0.s2 State: up PO: 988 MB Size: 573 MB 1464S vinum0.p0.s3 State: up PO: 1561 MB Size: 573 MB 1465.Ed 1466.Pp 1467In this case, the complete space on all four disks was used, giving a volume 14682134 MB in size. 1469.Ss Volume with a single striped plex 1470A volume with a single striped plex may give better performance than a 1471concatenated plex, but restrictions on striped plexes can mean that the volume 1472is smaller. It will also not be resilient to a drive failure: 1473.Bd -literal 1474vinum -> stripe -v /dev/da1s0h /dev/da2s0h /dev/da3s0h /dev/da4s0h 1475drive vinumdrive0 device /dev/da1s0h 1476drive vinumdrive1 device /dev/da2s0h 1477drive vinumdrive2 device /dev/da3s0h 1478drive vinumdrive3 device /dev/da4s0h 1479volume vinum0 1480 plex name vinum0.p0 org striped 256k 1481 sd name vinum0.p0.s0 drive vinumdrive0 size 849825b 1482 sd name vinum0.p0.s1 drive vinumdrive1 size 849825b 1483 sd name vinum0.p0.s2 drive vinumdrive2 size 849825b 1484 sd name vinum0.p0.s3 drive vinumdrive3 size 849825b 1485V vinum0 State: up Plexes: 1 Size: 1659 MB 1486P vinum0.p0 S State: up Subdisks: 4 Size: 1659 MB 1487S vinum0.p0.s0 State: up PO: 0 B Size: 414 MB 1488S vinum0.p0.s1 State: up PO: 256 kB Size: 414 MB 1489S vinum0.p0.s2 State: up PO: 512 kB Size: 414 MB 1490S vinum0.p0.s3 State: up PO: 768 kB Size: 414 MB 1491.Ed 1492.Pp 1493In this case, the size of the subdisks has been limited to the smallest 1494available disk, so the resulting volume is only 1659 MB in size. 1495.Ss Mirrored volume with two concatenated plexes 1496For more reliability, use a mirrored, concatenated volume: 1497.Bd -literal 1498vinum -> mirror -v -n mirror /dev/da1s0h /dev/da2s0h /dev/da3s0h /dev/da4s0h 1499drive vinumdrive0 device /dev/da1s0h 1500drive vinumdrive1 device /dev/da2s0h 1501drive vinumdrive2 device /dev/da3s0h 1502drive vinumdrive3 device /dev/da4s0h 1503volume mirror setupstate 1504 plex name mirror.p0 org concat 1505 sd name mirror.p0.s0 drive vinumdrive0 size 0b 1506 sd name mirror.p0.s1 drive vinumdrive2 size 0b 1507 plex name mirror.p1 org concat 1508 sd name mirror.p1.s0 drive vinumdrive1 size 0b 1509 sd name mirror.p1.s1 drive vinumdrive3 size 0b 1510V mirror State: up Plexes: 2 Size: 1146 MB 1511P mirror.p0 C State: up Subdisks: 2 Size: 988 MB 1512P mirror.p1 C State: up Subdisks: 2 Size: 1146 MB 1513S mirror.p0.s0 State: up PO: 0 B Size: 414 MB 1514S mirror.p0.s1 State: up PO: 414 MB Size: 573 MB 1515S mirror.p1.s0 State: up PO: 0 B Size: 573 MB 1516S mirror.p1.s1 State: up PO: 573 MB Size: 573 MB 1517.Ed 1518.Pp 1519This example specifies the name of the volume, 1520.Ar mirror . 1521Since one drive is smaller than the others, the two plexes are of different 1522size, and the last 158 MB of the volume is non-resilient. To ensure complete 1523reliability in such a situation, use the 1524.Ic create 1525command to create a volume with 988 MB. 1526.Ss Mirrored volume with two striped plexes 1527Alternatively, use the 1528.Fl s 1529option to create a mirrored volume with two striped plexes: 1530.Bd -literal 1531vinum -> mirror -v -n raid10 -s /dev/da1s0h /dev/da2s0h /dev/da3s0h /dev/da4s0h 1532drive vinumdrive0 device /dev/da1s0h 1533drive vinumdrive1 device /dev/da2s0h 1534drive vinumdrive2 device /dev/da3s0h 1535drive vinumdrive3 device /dev/da4s0h 1536volume raid10 setupstate 1537 plex name raid10.p0 org striped 256k 1538 sd name raid10.p0.s0 drive vinumdrive0 size 849825b 1539 sd name raid10.p0.s1 drive vinumdrive2 size 849825b 1540 plex name raid10.p1 org striped 256k 1541 sd name raid10.p1.s0 drive vinumdrive1 size 1173665b 1542 sd name raid10.p1.s1 drive vinumdrive3 size 1173665b 1543V raid10 State: up Plexes: 2 Size: 1146 MB 1544P raid10.p0 S State: up Subdisks: 2 Size: 829 MB 1545P raid10.p1 S State: up Subdisks: 2 Size: 1146 MB 1546S raid10.p0.s0 State: up PO: 0 B Size: 414 MB 1547S raid10.p0.s1 State: up PO: 256 kB Size: 414 MB 1548S raid10.p1.s0 State: up PO: 0 B Size: 573 MB 1549S raid10.p1.s1 State: up PO: 256 kB Size: 573 MB 1550.Ed 1551.Pp 1552In this case, the usable part of the volume is even smaller, since the first 1553plex has shrunken to match the smallest drive. 1554.Sh CONFIGURATION FILE 1555.Nm 1556requires that all parameters to the 1557.Ic create 1558commands must be in a configuration file. Entries in the configuration file 1559define volumes, plexes and subdisks, and may be in free format, except that each 1560entry must be on a single line. 1561.Ss Scale factors 1562Some configuration file parameters specify a size (lengths, stripe sizes). 1563These values can be specified as bytes, or one of the following scale factors 1564may be appended: 1565.Bl -tag -width indent 1566.It s 1567specifies that the value is a number of sectors of 512 bytes. 1568.It k 1569specifies that the value is a number of kilobytes (1024 bytes). 1570.It m 1571specifies that the value is a number of megabytes (1048576 bytes). 1572.It g 1573specifies that the value is a number of gigabytes (1073741824 bytes). 1574.It b 1575is used for compatibility with 1576.Tn VERITAS . 1577It stands for blocks of 512 bytes. 1578This abbreviation is confusing, since the word 1579.Dq block 1580is used in different 1581meanings, and its use is deprecated. 1582.El 1583.Pp 1584For example, the value 16777216 bytes can also be written as 1585.Em 16m , 1586.Em 16384k 1587or 1588.Em 32768s . 1589.Pp 1590The configuration file can contain the following entries: 1591.Bl -tag -width 4n 1592.It Ic drive Ar name devicename Op Ar options 1593Define a drive. The options are: 1594.Bl -tag -width 18n 1595.It Cm device Ar devicename 1596Specify the device on which the drive resides. 1597.Ar devicename 1598must be the name of a disk partition, for example 1599.Pa /dev/da1s0e 1600or 1601.Pa /dev/ad3s2h , 1602and it must be of type 1603.Em vinum . 1604Do not use the 1605.Dq Li c 1606partition, which is reserved for the complete disk. 1607.It Cm hotspare 1608Define the drive to be a 1609.Dq hot spare 1610drive, which is maintained to automatically replace a failed drive. 1611.Nm 1612does not allow this drive to be used for any other purpose. In particular, it 1613is not possible to create subdisks on it. This functionality has not been 1614completely implemented. 1615.El 1616.It Ic volume Ar name Op Ar options 1617Define a volume with name 1618.Ar name . 1619Options are: 1620.Bl -tag -width 18n 1621.It Cm plex Ar plexname 1622Add the specified plex to the volume. If 1623.Ar plexname 1624is specified as 1625.Cm * , 1626.Nm 1627will look for the definition of the plex as the next possible entry in the 1628configuration file after the definition of the volume. 1629.It Cm readpol Ar policy 1630Define a 1631.Em read policy 1632for the volume. 1633.Ar policy 1634may be either 1635.Cm round 1636or 1637.Cm prefer Ar plexname . 1638.Nm 1639satisfies a read request from only one of the plexes. A 1640.Cm round 1641read policy specifies that each read should be performed from a different plex 1642in 1643.Em round-robin 1644fashion. A 1645.Cm prefer 1646read policy reads from the specified plex every time. 1647.It Cm setupstate 1648When creating a multi-plex volume, assume that the contents of all the plexes 1649are consistent. This is normally not the case, so by default 1650.Nm 1651sets all plexes except the first one to the 1652.Em faulty 1653state. Use the 1654.Ic start 1655command to first bring them to a consistent state. In the case of striped and 1656concatenated plexes, however, it does not normally cause problems to leave them 1657inconsistent: when using a volume for a file system or a swap partition, the 1658previous contents of the disks are not of interest, so they may be ignored. 1659If you want to take this risk, use the 1660.Cm setupstate 1661keyword. It will only apply to the plexes defined immediately after the volume 1662in the configuration file. If you add plexes to a volume at a later time, you 1663must integrate them manually with the 1664.Ic start 1665command. 1666.Pp 1667Note that you 1668.Em must 1669use the 1670.Ic init 1671command with RAID-5 plexes: otherwise extreme data corruption will result if one 1672subdisk fails. 1673.El 1674.It Ic plex Op Ar options 1675Define a plex. Unlike a volume, a plex does not need a name. The options may 1676be: 1677.Bl -tag -width 18n 1678.It Cm name Ar plexname 1679Specify the name of the plex. Note that you must use the keyword 1680.Cm name 1681when naming a plex or subdisk. 1682.It Cm org Ar organization Op Ar stripesize 1683Specify the organization of the plex. 1684.Ar organization 1685can be one of 1686.Cm concat , striped 1687or 1688.Cm raid5 . 1689For 1690.Cm striped 1691and 1692.Cm raid5 1693plexes, the parameter 1694.Ar stripesize 1695must be specified, while for 1696.Cm concat 1697it must be omitted. For type 1698.Cm striped , 1699it specifies the width of each stripe. For type 1700.Cm raid5 , 1701it specifies the size of a group. A group is a portion of a plex which 1702stores the parity bits all in the same subdisk. It must be a factor of the plex size (in 1703other words, the result of dividing the plex size by the stripe size must be an 1704integer), and it must be a multiple of a disk sector (512 bytes). 1705.Pp 1706For optimum performance, stripes should be at least 128 kB in size: anything 1707smaller will result in a significant increase in I/O activity due to mapping of 1708individual requests over multiple disks. The performance improvement due to the 1709increased number of concurrent transfers caused by this mapping will not make up 1710for the performance drop due to the increase in latency. A good guideline for 1711stripe size is between 256 kB and 512 kB. Avoid powers of 2, however: they tend 1712to cause all superblocks to be placed on the first subdisk. 1713.Pp 1714A striped plex must have at least two subdisks (otherwise it is a concatenated 1715plex), and each must be the same size. A RAID-5 plex must have at least three 1716subdisks, and each must be the same size. In practice, a RAID-5 plex should 1717have at least 5 subdisks. 1718.It Cm volume Ar volname 1719Add the plex to the specified volume. If no 1720.Cm volume 1721keyword is specified, the plex will be added to the last volume mentioned in the 1722configuration file. 1723.It Cm sd Ar sdname offset 1724Add the specified subdisk to the plex at offset 1725.Ar offset . 1726.El 1727.It Ic subdisk Op Ar options 1728Define a subdisk. Options may be: 1729.Bl -hang -width 18n 1730.It Cm name Ar name 1731Specify the name of a subdisk. It is not necessary to specify a name for a 1732subdisk. 1733Note that you must specify the keyword 1734.Cm name 1735if you wish to name a subdisk. 1736.It Cm plexoffset Ar offset 1737Specify the starting offset of the subdisk in the plex. If not specified, 1738.Nm 1739allocates the space immediately after the previous subdisk, if any, or otherwise 1740at the beginning of the plex. 1741.It Cm driveoffset Ar offset 1742Specify the starting offset of the subdisk in the drive. If not specified, 1743.Nm 1744allocates the first contiguous 1745.Ar length 1746bytes of free space on the drive. 1747.It Cm length Ar length 1748Specify the length of the subdisk. This keyword must be specified. There is no 1749default, but the value 0 may be specified to mean 1750.Dq "use the largest available contiguous free area on the drive" . 1751If the drive is empty, this means that the entire drive will be used for the 1752subdisk. 1753.Cm length 1754may be shortened to 1755.Cm len . 1756.It Cm plex Ar plex 1757Specify the plex to which the subdisk belongs. By default, the subdisk belongs 1758to the last plex specified. 1759.It Cm drive Ar drive 1760Specify the drive on which the subdisk resides. By default, the subdisk resides 1761on the last drive specified. 1762.El 1763.El 1764.Sh EXAMPLE CONFIGURATION FILE 1765.Bd -literal 1766# Sample vinum configuration file 1767# 1768# Our drives 1769drive drive1 device /dev/da1s0h 1770drive drive2 device /dev/da2s0h 1771drive drive3 device /dev/da3s0h 1772drive drive4 device /dev/da4s0h 1773drive drive5 device /dev/da5s0h 1774drive drive6 device /dev/da6s0h 1775# A volume with one striped plex 1776volume tinyvol 1777 plex org striped 512b 1778 sd length 64m drive drive2 1779 sd length 64m drive drive4 1780volume stripe 1781 plex org striped 512b 1782 sd length 512m drive drive2 1783 sd length 512m drive drive4 1784# Two plexes 1785volume concat 1786 plex org concat 1787 sd length 100m drive drive2 1788 sd length 50m drive drive4 1789 plex org concat 1790 sd length 150m drive drive4 1791# A volume with one striped plex and one concatenated plex 1792volume strcon 1793 plex org striped 512b 1794 sd length 100m drive drive2 1795 sd length 100m drive drive4 1796 plex org concat 1797 sd length 150m drive drive2 1798 sd length 50m drive drive4 1799# a volume with a RAID-5 and a striped plex 1800# note that the RAID-5 volume is longer by 1801# the length of one subdisk 1802volume vol5 1803 plex org striped 64k 1804 sd length 1000m drive drive2 1805 sd length 1000m drive drive4 1806 plex org raid5 32k 1807 sd length 500m drive drive1 1808 sd length 500m drive drive2 1809 sd length 500m drive drive3 1810 sd length 500m drive drive4 1811 sd length 500m drive drive5 1812.Ed 1813.Sh DRIVE LAYOUT CONSIDERATIONS 1814.Nm 1815drives are currently 1816.Bx 1817disk partitions. They must be of type 1818.Em vinum 1819in order to avoid overwriting data used for other purposes. Use 1820.Nm disklabel Fl e 1821to edit a partition type definition. The following display shows a typical 1822partition layout as shown by 1823.Xr disklabel 8 : 1824.Bd -literal 182516 partitions: 1826# size offset fstype 1827 a: 81920 344064 4.2BSD # 40.000M 1828 b: 262144 81920 swap # 128.000M 1829 c: 4226725 0 unused # 2063.830M 1830 e: 81920 0 4.2BSD # 40.000M 1831 f: 1900000 425984 4.2BSD # 927.734M 1832 g: 1900741 2325984 vinum # 928.095M 1833.Ed 1834.Pp 1835In this example, partition 1836.Dq Li g 1837may be used as a 1838.Nm 1839partition. Partitions 1840.Dq Li a , 1841.Dq Li e 1842and 1843.Dq Li f 1844may be used as 1845.Xr UFS 5 1846file systems. 1847Partition 1848.Dq Li b 1849is a swap partition, and partition 1850.Dq Li c 1851represents the whole disk and should not be used for any other purpose. 1852.Pp 1853.Nm 1854uses the first 265 sectors on each partition for configuration information, so 1855the maximum size of a subdisk is 265 sectors smaller than the drive. 1856.Sh LOG FILE 1857.Nm 1858maintains a log file, by default 1859.Pa /var/tmp/vinum_history , 1860in which it keeps track of the commands issued to 1861.Nm . 1862You can override the name of this file by setting the environment variable 1863.Ev VINUM_HISTORY 1864to the name of the file. 1865.Pp 1866Each message in the log file is preceded by a date. The default format is 1867.Qq Li %e %b %Y %H:%M:%S . 1868See 1869.Xr strftime 3 1870for further details of the format string. It can be overridden by the 1871environment variable 1872.Ev VINUM_DATEFORMAT . 1873.Sh HOW TO SET UP VINUM 1874This section gives practical advice about how to implement a 1875.Nm 1876system. 1877.Ss Where to put the data 1878The first choice you need to make is where to put the data. You need dedicated 1879disk partitions for 1880.Nm . 1881They should be partitions, not devices, and they should not be partition 1882.Dq Li c . 1883For example, good names are 1884.Pa /dev/da0s0e 1885or 1886.Pa /dev/ad3s4a . 1887Bad names are 1888.Pa /dev/da0 1889and 1890.Pa /dev/da0s1 , 1891both of which represent a device, not a partition, and 1892.Pa /dev/ad1s0c , 1893which represents a complete disk and should be of type 1894.Em unused . 1895See the example under 1896.Sx DRIVE LAYOUT CONSIDERATIONS 1897above. 1898.Ss Designing volumes 1899The way you set up 1900.Nm 1901volumes depends on your intentions. There are a number of possibilities: 1902.Bl -enum 1903.It 1904You may want to join up a number of small disks to make a reasonable sized file 1905system. For example, if you had five small drives and wanted to use all the 1906space for a single volume, you might write a configuration file like: 1907.Bd -literal -offset indent 1908drive d1 device /dev/da2s0e 1909drive d2 device /dev/da3s0e 1910drive d3 device /dev/da4s0e 1911drive d4 device /dev/da5s0e 1912drive d5 device /dev/da6s0e 1913volume bigger 1914 plex org concat 1915 sd length 0 drive d1 1916 sd length 0 drive d2 1917 sd length 0 drive d3 1918 sd length 0 drive d4 1919 sd length 0 drive d5 1920.Ed 1921.Pp 1922In this case, you specify the length of the subdisks as 0, which means 1923.Dq "use the largest area of free space that you can find on the drive" . 1924If the subdisk is the only subdisk on the drive, it will use all available 1925space. 1926.It 1927You want to set up 1928.Nm 1929to obtain additional resilience against disk failures. You have the choice of 1930RAID-1, also called 1931.Dq mirroring , 1932or RAID-5, also called 1933.Dq parity . 1934.Pp 1935To set up mirroring, create multiple plexes in a volume. For example, to create 1936a mirrored volume of 2 GB, you might create the following configuration file: 1937.Bd -literal -offset indent 1938drive d1 device /dev/da2s0e 1939drive d2 device /dev/da3s0e 1940volume mirror 1941 plex org concat 1942 sd length 2g drive d1 1943 plex org concat 1944 sd length 2g drive d2 1945.Ed 1946.Pp 1947When creating mirrored drives, it is important to ensure that the data from each 1948plex is on a different physical disk so that 1949.Nm 1950can access the complete address space of the volume even if a drive fails. 1951Note that each plex requires as much data as the complete volume: in this 1952example, the volume has a size of 2 GB, but each plex (and each subdisk) 1953requires 2 GB, so the total disk storage requirement is 4 GB. 1954.Pp 1955To set up RAID-5, create a single plex of type 1956.Cm raid5 . 1957For example, to create an equivalent resilient volume of 2 GB, you might use the 1958following configuration file: 1959.Bd -literal -offset indent 1960drive d1 device /dev/da2s0e 1961drive d2 device /dev/da3s0e 1962drive d3 device /dev/da4s0e 1963drive d4 device /dev/da5s0e 1964drive d5 device /dev/da6s0e 1965volume raid 1966 plex org raid5 512k 1967 sd length 512m drive d1 1968 sd length 512m drive d2 1969 sd length 512m drive d3 1970 sd length 512m drive d4 1971 sd length 512m drive d5 1972.Ed 1973.Pp 1974RAID-5 plexes require at least three subdisks, one of which is used for storing 1975parity information and is lost for data storage. The more disks you use, the 1976greater the proportion of the disk storage can be used for data storage. In 1977this example, the total storage usage is 2.5 GB, compared to 4 GB for a mirrored 1978configuration. If you were to use the minimum of only three disks, you would 1979require 3 GB to store the information, for example: 1980.Bd -literal -offset indent 1981drive d1 device /dev/da2s0e 1982drive d2 device /dev/da3s0e 1983drive d3 device /dev/da4s0e 1984volume raid 1985 plex org raid5 512k 1986 sd length 1g drive d1 1987 sd length 1g drive d2 1988 sd length 1g drive d3 1989.Ed 1990.Pp 1991As with creating mirrored drives, it is important to ensure that the data from 1992each subdisk is on a different physical disk so that 1993.Nm 1994can access the complete address space of the volume even if a drive fails. 1995.It 1996You want to set up 1997.Nm 1998to allow more concurrent access to a file system. In many cases, access to a 1999file system is limited by the speed of the disk. By spreading the volume across 2000multiple disks, you can increase the throughput in multi-access environments. 2001This technique shows little or no performance improvement in single-access 2002environments. 2003.Nm 2004uses a technique called 2005.Dq striping , 2006or sometimes RAID-0, to increase this concurrency of access. The name RAID-0 is 2007misleading: striping does not provide any redundancy or additional reliability. 2008In fact, it decreases the reliability, since the failure of a single disk will 2009render the volume useless, and the more disks you have, the more likely it is 2010that one of them will fail. 2011.Pp 2012To implement striping, use a 2013.Cm striped 2014plex: 2015.Bd -literal -offset indent 2016drive d1 device /dev/da2s0e 2017drive d2 device /dev/da3s0e 2018drive d3 device /dev/da4s0e 2019drive d4 device /dev/da5s0e 2020volume raid 2021 plex org striped 512k 2022 sd length 512m drive d1 2023 sd length 512m drive d2 2024 sd length 512m drive d3 2025 sd length 512m drive d4 2026.Ed 2027.Pp 2028A striped plex must have at least two subdisks, but the increase in performance 2029is greater if you have a larger number of disks. 2030.It 2031You may want to have the best of both worlds and have both resilience and 2032performance. This is sometimes called RAID-10 (a combination of RAID-1 and 2033RAID-0), though again this name is misleading. With 2034.Nm 2035you can do this with the following configuration file: 2036.Bd -literal -offset indent 2037drive d1 device /dev/da2s0e 2038drive d2 device /dev/da3s0e 2039drive d3 device /dev/da4s0e 2040drive d4 device /dev/da5s0e 2041volume raid setupstate 2042 plex org striped 512k 2043 sd length 512m drive d1 2044 sd length 512m drive d2 2045 sd length 512m drive d3 2046 sd length 512m drive d4 2047 plex org striped 512k 2048 sd length 512m drive d4 2049 sd length 512m drive d3 2050 sd length 512m drive d2 2051 sd length 512m drive d1 2052.Ed 2053.Pp 2054Here the plexes are striped, increasing performance, and there are two of them, 2055increasing reliability. Note that this example shows the subdisks of the second 2056plex in reverse order from the first plex. This is for performance reasons and 2057will be discussed below. In addition, the volume specification includes the 2058keyword 2059.Cm setupstate , 2060which ensures that all plexes are 2061.Em up 2062after creation. 2063.El 2064.Ss Creating the volumes 2065Once you have created your configuration files, start 2066.Nm 2067and create the volumes. In this example, the configuration is in the file 2068.Pa configfile : 2069.Bd -literal -offset 2n 2070# vinum create -v configfile 2071 1: drive d1 device /dev/da2s0e 2072 2: drive d2 device /dev/da3s0e 2073 3: volume mirror 2074 4: plex org concat 2075 5: sd length 2g drive d1 2076 6: plex org concat 2077 7: sd length 2g drive d2 2078Configuration summary 2079 2080Drives: 2 (4 configured) 2081Volumes: 1 (4 configured) 2082Plexes: 2 (8 configured) 2083Subdisks: 2 (16 configured) 2084 2085Drive d1: Device /dev/da2s0e 2086 Created on vinum.lemis.com at Tue Mar 23 12:30:31 1999 2087 Config last updated Tue Mar 23 14:30:32 1999 2088 Size: 60105216000 bytes (57320 MB) 2089 Used: 2147619328 bytes (2048 MB) 2090 Available: 57957596672 bytes (55272 MB) 2091 State: up 2092 Last error: none 2093Drive d2: Device /dev/da3s0e 2094 Created on vinum.lemis.com at Tue Mar 23 12:30:32 1999 2095 Config last updated Tue Mar 23 14:30:33 1999 2096 Size: 60105216000 bytes (57320 MB) 2097 Used: 2147619328 bytes (2048 MB) 2098 Available: 57957596672 bytes (55272 MB) 2099 State: up 2100 Last error: none 2101 2102Volume mirror: Size: 2147483648 bytes (2048 MB) 2103 State: up 2104 Flags: 2105 2 plexes 2106 Read policy: round robin 2107 2108Plex mirror.p0: Size: 2147483648 bytes (2048 MB) 2109 Subdisks: 1 2110 State: up 2111 Organization: concat 2112 Part of volume mirror 2113Plex mirror.p1: Size: 2147483648 bytes (2048 MB) 2114 Subdisks: 1 2115 State: up 2116 Organization: concat 2117 Part of volume mirror 2118 2119Subdisk mirror.p0.s0: 2120 Size: 2147483648 bytes (2048 MB) 2121 State: up 2122 Plex mirror.p0 at offset 0 2123 2124Subdisk mirror.p1.s0: 2125 Size: 2147483648 bytes (2048 MB) 2126 State: up 2127 Plex mirror.p1 at offset 0 2128.Ed 2129.Pp 2130The 2131.Fl v 2132option tells 2133.Nm 2134to list the file as it configures. Subsequently it lists the current 2135configuration in the same format as the 2136.Ic list Fl v 2137command. 2138.Ss Creating more volumes 2139Once you have created the 2140.Nm 2141volumes, 2142.Nm 2143keeps track of them in its internal configuration files. You do not need to 2144create them again. In particular, if you run the 2145.Ic create 2146command again, you will create additional objects: 2147.Bd -literal 2148# vinum create sampleconfig 2149Configuration summary 2150 2151Drives: 2 (4 configured) 2152Volumes: 1 (4 configured) 2153Plexes: 4 (8 configured) 2154Subdisks: 4 (16 configured) 2155 2156D d1 State: up Device /dev/da2s0e Avail: 53224/57320 MB (92%) 2157D d2 State: up Device /dev/da3s0e Avail: 53224/57320 MB (92%) 2158 2159V mirror State: up Plexes: 4 Size: 2048 MB 2160 2161P mirror.p0 C State: up Subdisks: 1 Size: 2048 MB 2162P mirror.p1 C State: up Subdisks: 1 Size: 2048 MB 2163P mirror.p2 C State: up Subdisks: 1 Size: 2048 MB 2164P mirror.p3 C State: up Subdisks: 1 Size: 2048 MB 2165 2166S mirror.p0.s0 State: up PO: 0 B Size: 2048 MB 2167S mirror.p1.s0 State: up PO: 0 B Size: 2048 MB 2168S mirror.p2.s0 State: up PO: 0 B Size: 2048 MB 2169S mirror.p3.s0 State: up PO: 0 B Size: 2048 MB 2170.Ed 2171.Pp 2172As this example (this time with the 2173.Fl f 2174option) shows, re-running the 2175.Ic create 2176has created four new plexes, each with a new subdisk. If you want to add other 2177volumes, create new configuration files for them. They do not need to reference 2178the drives that 2179.Nm 2180already knows about. For example, to create a volume 2181.Pa raid 2182on the four drives 2183.Pa /dev/da1s0e , /dev/da2s0e , /dev/da3s0e 2184and 2185.Pa /dev/da4s0e , 2186you only need to mention the other two: 2187.Bd -literal -offset indent 2188drive d3 device /dev/da1s0e 2189drive d4 device /dev/da4s0e 2190volume raid 2191 plex org raid5 512k 2192 sd size 2g drive d1 2193 sd size 2g drive d2 2194 sd size 2g drive d3 2195 sd size 2g drive d4 2196.Ed 2197.Pp 2198With this configuration file, we get: 2199.Bd -literal 2200# vinum create newconfig 2201Configuration summary 2202 2203Drives: 4 (4 configured) 2204Volumes: 2 (4 configured) 2205Plexes: 5 (8 configured) 2206Subdisks: 8 (16 configured) 2207 2208D d1 State: up Device /dev/da2s0e Avail: 51176/57320 MB (89%) 2209D d2 State: up Device /dev/da3s0e Avail: 53220/57320 MB (89%) 2210D d3 State: up Device /dev/da1s0e Avail: 53224/57320 MB (92%) 2211D d4 State: up Device /dev/da4s0e Avail: 53224/57320 MB (92%) 2212 2213V mirror State: down Plexes: 4 Size: 2048 MB 2214V raid State: down Plexes: 1 Size: 6144 MB 2215 2216P mirror.p0 C State: init Subdisks: 1 Size: 2048 MB 2217P mirror.p1 C State: init Subdisks: 1 Size: 2048 MB 2218P mirror.p2 C State: init Subdisks: 1 Size: 2048 MB 2219P mirror.p3 C State: init Subdisks: 1 Size: 2048 MB 2220P raid.p0 R5 State: init Subdisks: 4 Size: 6144 MB 2221 2222S mirror.p0.s0 State: up PO: 0 B Size: 2048 MB 2223S mirror.p1.s0 State: up PO: 0 B Size: 2048 MB 2224S mirror.p2.s0 State: up PO: 0 B Size: 2048 MB 2225S mirror.p3.s0 State: up PO: 0 B Size: 2048 MB 2226S raid.p0.s0 State: empty PO: 0 B Size: 2048 MB 2227S raid.p0.s1 State: empty PO: 512 kB Size: 2048 MB 2228S raid.p0.s2 State: empty PO: 1024 kB Size: 2048 MB 2229S raid.p0.s3 State: empty PO: 1536 kB Size: 2048 MB 2230.Ed 2231.Pp 2232Note the size of the RAID-5 plex: it is only 6 GB, although together its 2233components use 8 GB of disk space. This is because the equivalent of one 2234subdisk is used for storing parity data. 2235.Ss Restarting Vinum 2236On rebooting the system, start 2237.Nm 2238with the 2239.Ic start 2240command: 2241.Pp 2242.Dl "# vinum start" 2243.Pp 2244This will start all the 2245.Nm 2246drives in the system. If for some reason you wish to start only some of them, 2247use the 2248.Ic read 2249command. 2250.Ss Performance considerations 2251A number of misconceptions exist about how to set up a RAID array for best 2252performance. In particular, most systems use far too small a stripe size. The 2253following discussion applies to all RAID systems, not just to 2254.Nm . 2255.Pp 2256The 2257.Dx 2258block I/O system issues requests of between .5kB and 128 kB; a 2259typical mix is somewhere round 8 kB. You can't stop any striping system from 2260breaking a request into two physical requests, and if you make the stripe small 2261enough, it can be broken into several. This will result in a significant drop 2262in performance: the decrease in transfer time per disk is offset by the order of 2263magnitude greater increase in latency. 2264.Pp 2265With modern disk sizes and the 2266.Dx 2267I/O system, you can expect to have a 2268reasonably small number of fragmented requests with a stripe size between 256 kB 2269and 512 kB; with correct RAID implementations there is no obvious reason not to 2270increase the size to 2 or 4 MB on a large disk. 2271.Pp 2272When choosing a stripe size, consider that most current 2273.Xr UFS 5 2274file systems have 2275cylinder groups 32 MB in size. If you have a stripe size and number of disks 2276both of which are a power of two, it is probable that all superblocks and inodes 2277will be placed on the same subdisk, which will impact performance significantly. 2278Choose an odd number instead, for example 479 kB. 2279.Pp 2280The easiest way to consider the impact of any transfer in a multi-access system 2281is to look at it from the point of view of the potential bottleneck, the disk 2282subsystem: how much total disk time does the transfer use? 2283Since just about 2284everything is cached, the time relationship between the request and its 2285completion is not so important: the important parameter is the total time that 2286the request keeps the disks active, the time when the disks are not available to 2287perform other transfers. As a result, it doesn't really matter if the transfers 2288are happening at the same time or different times. In practical terms, the time 2289we're looking at is the sum of the total latency (positioning time and 2290rotational latency, or the time it takes for the data to arrive under the disk 2291heads) and the total transfer time. For a given transfer to disks of the same 2292speed, the transfer time depends only on the total size of the transfer. 2293.Pp 2294Consider a typical news article or web page of 24 kB, which will probably be 2295read in a single I/O. Take disks with a transfer rate of 6 MB/s and an average 2296positioning time of 8 ms, and a file system with 4 kB blocks. Since it's 24 kB, 2297we don't have to worry about fragments, so the file will start on a 4 kB 2298boundary. The number of transfers required depends on where the block starts: 2299it's (S + F - 1) / S, where S is the stripe size in file system blocks, and F is 2300the file size in file system blocks. 2301.Bl -enum 2302.It 2303Stripe size of 4 kB. You'll have 6 transfers. Total subsystem load: 48 ms 2304latency, 2 ms transfer, 50 ms total. 2305.It 2306Stripe size of 8 kB. On average, you'll have 3.5 transfers. Total subsystem 2307load: 28 ms latency, 2 ms transfer, 30 ms total. 2308.It 2309Stripe size of 16 kB. On average, you'll have 2.25 transfers. Total subsystem 2310load: 18 ms latency, 2 ms transfer, 20 ms total. 2311.It 2312Stripe size of 256 kB. On average, you'll have 1.08 transfers. Total subsystem 2313load: 8.6 ms latency, 2 ms transfer, 10.6 ms total. 2314.It 2315Stripe size of 4 MB. On average, you'll have 1.0009 transfers. Total subsystem 2316load: 8.01 ms latency, 2 ms transfer, 10.01 ms total. 2317.El 2318.Pp 2319It appears that some hardware RAID systems have problems with large stripes: 2320they appear to always transfer a complete stripe to or from disk, so that a 2321large stripe size will have an adverse effect on performance. 2322.Nm 2323does not suffer from this problem: it optimizes all disk transfers and does not 2324transfer unneeded data. 2325.Pp 2326Note that no well-known benchmark program tests true multi-access conditions 2327(more than 100 concurrent users), so it is difficult to demonstrate the validity 2328of these statements. 2329.Pp 2330Given these considerations, the following factors affect the performance of a 2331.Nm 2332volume: 2333.Bl -bullet 2334.It 2335Striping improves performance for multiple access only, since it increases the 2336chance of individual requests being on different drives. 2337.It 2338Concatenating 2339.Xr UFS 5 2340file systems across multiple drives can also improve 2341performance for multiple file access, since 2342.Xr UFS 5 2343divides a file system into 2344cylinder groups and attempts to keep files in a single cylinder group. In 2345general, it is not as effective as striping. 2346.It 2347Mirroring can improve multi-access performance for reads, since by default 2348.Nm 2349issues consecutive reads to consecutive plexes. 2350.It 2351Mirroring decreases performance for all writes, whether multi-access or single 2352access, since the data must be written to both plexes. This explains the 2353subdisk layout in the example of a mirroring configuration above: if the 2354corresponding subdisk in each plex is on a different physical disk, the write 2355commands can be issued in parallel, whereas if they are on the same physical 2356disk, they will be performed sequentially. 2357.It 2358RAID-5 reads have essentially the same considerations as striped reads, unless 2359the striped plex is part of a mirrored volume, in which case the performance of 2360the mirrored volume will be better. 2361.It 2362RAID-5 writes are approximately 25% of the speed of striped writes: to perform 2363the write, 2364.Nm 2365must first read the data block and the corresponding parity block, perform some 2366calculations and write back the parity block and the data block, four times as 2367many transfers as for writing a striped plex. On the other hand, this is offset 2368by the cost of mirroring, so writes to a volume with a single RAID-5 plex are 2369approximately half the speed of writes to a correctly configured volume with two 2370striped plexes. 2371.It 2372When the 2373.Nm 2374configuration changes (for example, adding or removing objects, or the change of 2375state of one of the objects), 2376.Nm 2377writes up to 128 kB of updated configuration to each drive. The larger the 2378number of drives, the longer this takes. 2379.El 2380.Ss Creating file systems on Vinum volumes 2381You do not need to run 2382.Xr disklabel 8 2383before creating a file system on a 2384.Nm 2385volume. Just run 2386.Xr newfs 8 . 2387Use the 2388.Fl v 2389option to state that the device is not divided into partitions. For example, to 2390create a file system on volume 2391.Pa mirror , 2392enter the following command: 2393.Pp 2394.Dl "# newfs -v /dev/vinum/mirror" 2395.Pp 2396A number of other considerations apply to 2397.Nm 2398configuration: 2399.Bl -bullet 2400.It 2401There is no advantage in creating multiple drives on a single disk. Each drive 2402uses 131.5 kB of data for label and configuration information, and performance 2403will suffer when the configuration changes. Use appropriately sized subdisks instead. 2404.It 2405It is possible to increase the size of a concatenated 2406.Nm 2407plex, but currently the size of striped and RAID-5 plexes cannot be increased. 2408Currently the size of an existing 2409.Xr UFS 5 2410file system also cannot be increased, but 2411it is planned to make both plexes and file systems extensible. 2412.El 2413.Sh STATE MANAGEMENT 2414Vinum objects have the concept of 2415.Em state . 2416See 2417.Xr vinum 4 2418for more details. They are only completely accessible if their state is 2419.Em up . 2420To change an object state to 2421.Em up , 2422use the 2423.Ic start 2424command. To change an object state to 2425.Em down , 2426use the 2427.Ic stop 2428command. Normally other states are created automatically by the relationship 2429between objects. For example, if you add a plex to a volume, the subdisks of 2430the plex will be set in the 2431.Em empty 2432state, indicating that, though the hardware is accessible, the data on the 2433subdisk is invalid. As a result of this state, the plex will be set in the 2434.Em faulty 2435state. 2436.Ss The `reviving' state 2437In many cases, when you start a subdisk the system must copy data to the 2438subdisk. Depending on the size of the subdisk, this can take a long time. 2439During this time, the subdisk is set in the 2440.Em reviving 2441state. On successful completion of the copy operation, it is automatically set 2442to the 2443.Em up 2444state. It is possible for the process performing the revive to be stopped and 2445restarted. The system keeps track of how far the subdisk has been revived, and 2446when the 2447.Ic start 2448command is reissued, the copying continues from this point. 2449.Pp 2450In order to maintain the consistency of a volume while one or more of its plexes 2451is being revived, 2452.Nm 2453writes to subdisks which have been revived up to the point of the write. It may 2454also read from the plex if the area being read has already been revived. 2455.Sh GOTCHAS 2456The following points are not bugs, and they have good reasons for existing, but 2457they have shown to cause confusion. Each is discussed in the appropriate 2458section above. 2459.Bl -enum 2460.It 2461.Nm 2462drives are 2463.Ux 2464disk partitions and must have the partition type 2465.Em vinum . 2466.Pp 2467The 2468.Nm Ic start 2469command will not accept a drive on partition 2470.Dq Li c . 2471Partition 2472.Dq Li c 2473is used by the system to represent the whole disk, and must be of type 2474.Em unused . 2475Clearly there is a conflict here, which 2476.Nm 2477resolves by not using the 2478.Dq Li c 2479partition. 2480.It 2481When you create a volume with multiple plexes, 2482.Nm 2483does not automatically initialize the plexes. This means that the contents are 2484not known, but they are certainly not consistent. As a result, by default 2485.Nm 2486sets the state of all newly-created plexes except the first to 2487.Em faulty . 2488In order to synchronize them with the first plex, you must 2489.Ic start 2490them, which causes 2491.Nm 2492to copy the data from a plex which is in the 2493.Em up 2494state. Depending on the size of the subdisks involved, this can take a long 2495time. 2496.Pp 2497In practice, people aren't too interested in what was in the plex when it was 2498created, and other volume managers cheat by setting them 2499.Em up 2500anyway. 2501.Nm 2502provides two ways to ensure that newly created plexes are 2503.Em up : 2504.Bl -bullet 2505.It 2506Create the plexes and then synchronize them with 2507.Nm Ic start . 2508.It 2509Create the volume (not the plex) with the keyword 2510.Cm setupstate , 2511which tells 2512.Nm 2513to ignore any possible inconsistency and set the plexes to be 2514.Em up . 2515.El 2516.It 2517Some of the commands currently supported by 2518.Nm 2519are not really needed. For reasons which I don't understand, however, I find 2520that users frequently try the 2521.Ic label 2522and 2523.Ic resetconfig 2524commands, though especially 2525.Ic resetconfig 2526outputs all sort of dire warnings. Don't use these commands unless you have a 2527good reason to do so. 2528.It 2529Some state transitions are not very intuitive. In fact, it's not clear whether 2530this is a bug or a feature. If you find that you can't start an object in some 2531strange state, such as a 2532.Em reborn 2533subdisk, try first to get it into 2534.Em stopped 2535state, with the 2536.Ic stop 2537or 2538.Ic stop Fl f 2539commands. If that works, you should then be able to start it. If you find 2540that this is the only way to get out of a position where easier methods fail, 2541please report the situation. 2542.It 2543If you build the kernel module with the 2544.Fl D Ns Dv VINUMDEBUG 2545option, you must also build 2546.Nm 2547with the 2548.Fl D Ns Dv VINUMDEBUG 2549option, since the size of some data objects used by both components depends on 2550this option. If you don't do so, commands will fail with the message 2551.Sy Invalid argument , 2552and a console message will be logged such as 2553.Bl -diag 2554.It "vinumioctl: invalid ioctl from process 247 (vinum): c0e44642" 2555.El 2556.Pp 2557This error may also occur if you use old versions of KLD or userland program. 2558.It 2559The 2560.Nm Ic read 2561command has a particularly emetic syntax. Once it was the only way to start 2562.Nm , 2563but now the preferred method is with 2564.Nm Ic start . 2565.Nm Ic read 2566should be used for maintenance purposes only. Note that its syntax has changed, 2567and the arguments must be disk slices, such as 2568.Pa /dev/da0s0 , 2569not partitions such as 2570.Pa /dev/da0s0e . 2571.El 2572.Sh ENVIRONMENT 2573.Bl -tag -width VINUM_DATEFORMAT 2574.It Ev VINUM_HISTORY 2575The name of the log file, by default 2576.Pa /var/log/vinum_history . 2577.It Ev VINUM_DATEFORMAT 2578The format of dates in the log file, by default 2579.Qq Li %e %b %Y %H:%M:%S . 2580.It Ev EDITOR 2581The name of the editor to use for editing configuration files, by default 2582.Nm vi . 2583.El 2584.Sh FILES 2585.Bl -tag -width /dev/vinum/control -compact 2586.It Pa /dev/vinum 2587directory with device nodes for 2588.Nm 2589objects 2590.It Pa /dev/vinum/control 2591control device for 2592.Nm 2593.It Pa /dev/vinum/plex 2594directory containing device nodes for 2595.Nm 2596plexes 2597.It Pa /dev/vinum/sd 2598directory containing device nodes for 2599.Nm 2600subdisks 2601.El 2602.Sh SEE ALSO 2603.Xr strftime 3 , 2604.Xr vinum 4 , 2605.Xr disklabel 8 , 2606.Xr newfs 8 2607.Pp 2608.Pa http://www.vinumvm.org/vinum/ , 2609.Pa http://www.vinumvm.org/vinum/how-to-debug.html . 2610.Sh HISTORY 2611The 2612.Nm 2613command first appeared in 2614.Fx 3.0 . 2615The RAID-5 component of 2616.Nm 2617was developed for Cybernet Inc.\& 2618.Pq Pa www.cybernet.com 2619for its NetMAX product. 2620.Sh AUTHORS 2621.An Greg Lehey Aq Mt grog@lemis.com 2622