1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright 2013 Darik Horn <dajhorn@vanadac.com>. All rights reserved. 22.\" 23.\" lint-ok: WARNING: sections out of conventional order: Sh SYNOPSIS 24.\" 25.Dd May 26, 2021 26.Dt ZINJECT 8 27.Os 28. 29.Sh NAME 30.Nm zinject 31.Nd ZFS Fault Injector 32.Sh DESCRIPTION 33.Nm 34creates artificial problems in a ZFS pool by simulating data corruption 35or device failures. 36This program is dangerous. 37. 38.Sh SYNOPSIS 39.Bl -tag -width Ds 40.It Xo 41.Nm zinject 42.Xc 43List injection records. 44. 45.It Xo 46.Nm zinject 47.Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 48.Op Fl f Ar frequency 49.Fl amu 50.Op pool 51.Xc 52Force an error into the pool at a bookmark. 53. 54.It Xo 55.Nm zinject 56.Fl c Ar id Ns | Ns Sy all 57.Xc 58Cancel injection records. 59. 60.It Xo 61.Nm zinject 62.Fl d Ar vdev 63.Fl A Sy degrade Ns | Ns Sy fault 64.Ar pool 65.Xc 66Force a vdev into the DEGRADED or FAULTED state. 67. 68.It Xo 69.Nm zinject 70.Fl d Ar vdev 71.Fl D Ar latency : Ns Ar lanes 72.Op Fl T Ar read|write 73.Ar pool 74.Xc 75Add an artificial delay to I/O requests on a particular 76device, such that the requests take a minimum of 77.Ar latency 78milliseconds to complete. 79Each delay has an associated number of 80.Ar lanes 81which defines the number of concurrent 82I/O requests that can be processed. 83.Pp 84For example, with a single lane delay of 10 ms 85.No (\& Ns Fl D Ar 10 : Ns Ar 1 ) , 86the device will only be able to service a single I/O request 87at a time with each request taking 10 ms to complete. 88So, if only a single request is submitted every 10 ms, the 89average latency will be 10 ms; but if more than one request 90is submitted every 10 ms, the average latency will be more 91than 10 ms. 92.Pp 93Similarly, if a delay of 10 ms is specified to have two 94lanes 95.No (\& Ns Fl D Ar 10 : Ns Ar 2 ) , 96then the device will be able to service 97two requests at a time, each with a minimum latency of 10 ms. 98So, if two requests are submitted every 10 ms, then 99the average latency will be 10 ms; but if more than two 100requests are submitted every 10 ms, the average latency 101will be more than 10 ms. 102.Pp 103Also note, these delays are additive. 104So two invocations of 105.Fl D Ar 10 : Ns Ar 1 106are roughly equivalent to a single invocation of 107.Fl D Ar 10 : Ns Ar 2 . 108This also means, that one can specify multiple 109lanes with differing target latencies. 110For example, an invocation of 111.Fl D Ar 10 : Ns Ar 1 112followed by 113.Fl D Ar 25 : Ns Ar 2 114will create 3 lanes on the device: one lane with a latency 115of 10 ms and two lanes with a 25 ms latency. 116. 117.It Xo 118.Nm zinject 119.Fl d Ar vdev 120.Op Fl e Ar device_error 121.Op Fl L Ar label_error 122.Op Fl T Ar failure 123.Op Fl f Ar frequency 124.Op Fl F 125.Ar pool 126.Xc 127Force a vdev error. 128. 129.It Xo 130.Nm zinject 131.Fl I 132.Op Fl s Ar seconds Ns | Ns Fl g Ar txgs 133.Ar pool 134.Xc 135Simulate a hardware failure that fails to honor a cache flush. 136. 137.It Xo 138.Nm zinject 139.Fl p Ar function 140.Ar pool 141.Xc 142Panic inside the specified function. 143. 144.It Xo 145.Nm zinject 146.Fl t Sy data 147.Fl C Ar dvas 148.Op Fl e Ar device_error 149.Op Fl f Ar frequency 150.Op Fl l Ar level 151.Op Fl r Ar range 152.Op Fl amq 153.Ar path 154.Xc 155Force an error into the contents of a file. 156. 157.It Xo 158.Nm zinject 159.Fl t Sy dnode 160.Fl C Ar dvas 161.Op Fl e Ar device_error 162.Op Fl f Ar frequency 163.Op Fl l Ar level 164.Op Fl amq 165.Ar path 166.Xc 167Force an error into the metadnode for a file or directory. 168. 169.It Xo 170.Nm zinject 171.Fl t Ar mos_type 172.Fl C Ar dvas 173.Op Fl e Ar device_error 174.Op Fl f Ar frequency 175.Op Fl l Ar level 176.Op Fl r Ar range 177.Op Fl amqu 178.Ar pool 179.Xc 180Force an error into the MOS of a pool. 181.El 182.Sh OPTIONS 183.Bl -tag -width "-C dvas" 184.It Fl a 185Flush the ARC before injection. 186.It Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 187Force an error into the pool at this bookmark tuple. 188Each number is in hexadecimal, and only one block can be specified. 189.It Fl C Ar dvas 190Inject the given error only into specific DVAs. 191The mask should be specified as a list of 0-indexed DVAs separated by commas 192.No (e.g . Ar 0,2 Ns No ). 193This option is not applicable to logical data errors such as 194.Sy decompress 195and 196.Sy decrypt . 197.It Fl d Ar vdev 198A vdev specified by path or GUID. 199.It Fl e Ar device_error 200Specify 201.Bl -tag -compact -width "decompress" 202.It Sy checksum 203for an ECKSUM error, 204.It Sy decompress 205for a data decompression error, 206.It Sy decrypt 207for a data decryption error, 208.It Sy corrupt 209to flip a bit in the data after a read, 210.It Sy dtl 211for an ECHILD error, 212.It Sy io 213for an EIO error where reopening the device will succeed, or 214.It Sy nxio 215for an ENXIO error where reopening the device will fail. 216.El 217.Pp 218For EIO and ENXIO, the "failed" reads or writes still occur. 219The probe simply sets the error value reported by the I/O pipeline 220so it appears the read or write failed. 221Decryption errors only currently work with file data. 222.It Fl f Ar frequency 223Only inject errors a fraction of the time. 224Expressed as a real number percentage between 225.Sy 0.0001 226and 227.Sy 100 . 228.It Fl F 229Fail faster. 230Do fewer checks. 231.It Fl f Ar txgs 232Run for this many transaction groups before reporting failure. 233.It Fl h 234Print the usage message. 235.It Fl l Ar level 236Inject an error at a particular block level. 237The default is 238.Sy 0 . 239.It Fl L Ar label_error 240Set the label error region to one of 241.Sy nvlist , 242.Sy pad1 , 243.Sy pad2 , 244or 245.Sy uber . 246.It Fl m 247Automatically remount the underlying filesystem. 248.It Fl q 249Quiet mode. 250Only print the handler number added. 251.It Fl r Ar range 252Inject an error over a particular logical range of an object, which 253will be translated to the appropriate blkid range according to the 254object's properties. 255.It Fl s Ar seconds 256Run for this many seconds before reporting failure. 257.It Fl T Ar failure 258Set the failure type to one of 259.Sy all , 260.Sy claim , 261.Sy free , 262.Sy read , 263or 264.Sy write . 265.It Fl t Ar mos_type 266Set this to 267.Bl -tag -compact -width "spacemap" 268.It Sy mos 269for any data in the MOS, 270.It Sy mosdir 271for an object directory, 272.It Sy config 273for the pool configuration, 274.It Sy bpobj 275for the block pointer list, 276.It Sy spacemap 277for the space map, 278.It Sy metaslab 279for the metaslab, or 280.It Sy errlog 281for the persistent error log. 282.El 283.It Fl u 284Unload the pool after injection. 285.El 286. 287.Sh ENVIRONMENT VARIABLES 288.Bl -tag -width "ZF" 289.It Ev ZFS_HOSTID 290Run 291.Nm 292in debug mode. 293.El 294. 295.Sh SEE ALSO 296.Xr zfs 8 , 297.Xr zpool 8 298