1 2CTL - CAM Target Layer Description 3 4Revision 1.4 (December 29th, 2011) 5Ken Merry <ken@FreeBSD.org> 6 7Table of Contents: 8================= 9 10Introduction 11Features 12Configuring and Running CTL 13Revision 1.N Changes 14To Do List 15Code Roadmap 16Userland Commands 17 18Introduction: 19============ 20 21CTL is a disk, processor and cdrom device emulation subsystem originally 22written for Copan Systems under Linux starting in 2003. It has been 23shipping in Copan (now SGI) products since 2005. 24 25It was ported to FreeBSD in 2008, and thanks to an agreement between SGI 26(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is 27available under a BSD-style license. The intent behind the agreement was 28that Spectra would work to get CTL into the FreeBSD tree. 29 30Features: 31======== 32 33 - Disk, processor and cdrom device emulation. 34 - Tagged queueing 35 - SCSI task attribute support (ordered, head of queue, simple tags) 36 - SCSI implicit command ordering support. (e.g. if a read follows a mode 37 select, the read will be blocked until the mode select completes.) 38 - Full task management support (abort, LUN reset, target reset, etc.) 39 - Support for multiple ports 40 - Support for multiple simultaneous initiators 41 - Support for multiple simultaneous backing stores 42 - Support for VMWare VAAI: COMPARE AND WRITE, XCOPY, WRITE SAME and 43 UNMAP commands 44 - Support for Microsoft ODX: POPULATE TOKEN/WRITE USING TOKEN, WRITE SAME 45 and UNMAP commands 46 - Persistent reservation support 47 - Mode sense/select support 48 - Error injection support 49 - High Availability clustering support with ALUA 50 - All I/O handled in-kernel, no userland context switch overhead. 51 52Configuring and Running CTL: 53=========================== 54 55 - Add 'device ctl' to your kernel configuration file or load the module. 56 57 - If you're running with a 8Gb or 4Gb Qlogic FC board, add 58 'options ISP_TARGET_MODE' to your kernel config file. 'device ispfw' or 59 loading the ispfw module is also recommended. 60 61 - Rebuild and install a new kernel. 62 63 - Reboot with the new kernel. 64 65 - To add a LUN with the RAM disk backend: 66 67 ctladm create -b ramdisk -s 10485760000000000000 68 ctladm port -o on 69 70 - You should now see the CTL disk LUN through camcontrol devlist: 71 72scbus6 on ctl2cam0 bus 0: 73<FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32) 74<> at scbus6 target -1 lun -1 () 75 76 This is visible through the CTL CAM SIM. This allows using CTL without 77 any physical hardware. You should be able to issue any normal SCSI 78 commands to the device via the pass(4)/da(4) devices. 79 80 If any target-capable HBAs are in the system (e.g. isp(4)), and have 81 target mode enabled, you should now also be able to see the CTL LUNs via 82 that target interface. 83 84 Note that all CTL LUNs are presented to all frontends. There is no 85 LUN masking, or separate, per-port configuration. 86 87 - Note that the ramdisk backend is a "fake" ramdisk. That is, it is 88 backed by a small amount of RAM that is used for all I/O requests. This 89 is useful for performance testing, but not for any data integrity tests. 90 91 - To add a LUN with the block/file backend: 92 93 truncate -s +1T myfile 94 ctladm create -b block -o file=myfile 95 ctladm port -o on 96 97 - You can also see a list of LUNs and their backends like this: 98 99# ctladm devlist 100LUN Backend Size (Blocks) BS Serial Number Device ID 101 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 102 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 103 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 104 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 105 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 106 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 107 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 108 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 109 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 110 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 111 10 block 2147483648 512 MYSERIAL 10 MYDEVID 10 112 11 block 2147483648 512 MYSERIAL 11 MYDEVID 11 113 114 - You can see the LUN type and backing store for block/file backend LUNs 115 like this: 116 117# ctladm devlist -v 118LUN Backend Size (Blocks) BS Serial Number Device ID 119 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 120 lun_type=0 121 num_threads=14 122 file=testdisk0 123 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 124 lun_type=0 125 num_threads=14 126 file=testdisk1 127 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 128 lun_type=0 129 num_threads=14 130 file=testdisk2 131 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 132 lun_type=0 133 num_threads=14 134 file=testdisk3 135 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 136 lun_type=0 137 num_threads=14 138 file=testdisk4 139 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 140 lun_type=0 141 num_threads=14 142 file=testdisk5 143 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 144 lun_type=0 145 num_threads=14 146 file=testdisk6 147 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 148 lun_type=0 149 num_threads=14 150 file=testdisk7 151 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 152 lun_type=0 153 num_threads=14 154 file=testdisk8 155 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 156 lun_type=0 157 num_threads=14 158 file=testdisk9 159 10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0 160 lun_type=3 161 11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1 162 lun_type=0 163 164Revision 1.4 Changes 165==================== 166 - Added in the second HA mode (where CTL does the data transfers instead 167 of having data transfers done below CTL), and abstracted out the Copan 168 HA API. 169 170 - Fixed the phantom device problem in the CTL CAM SIM and improved the 171 CAM SIM to automatically trigger a rescan when the port is enabled and 172 disabled. 173 174 - Made the number of threads in the block backend configurable via sysctl, 175 loader tunable and the ctladm command line. (You can now specify 176 -o num_threads=4 when creating a LUN with ctladm create.) 177 178 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection 179 of LUN numbers up to 1023. 180 181 - General cleanup. 182 183 - This version intended for public release. 184 185Revision 1.3 Changes 186==================== 187 - Added descriptor sense support to CTL. It can be enabled through the 188 control mode page (10), but is disabled by default. 189 190 - Improved error injection support. The number of errors that can be 191 injected with 'ctladm inject' has been increased, and any arbitrary 192 sense data may now be injected as well. 193 194 - The port infrastructure has been revamped. Individual ports and types 195 of ports may now be enabled and disabled from the command line. ctladm 196 now has the ability to set the WWNN and WWPN for each port. 197 198 - The block backend can now send multiple I/Os to backing files. Multiple 199 writes are only allowed for ZFS, but multiple readers are allowed for 200 any filesystem. 201 202 - The block and ramdisk backends now support setting the LUN blocksize. 203 There are some restrictions when the backing device is a block device, 204 but otherwise the blocksize may be set to anything. 205 206Revision 1.2 Changes 207==================== 208 209 - CTL initialization process has been revamped. Instead of using an 210 ad-hoc method, it is now sequenced through SYSINIT() calls. 211 212 - A block/file backend has been added. This allows using arbitrary files 213 or block devices as a backing store. 214 215 - The userland LUN configuration interface has been completely rewritten. 216 Configuration is now done out of band. 217 218 - The ctladm(8) command line interface has been revamped, and is now 219 similar to camcontrol(8). 220 221To Do List: 222========== 223 224 - Use devstat(9) for CTL's statistics collection. CTL uses a home-grown 225 statistics collection system that is similar to devstat(9). ctlstat 226 should be retired in favor of iostat, etc., once aggregation modes are 227 available in iostat to match the behavior of ctlstat -t and dump modes 228 are available to match the behavior of ctlstat -d/ctlstat -J. 229 230 - ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC 231 (Adaptive Replacement Cache), running the block/file backend on top of a 232 ZFS-backed zdev or file will involve an extra set of copies. The 233 optimal solution for backing targets served by CTL with ZFS would be to 234 allocate buffers out of the ARC directly, and DMA to/from them directly. 235 That would eliminate an extra data buffer allocation and copy. 236 237 - Switch CTL over to using CAM CCBs instead of its own union ctl_io. This 238 will likely require a significant amount of work, but will eliminate 239 another data structure in the stack, more memory allocations, etc. This 240 will also require changes to the CAM CCB structure to support CTL. 241 242Code Roadmap: 243============ 244 245CTL has the concept of pluggable frontend ports and backends. All 246frontends and backends can be active at the same time. You can have a 247ramdisk-backed LUN present along side a file backed LUN. 248 249ctl.c: 250----- 251 252This is the core of CTL, where all of the command handlers and a lot of 253other things live. Yes, it is large. It started off small and grew to its 254current size over time. Perhaps it can be split into more files at some 255point. 256 257Here is a roadmap of some of the primary functions in ctl.c. Starting here 258and following the various leaf functions will show the command flow. 259 260ctl_queue() / ctl_run() This is where commands from the frontend ports come 261 in. 262 263ctl_queue_sense() This is only used for non-packetized SCSI. i.e. 264 parallel SCSI prior to U320 and perhaps U160. 265 266ctl_work_thread() This is the primary work thread, and everything gets 267 executed from there. 268 269ctl_scsiio_precheck() This where all of the initial checks are done, and I/O 270 is either queued for execution or blocked. 271 272ctl_scsiio() This is where the command handler is actually 273 executed. (See ctl_cmd_table.c for the mapping of 274 SCSI opcode to command handler function.) 275 276ctl_done() This is the routine called (or ctl_done_lock()) to 277 initiate the command completion process. 278 279ctl_process_done() This is where command completion actually happens. 280 281ctl.h: 282----- 283 284Basic function declarations and data structures. 285 286ctl_backend.c, 287ctl_backend.h: 288------------- 289 290These files define the basic CTL backend API. The comments in the header 291explain the API. 292 293ctl_backend_block.c 294------------------- 295 296The block and file backend. This allows for using a disk or a file as the 297backing store for a LUN. Multiple threads are started to do I/O to the 298backing device, primarily because the VFS API requires that to get any 299concurrency. 300 301ctl_backend_ramdisk.c: 302--------------------- 303 304A "fake" ramdisk backend. It only allocates a small amount of memory to 305act as a source and sink for reads and writes from an initiator. Therefore 306it cannot be used for any real data, but it can be used to test for 307throughput. It can also be used to test initiators' support for extremely 308large LUNs. 309 310ctl_cmd_table.c: 311--------------- 312 313This is a table with all 256 possible SCSI opcodes, and command handler 314functions defined for supported opcodes. It is included in ctl.c. 315 316ctl_debug.h: 317----------- 318 319Simplistic debugging support. 320 321ctl_error.c, 322ctl_error.h: 323----------- 324 325CTL-specific wrappers around the CAM sense building functions. 326 327ctl_frontend.c, 328ctl_frontend.h: 329-------------- 330 331These files define the basic CTL frontend port API. The comments in the 332header explain the API. 333 334ctl_frontend_cam_sim.c: 335---------------------- 336 337This is a CTL frontend port that is also a CAM SIM. The idea is that this 338frontend allows for using CTL without any target-capable hardware. So any 339LUNs you create in CTL are visible via this port. 340 341ctl_ha.c: 342ctl_ha.h: 343-------- 344 345This is a High Availability API and TCP-based interlink implementation. 346 347ctl_io.h: 348-------- 349 350This defines most of the core CTL I/O structures. union ctl_io is 351conceptually very similar to CAM's union ccb. 352 353ctl_ioctl.h: 354----------- 355 356This defines all ioctls available through the CTL character device, and 357the data structures needed for those ioctls. 358 359ctl_private.h: 360------------- 361 362Private data structres (e.g. CTL softc) and function prototypes. This also 363includes the SCSI vendor and product names used by CTL. 364 365ctl_scsi_all.c 366ctl_scsi_all.h: 367-------------- 368 369CTL wrappers around CAM sense printing functions. 370 371ctl_ser_table.c: 372--------------- 373 374Command serialization table. This defines what happens when one type of 375command is followed by another type of command. e.g., what do you do when 376you have a mode select followed by a write? You block the write until the 377mode select is complete. That is defined in this table. 378 379ctl_util.c 380ctl_util.h: 381---------- 382 383CTL utility functions, primarily designed to be used from userland. See 384ctladm for the primary consumer of these functions. These include CDB 385building functions. 386 387scsi_ctl.c: 388---------- 389 390CAM target peripheral driver and CTL frontend port. This is the path into 391CTL for commands from target-capable hardware/SIMs. 392 393Userland Commands: 394================= 395 396ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs, 397issuing commands, injecting errors and various other control functions. 398 399ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics 400for CTL. 401