1.\" 2.\" Copyright (c) 2006, 2007 3.\" The DragonFly Project. All rights reserved. 4.\" 5.\" Redistribution and use in source and binary forms, with or without 6.\" modification, are permitted provided that the following conditions 7.\" are met: 8.\" 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in 13.\" the documentation and/or other materials provided with the 14.\" distribution. 15.\" 3. Neither the name of The DragonFly Project nor the names of its 16.\" contributors may be used to endorse or promote products derived 17.\" from this software without specific, prior written permission. 18.\" 19.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 22.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 23.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 25.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 26.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 27.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 29.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 30.\" SUCH DAMAGE. 31.\" 32.Dd June 20, 2015 33.Dt VKERNEL 7 34.Os 35.Sh NAME 36.Nm vkernel , 37.Nm vcd , 38.Nm vkd , 39.Nm vke 40.Nd virtual kernel architecture 41.Sh SYNOPSIS 42.Cd "platform vkernel64 # for 64 bit vkernels" 43.Cd "device vcd" 44.Cd "device vkd" 45.Cd "device vke" 46.Pp 47.Pa /var/vkernel/boot/kernel/kernel 48.Op Fl hdstUvz 49.Op Fl c Ar file 50.Op Fl e Ar name Ns = Ns Li value : Ns Ar name Ns = Ns Li value : Ns ... 51.Op Fl i Ar file 52.Op Fl I Ar interface Ns Op Ar :address1 Ns Oo Ar :address2 Oc Ns Oo Ar /netmask Oc Ns Oo Ar =mac Oc 53.Op Fl l Ar cpulock 54.Op Fl m Ar size 55.Op Fl n Ar numcpus Ns Op Ar :lbits Ns Oo Ar :cbits Oc 56.Op Fl p Ar pidfile 57.Op Fl r Ar file Ns Op Ar :serno 58.Op Fl R Ar file Ns Op Ar :serno 59.Sh DESCRIPTION 60The 61.Nm 62architecture allows for running 63.Dx 64kernels in userland. 65.Pp 66The following options are available: 67.Bl -tag -width ".Fl m Ar size" 68.It Fl c Ar file 69Specify a readonly CD-ROM image 70.Ar file 71to be used by the kernel, with the first 72.Fl c 73option defining 74.Li vcd0 , 75the second one 76.Li vcd1 , 77and so on. 78The first 79.Fl r , 80.Fl R , 81or 82.Fl c 83option specified on the command line will be the boot disk. 84The CD9660 filesystem is assumed when booting from this media. 85.It Fl d 86Disables hardware pagetable for 87.Nm . 88.It Fl e Ar name Ns = Ns Li value : Ns Ar name Ns = Ns Li value : Ns ... 89Specify an environment to be used by the kernel. 90This option can be specified more than once. 91.It Fl h 92Shows a list of available options, each with a short description. 93.It Fl i Ar file 94Specify a memory image 95.Ar file 96to be used by the virtual kernel. 97If no 98.Fl i 99option is given, the kernel will generate a name of the form 100.Pa /var/vkernel/memimg.XXXXXX , 101with the trailing 102.Ql X Ns s 103being replaced by a sequential number, e.g.\& 104.Pa memimg.000001 . 105.It Fl I Ar interface Ns Op Ar :address1 Ns Oo Ar :address2 Oc Ns Oo Ar /netmask Oc Ns Oo Ar =MAC Oc 106Create a virtual network device, with the first 107.Fl I 108option defining 109.Li vke0 , 110the second one 111.Li vke1 , 112and so on. 113.Pp 114The 115.Ar interface 116argument is the name of a 117.Xr tap 4 118device node or the path to a 119.Xr vknetd 8 120socket. 121The 122.Pa /dev/ 123path prefix does not have to be specified and will be automatically prepended 124for a device node. 125Specifying 126.Cm auto 127will pick the first unused 128.Xr tap 4 129device. 130.Pp 131The 132.Ar address1 133and 134.Ar address2 135arguments are the IP addresses of the 136.Xr tap 4 137and 138.Nm vke 139interfaces. 140Optionally, 141.Ar address1 142may be of the form 143.Li bridge Ns Em X 144in which case the 145.Xr tap 4 146interface is added to the specified 147.Xr bridge 4 148interface. 149The 150.Nm vke 151address is not assigned until the interface is brought up in the guest. 152.Pp 153The 154.Ar netmask 155argument applies to all interfaces for which an address is specified. 156.Pp 157The 158.Ar MAC 159argument is the MAC address of the 160.Xr vke 4 161interface. 162If not specified, a pseudo-random one will be generated. 163.Pp 164When running multiple vkernels it is often more convenient to simply 165connect to a 166.Xr vknetd 8 167socket and let vknetd deal with the tap and/or bridge. 168An example of this would be 169.Pa /var/run/vknet:0.0.0.0:10.2.0.2/16 . 170.It Fl l Ar cpulock 171Specify which, if any, real CPUs to lock virtual CPUs to. 172.Ar cpulock 173is one of 174.Cm any , 175.Cm map Ns Op Ns , Ns Ar startCPU , 176or 177.Ar CPU . 178.Pp 179.Cm any 180does not map virtual CPUs to real CPUs. 181This is the default. 182.Pp 183.Cm map Ns Op Ns , Ns Ar startCPU 184maps each virtual CPU to a real CPU starting with real CPU 0 or 185.Ar startCPU 186if specified. 187.Pp 188.Ar CPU 189locks all virtual CPUs to the real CPU specified by 190.Ar CPU . 191.Pp 192Locking the vkernel to a set of cpus is recommended on multi-socket systems 193to improve NUMA locality of reference. 194.It Fl m Ar size 195Specify the amount of memory to be used by the kernel in bytes, 196.Cm K 197.Pq kilobytes , 198.Cm M 199.Pq megabytes 200or 201.Cm G 202.Pq gigabytes . 203Lowercase versions of 204.Cm K , M , 205and 206.Cm G 207are allowed. 208.It Fl n Ar numcpus Ns Op Ar :lbits Ns Oo Ar :cbits Oc 209.Ar numcpus 210specifies the number of CPUs you wish to emulate. 211Up to 16 CPUs are supported with 2 being the default unless otherwise 212specified. 213.Pp 214.Ar lbits 215specifies the number of bits within APICID(=CPUID) needed for representing 216the logical ID. 217Controls the number of threads/core (0 bits - 1 thread, 1 bit - 2 threads). 218This parameter is optional (mandatory only if 219.Ar cbits 220is specified). 221.Pp 222.Ar cbits 223specifies the number of bits within APICID(=CPUID) needed for representing 224the core ID. 225Controls the number of core/package (0 bits - 1 core, 1 bit - 2 cores). 226This parameter is optional. 227.It Fl p Ar pidfile 228Specify a pidfile in which to store the process ID. 229Scripts can use this file to locate the vkernel pid for the purpose of 230shutting down or killing it. 231.Pp 232The vkernel will hold a lock on the pidfile while running. 233Scripts may test for the lock to determine if the pidfile is valid or 234stale so as to avoid accidentally killing a random process. 235Something like '/usr/bin/lockf -ks -t 0 pidfile echo -n' may be used 236to test the lock. 237A non-zero exit code indicates that the pidfile represents a running 238vkernel. 239.Pp 240An error is issued and the vkernel exits if this file cannot be opened for 241writing or if it is already locked by an active vkernel process. 242.It Fl r Ar file Ns Op Ar :serno 243Specify a R/W disk image 244.Ar file 245to be used by the kernel, with the first 246.Fl r 247option defining 248.Li vkd0 , 249the second one 250.Li vkd1 , 251and so on. 252A serial number for the virtual disk can be specified in 253.Ar serno . 254.Pp 255The first 256.Fl r 257or 258.Fl c 259option specified on the command line will be the boot disk. 260.It Fl R Ar file Ns Op Ar :serno 261Works like 262.Fl r 263but treats the disk image as copy-on-write. This allows 264a private copy of the image to be modified but does not 265modify the image file. The image file will not be locked 266in this situation and multiple vkernels can run off the 267same image file if desired. 268.Pp 269Since modifications are thrown away, any data you wish 270to retain across invocations needs to be exported over 271the network prior to shutdown. 272This gives you the flexibility to mount the disk image 273either read-only or read-write depending on what is 274convenient. 275However, keep in mind that when mounting a COW image 276read-write, modifications will eat system memory and 277swap space until the vkernel is shut down. 278.It Fl s 279Boot into single-user mode. 280.It Fl t 281Tell the vkernel to use a precise host timer when calculating clock values. 282This will impose higher overhead on the vkernel as it will have to make 283a system call to the real host every time it wants to get the time. 284However, the more precise timer might be necessary for your application. 285.Pp 286By default, the vkernel uses an imprecise (host-tick-resolution) timer 287which uses a user-mapped kernel page and does not have any syscall overhead. 288.It Fl U 289Enable writing to kernel memory and module loading. 290By default, those are disabled for security reasons. 291.It Fl v 292Turn on verbose booting. 293.It Fl z 294Force the vkernel's ram to be pre-zerod. Useful for benchmarking on 295single-socket systems where the memory allocation does not have to be 296NUMA-friendly. 297This options is not recommended on multi-socket systems or when the 298.Fl l 299option is used. 300.El 301.Sh DEVICES 302A number of virtual device drivers exist to supplement the virtual kernel. 303.Ss Disk device 304The 305.Nm vkd 306driver allows for up to 16 307.Xr vn 4 308based disk devices. 309The root device will be 310.Li vkd0 311(see 312.Sx EXAMPLES 313for further information on how to prepare a root image). 314.Ss CD-ROM device 315The 316.Nm vcd 317driver allows for up to 16 virtual CD-ROM devices. 318Basically this is a read only 319.Nm vkd 320device with a block size of 2048. 321.Ss Network interface 322The 323.Nm vke 324driver supports up to 16 virtual network interfaces which are associated with 325.Xr tap 4 326devices on the host. 327For each 328.Nm vke 329device, the per-interface read only 330.Xr sysctl 3 331variable 332.Va hw.vke Ns Em X Ns Va .tap_unit 333holds the unit number of the associated 334.Xr tap 4 335device. 336.Pp 337By default, half of the total mbuf clusters available is distributed equally 338among all the vke devices up to 256. 339This can be overridden with the tunable 340.Va hw.vke.max_ringsize . 341Take into account the number passed will be aligned to the lower power of two. 342.Sh SIGNALS 343The virtual kernel only enables 344.Dv SIGQUIT 345and 346.Dv SIGTERM 347while operating in regular console mode. 348Sending 349.Ql \&^\e 350.Pq Dv SIGQUIT 351to the virtual kernel causes the virtual kernel to enter its internal 352.Xr ddb 4 353debugger and re-enable all other terminal signals. 354Sending 355.Dv SIGTERM 356to the virtual kernel triggers a clean shutdown by passing a 357.Dv SIGUSR2 358to the virtual kernel's 359.Xr init 8 360process. 361.Sh DEBUGGING 362It is possible to directly gdb the virtual kernel's process. 363It is recommended that you do a 364.Ql handle SIGSEGV noprint 365to ignore page faults processed by the virtual kernel itself and 366.Ql handle SIGUSR1 noprint 367to ignore signals used for simulating inter-processor interrupts. 368.Sh PROFILING 369To compile a vkernel with profiling support, the 370.Va CONFIGARGS 371variable needs to be used to pass 372.Fl p 373to 374.Xr config 8 . 375.Bd -literal 376cd /usr/src 377make -DNO_MODULES CONFIGARGS=-p buildkernel KERNCONF=VKERNEL64 378.Ed 379.Sh FILES 380.Bl -tag -width ".It Pa /sys/config/VKERNEL64" -compact 381.It Pa /dev/vcdX 382.Nm vcd 383device nodes 384.It Pa /dev/vkdX 385.Nm vkd 386device nodes 387.It Pa /sys/config/VKERNEL64 388.El 389.Pp 390.Nm 391configuration file, for 392.Xr config 8 . 393.Sh CONFIGURATION FILES 394Your virtual kernel is a complete 395.Dx 396system, but you might not want to run all the services a normal kernel runs. 397Here is what a typical virtual kernel's 398.Pa /etc/rc.conf 399file looks like, with some additional possibilities commented out. 400.Bd -literal 401hostname="vkernel" 402network_interfaces="lo0 vke0" 403ifconfig_vke0="DHCP" 404sendmail_enable="NO" 405#syslog_enable="NO" 406blanktime="NO" 407.Ed 408.Sh BOOT DRIVE SELECTION 409You can override the default boot drive selection and filesystem 410using a kernel environment variable. Note that the filesystem 411selected must be compiled into the vkernel and not loaded as 412a module. You need to escape some quotes around the variable data 413to avoid mis-interpretation of the colon in the 414.Fl e 415option. For example: 416.Pp 417.Fl e 418vfs.root.mountfrom=\\"hammer:vkd0s1d\\" 419.Sh DISKLESS OPERATION 420To boot a 421.Nm 422from a NFS root, a number of tunables need to be set: 423.Bl -tag -width indent 424.It Va boot.netif.ip 425IP address to be set in the vkernel interface. 426.It Va boot.netif.netmask 427Netmask for the IP to be set. 428.It Va boot.netif.name 429Network interface name inside the vkernel. 430.It Va boot.nfsroot.server 431Host running 432.Xr nfsd 8 . 433.It Va boot.nfsroot.path 434Host path where a world and distribution 435targets are properly installed. 436.El 437.Pp 438See an example on how to boot a diskless 439.Nm 440in the 441.Sx EXAMPLES 442section. 443.Sh EXAMPLES 444A couple of steps are necessary in order to prepare the system to build and 445run a virtual kernel. 446.Ss Setting up the filesystem 447The 448.Nm 449architecture needs a number of files which reside in 450.Pa /var/vkernel . 451Since these files tend to get rather big and the 452.Pa /var 453partition is usually of limited size, we recommend the directory to be 454created in the 455.Pa /home 456partition with a link to it in 457.Pa /var : 458.Bd -literal 459mkdir -p /home/var.vkernel/boot 460ln -s /home/var.vkernel /var/vkernel 461.Ed 462.Pp 463Next, a filesystem image to be used by the virtual kernel has to be 464created and populated (assuming world has been built previously). 465If the image is created on a UFS filesystem you might want to pre-zero it. 466On a HAMMER filesystem you should just truncate-extend to the image size 467as HAMMER does not re-use data blocks already present in the file. 468.Bd -literal 469vnconfig -c -S 2g -T vn0 /var/vkernel/rootimg.01 470disklabel -r -w vn0s0 auto 471disklabel -e vn0s0 # add `a' partition with fstype `4.2BSD' 472newfs /dev/vn0s0a 473mount /dev/vn0s0a /mnt 474cd /usr/src 475make installworld DESTDIR=/mnt 476cd etc 477make distribution DESTDIR=/mnt 478echo '/dev/vkd0s0a / ufs rw 1 1' >/mnt/etc/fstab 479echo 'proc /proc procfs rw 0 0' >>/mnt/etc/fstab 480.Ed 481.Pp 482Edit 483.Pa /mnt/etc/ttys 484and replace the 485.Li console 486entry with the following line and turn off all other gettys. 487.Bd -literal 488console "/usr/libexec/getty Pc" cons25 on secure 489.Ed 490.Pp 491Replace 492.Li \&Pc 493with 494.Li al.Pc 495if you would like to automatically log in as root. 496.Pp 497Then, unmount the disk. 498.Bd -literal 499umount /mnt 500vnconfig -u vn0 501.Ed 502.Ss Compiling the virtual kernel 503In order to compile a virtual kernel use the 504.Li VKERNEL64 505kernel configuration file residing in 506.Pa /sys/config 507(or a configuration file derived thereof): 508.Bd -literal 509cd /usr/src 510make -DNO_MODULES buildkernel KERNCONF=VKERNEL64 511make -DNO_MODULES installkernel KERNCONF=VKERNEL64 DESTDIR=/var/vkernel 512.Ed 513.Ss Enabling virtual kernel operation 514A special 515.Xr sysctl 8 , 516.Va vm.vkernel_enable , 517must be set to enable 518.Nm 519operation: 520.Bd -literal 521sysctl vm.vkernel_enable=1 522.Ed 523.Ss Configuring the network on the host system 524In order to access a network interface of the host system from the 525.Nm , 526you must add the interface to a 527.Xr bridge 4 528device which will then be passed to the 529.Fl I 530option: 531.Bd -literal 532kldload if_bridge.ko 533kldload if_tap.ko 534ifconfig bridge0 create 535ifconfig bridge0 addm re0 # assuming re0 is the host's interface 536ifconfig bridge0 up 537.Ed 538.Ss Running the kernel 539Finally, the virtual kernel can be run: 540.Bd -literal 541cd /var/vkernel 542\&./boot/kernel/kernel -m 1g -r rootimg.01 -I auto:bridge0 543.Ed 544.Pp 545You can issue the 546.Xr reboot 8 , 547.Xr halt 8 , 548or 549.Xr shutdown 8 550commands from inside a virtual kernel. 551After doing a clean shutdown the 552.Xr reboot 8 553command will re-exec the virtual kernel binary while the other two will 554cause the virtual kernel to exit. 555.Ss Diskless operation (vkernel as a NFS client) 556Booting a 557.Nm 558with a 559.Xr vknetd 8 560network configuration. The line continuation backslashes have been 561omitted. For convenience and to reduce confusion I recommend mounting 562the server's remote vkernel root onto the host running the vkernel binary 563using the same path as the NFS mount. It is assumed that a full system 564install has been made to /var/vkernel/root using a kernel KERNCONF=VKERNEL64 565for the kernel build. 566.Bd -literal 567\&/var/vkernel/root/boot/kernel/kernel 568 -m 1g -n 4 -I /var/run/vknet 569 -e boot.netif.ip=10.100.0.2 570 -e boot.netif.netmask=255.255.0.0 571 -e boot.netif.gateway=10.100.0.1 572 -e boot.netif.name=vke0 573 -e boot.nfsroot.server=10.0.0.55 574 -e boot.nfsroot.path=/var/vkernel/root 575.Ed 576.Pp 577In this example vknetd is assumed to have been started as shown below, before 578running the vkernel, using an unbridged TAP configuration routed through 579the host. 580IP forwarding must be turned on, and in this example the server resides 581on a different network accessible to the host executing the vkernel but not 582directly on the vkernel's subnet. 583.Bd -literal 584kldload if_tap 585sysctl net.inet.ip.forwarding=1 586vknetd -t tap0 10.100.0.1/16 587.Ed 588.Pp 589You can run multiple vkernels trivially with the same NFS root as long as 590you assign each one a different IP on the subnet (2, 3, 4, etc). You 591should also be careful with certain directories, particularly /var/run 592and possibly also /var/db depending on what your vkernels are going to be 593doing. 594This can complicate matters with /var/db/pkg. 595.Sh BUILDING THE WORLD UNDER A VKERNEL 596The virtual kernel platform does not have all the header files expected 597by a world build, so the easiest thing to do right now is to specify a 598pc64 (in a 64 bit vkernel) target when building the world under a virtual 599kernel, like this: 600.Bd -literal 601vkernel# make MACHINE_PLATFORM=pc64 buildworld 602vkernel# make MACHINE_PLATFORM=pc64 installworld 603.Ed 604.Sh SEE ALSO 605.Xr vknet 1 , 606.Xr bridge 4 , 607.Xr ifmedia 4 , 608.Xr tap 4 , 609.Xr vn 4 , 610.Xr sysctl.conf 5 , 611.Xr build 7 , 612.Xr config 8 , 613.Xr disklabel 8 , 614.Xr ifconfig 8 , 615.Xr vknetd 8 , 616.Xr vnconfig 8 617.Rs 618.%A Aggelos Economopoulos 619.%D March 2007 620.%T "A Peek at the DragonFly Virtual Kernel" 621.Re 622.Sh HISTORY 623Virtual kernels were introduced in 624.Dx 1.7 . 625.Sh AUTHORS 626.An -nosplit 627.An Matt Dillon 628thought up and implemented the 629.Nm 630architecture and wrote the 631.Nm vkd 632device driver. 633.An Sepherosa Ziehau 634wrote the 635.Nm vke 636device driver. 637This manual page was written by 638.An Sascha Wildner . 639