1.\" 2.\" Copyright (c) 2006, 2007 3.\" The DragonFly Project. All rights reserved. 4.\" 5.\" Redistribution and use in source and binary forms, with or without 6.\" modification, are permitted provided that the following conditions 7.\" are met: 8.\" 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in 13.\" the documentation and/or other materials provided with the 14.\" distribution. 15.\" 3. Neither the name of The DragonFly Project nor the names of its 16.\" contributors may be used to endorse or promote products derived 17.\" from this software without specific, prior written permission. 18.\" 19.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 22.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 23.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 25.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 26.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 27.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 29.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 30.\" SUCH DAMAGE. 31.\" 32.Dd January 5, 2019 33.Dt VKERNEL 7 34.Os 35.Sh NAME 36.Nm vkernel , 37.Nm vcd , 38.Nm vkd , 39.Nm vke 40.Nd virtual kernel architecture 41.Sh SYNOPSIS 42.Cd "platform vkernel64 # for 64 bit vkernels" 43.Cd "device vcd" 44.Cd "device vkd" 45.Cd "device vke" 46.Pp 47.Pa /var/vkernel/boot/kernel/kernel 48.Op Fl hdstUvz 49.Op Fl c Ar file 50.Op Fl e Ar name Ns = Ns Li value : Ns Ar name Ns = Ns Li value : Ns ... 51.Op Fl i Ar file 52.Op Fl I Ar interface Ns Op Ar :address1 Ns Oo Ar :address2 Oc Ns Oo Ar /netmask Oc Ns Oo Ar =mac Oc 53.Op Fl l Ar cpulock 54.Op Fl m Ar size 55.Op Fl n Ar numcpus Ns Op Ar :lbits Ns Oo Ar :cbits Oc 56.Op Fl p Ar pidfile 57.Op Fl r Ar file Ns Op Ar :serno 58.Op Fl R Ar file Ns Op Ar :serno 59.Sh DESCRIPTION 60The 61.Nm 62architecture allows for running 63.Dx 64kernels in userland. 65.Pp 66The following options are available: 67.Bl -tag -width ".Fl m Ar size" 68.It Fl c Ar file 69Specify a readonly CD-ROM image 70.Ar file 71to be used by the kernel, with the first 72.Fl c 73option defining 74.Li vcd0 , 75the second one 76.Li vcd1 , 77and so on. 78The first 79.Fl r , 80.Fl R , 81or 82.Fl c 83option specified on the command line will be the boot disk. 84The CD9660 filesystem is assumed when booting from this media. 85.It Fl d 86Disables hardware pagetable for 87.Nm . 88.It Fl e Ar name Ns = Ns Li value : Ns Ar name Ns = Ns Li value : Ns ... 89Specify an environment to be used by the kernel. 90This option can be specified more than once. 91.It Fl h 92Shows a list of available options, each with a short description. 93.It Fl i Ar file 94Specify a memory image 95.Ar file 96to be used by the virtual kernel. 97If no 98.Fl i 99option is given, the kernel will generate a name of the form 100.Pa /var/vkernel/memimg.XXXXXX , 101with the trailing 102.Ql X Ns s 103being replaced by a sequential number, e.g.\& 104.Pa memimg.000001 . 105.It Fl I Ar interface Ns Op Ar :address1 Ns Oo Ar :address2 Oc Ns Oo Ar /netmask Oc Ns Oo Ar =MAC Oc 106Create a virtual network device, with the first 107.Fl I 108option defining 109.Li vke0 , 110the second one 111.Li vke1 , 112and so on. 113.Pp 114The 115.Ar interface 116argument is the name of a 117.Xr tap 4 118device node or the path to a 119.Xr vknetd 8 120socket. 121The 122.Pa /dev/ 123path prefix does not have to be specified and will be automatically prepended 124for a device node. 125Specifying 126.Cm auto 127will pick the first unused 128.Xr tap 4 129device. 130.Pp 131The 132.Ar address1 133and 134.Ar address2 135arguments are the IP addresses of the 136.Xr tap 4 137and 138.Nm vke 139interfaces. 140Optionally, 141.Ar address1 142may be of the form 143.Li bridge Ns Em X 144in which case the 145.Xr tap 4 146interface is added to the specified 147.Xr bridge 4 148interface. 149The 150.Nm vke 151address is not assigned until the interface is brought up in the guest. 152.Pp 153The 154.Ar netmask 155argument applies to all interfaces for which an address is specified. 156.Pp 157The 158.Ar MAC 159argument is the MAC address of the 160.Xr vke 4 161interface. 162If not specified, a pseudo-random one will be generated. 163.Pp 164When running multiple vkernels it is often more convenient to simply 165connect to a 166.Xr vknetd 8 167socket and let vknetd deal with the tap and/or bridge. 168An example of this would be 169.Pa /var/run/vknet:0.0.0.0:10.2.0.2/16 . 170.It Fl l Ar cpulock 171Specify which, if any, real CPUs to lock virtual CPUs to. 172.Ar cpulock 173is one of 174.Cm any , 175.Cm map Ns Op Ns , Ns Ar startCPU , 176or 177.Ar CPU . 178.Pp 179.Cm any 180does not map virtual CPUs to real CPUs. 181This is the default. 182.Pp 183.Cm map Ns Op Ns , Ns Ar startCPU 184maps each virtual CPU to a real CPU starting with real CPU 0 or 185.Ar startCPU 186if specified. 187.Pp 188.Ar CPU 189locks all virtual CPUs to the real CPU specified by 190.Ar CPU . 191.Pp 192Locking the vkernel to a set of cpus is recommended on multi-socket systems 193to improve NUMA locality of reference. 194.It Fl m Ar size 195Specify the amount of memory to be used by the kernel in bytes, 196.Cm K 197.Pq kilobytes , 198.Cm M 199.Pq megabytes 200or 201.Cm G 202.Pq gigabytes . 203Lowercase versions of 204.Cm K , M , 205and 206.Cm G 207are allowed. 208.It Fl n Ar numcpus Ns Op Ar :lbits Ns Oo Ar :cbits Oc 209.Ar numcpus 210specifies the number of CPUs you wish to emulate. 211Up to 16 CPUs are supported with 2 being the default unless otherwise 212specified. 213.Pp 214.Ar lbits 215specifies the number of bits within APICID(=CPUID) needed for representing 216the logical ID. 217Controls the number of threads/core (0 bits - 1 thread, 1 bit - 2 threads). 218This parameter is optional (mandatory only if 219.Ar cbits 220is specified). 221.Pp 222.Ar cbits 223specifies the number of bits within APICID(=CPUID) needed for representing 224the core ID. 225Controls the number of core/package (0 bits - 1 core, 1 bit - 2 cores). 226This parameter is optional. 227.It Fl p Ar pidfile 228Specify a pidfile in which to store the process ID. 229Scripts can use this file to locate the vkernel pid for the purpose of 230shutting down or killing it. 231.Pp 232The vkernel will hold a lock on the pidfile while running. 233Scripts may test for the lock to determine if the pidfile is valid or 234stale so as to avoid accidentally killing a random process. 235Something like '/usr/bin/lockf -ks -t 0 pidfile echo -n' may be used 236to test the lock. 237A non-zero exit code indicates that the pidfile represents a running 238vkernel. 239.Pp 240An error is issued and the vkernel exits if this file cannot be opened for 241writing or if it is already locked by an active vkernel process. 242.It Fl r Ar file Ns Op Ar :serno 243Specify a R/W disk image 244.Ar file 245to be used by the kernel, with the first 246.Fl r 247option defining 248.Li vkd0 , 249the second one 250.Li vkd1 , 251and so on. 252A serial number for the virtual disk can be specified in 253.Ar serno . 254.Pp 255The first 256.Fl r 257or 258.Fl c 259option specified on the command line will be the boot disk. 260.It Fl R Ar file Ns Op Ar :serno 261Works like 262.Fl r 263but treats the disk image as copy-on-write. This allows 264a private copy of the image to be modified but does not 265modify the image file. The image file will not be locked 266in this situation and multiple vkernels can run off the 267same image file if desired. 268.Pp 269Since modifications are thrown away, any data you wish 270to retain across invocations needs to be exported over 271the network prior to shutdown. 272This gives you the flexibility to mount the disk image 273either read-only or read-write depending on what is 274convenient. 275However, keep in mind that when mounting a COW image 276read-write, modifications will eat system memory and 277swap space until the vkernel is shut down. 278.It Fl s 279Boot into single-user mode. 280.It Fl t 281Tell the vkernel to use a precise host timer when calculating clock values. 282If the TSC isn't used, this will impose higher overhead on the vkernel as it 283will have to make a system call to the real host every time it wants to get 284the time. 285However, the more precise timer might be necessary for your application. 286.Pp 287By default, the vkernel uses the TSC cpu timer if possible, or an imprecise 288(host-tick-resolution) timer which uses a user-mapped kernel page and does 289not have any syscall overhead. 290To disable the TSC cpu timer, use the 291.Fl e Ar hw.tsc_cputimer_enable=0 292flag. 293.It Fl U 294Enable writing to kernel memory and module loading. 295By default, those are disabled for security reasons. 296.It Fl v 297Turn on verbose booting. 298.It Fl z 299Force the vkernel's ram to be pre-zerod. Useful for benchmarking on 300single-socket systems where the memory allocation does not have to be 301NUMA-friendly. 302This options is not recommended on multi-socket systems or when the 303.Fl l 304option is used. 305.El 306.Sh DEVICES 307A number of virtual device drivers exist to supplement the virtual kernel. 308.Ss Disk device 309The 310.Nm vkd 311driver allows for up to 16 312.Xr vn 4 313based disk devices. 314The root device will be 315.Li vkd0 316(see 317.Sx EXAMPLES 318for further information on how to prepare a root image). 319.Ss CD-ROM device 320The 321.Nm vcd 322driver allows for up to 16 virtual CD-ROM devices. 323Basically this is a read only 324.Nm vkd 325device with a block size of 2048. 326.Ss Network interface 327The 328.Nm vke 329driver supports up to 16 virtual network interfaces which are associated with 330.Xr tap 4 331devices on the host. 332For each 333.Nm vke 334device, the per-interface read only 335.Xr sysctl 3 336variable 337.Va hw.vke Ns Em X Ns Va .tap_unit 338holds the unit number of the associated 339.Xr tap 4 340device. 341.Pp 342By default, half of the total mbuf clusters available is distributed equally 343among all the vke devices up to 256. 344This can be overridden with the tunable 345.Va hw.vke.max_ringsize . 346Take into account the number passed will be aligned to the lower power of two. 347.Sh SIGNALS 348The virtual kernel only enables 349.Dv SIGQUIT 350and 351.Dv SIGTERM 352while operating in regular console mode. 353Sending 354.Ql \&^\e 355.Pq Dv SIGQUIT 356to the virtual kernel causes the virtual kernel to enter its internal 357.Xr ddb 4 358debugger and re-enable all other terminal signals. 359Sending 360.Dv SIGTERM 361to the virtual kernel triggers a clean shutdown by passing a 362.Dv SIGUSR2 363to the virtual kernel's 364.Xr init 8 365process. 366.Sh DEBUGGING 367It is possible to directly gdb the virtual kernel's process. 368It is recommended that you do a 369.Ql handle SIGSEGV noprint 370to ignore page faults processed by the virtual kernel itself and 371.Ql handle SIGUSR1 noprint 372to ignore signals used for simulating inter-processor interrupts. 373.Sh FILES 374.Bl -tag -width ".It Pa /sys/config/VKERNEL64" -compact 375.It Pa /dev/vcdX 376.Nm vcd 377device nodes 378.It Pa /dev/vkdX 379.Nm vkd 380device nodes 381.It Pa /sys/config/VKERNEL64 382.El 383.Pp 384.Nm 385configuration file, for 386.Xr config 8 . 387.Sh CONFIGURATION FILES 388Your virtual kernel is a complete 389.Dx 390system, but you might not want to run all the services a normal kernel runs. 391Here is what a typical virtual kernel's 392.Pa /etc/rc.conf 393file looks like, with some additional possibilities commented out. 394.Bd -literal 395hostname="vkernel" 396network_interfaces="lo0 vke0" 397ifconfig_vke0="DHCP" 398sendmail_enable="NO" 399#syslog_enable="NO" 400blanktime="NO" 401.Ed 402.Sh BOOT DRIVE SELECTION 403You can override the default boot drive selection and filesystem 404using a kernel environment variable. Note that the filesystem 405selected must be compiled into the vkernel and not loaded as 406a module. You need to escape some quotes around the variable data 407to avoid mis-interpretation of the colon in the 408.Fl e 409option. For example: 410.Pp 411.Fl e 412vfs.root.mountfrom=\\"hammer:vkd0s1d\\" 413.Sh DISKLESS OPERATION 414To boot a 415.Nm 416from a NFS root, a number of tunables need to be set: 417.Bl -tag -width indent 418.It Va boot.netif.ip 419IP address to be set in the vkernel interface. 420.It Va boot.netif.netmask 421Netmask for the IP to be set. 422.It Va boot.netif.name 423Network interface name inside the vkernel. 424.It Va boot.nfsroot.server 425Host running 426.Xr nfsd 8 . 427.It Va boot.nfsroot.path 428Host path where a world and distribution 429targets are properly installed. 430.El 431.Pp 432See an example on how to boot a diskless 433.Nm 434in the 435.Sx EXAMPLES 436section. 437.Sh EXAMPLES 438A couple of steps are necessary in order to prepare the system to build and 439run a virtual kernel. 440.Ss Setting up the filesystem 441The 442.Nm 443architecture needs a number of files which reside in 444.Pa /var/vkernel . 445Since these files tend to get rather big and the 446.Pa /var 447partition is usually of limited size, we recommend the directory to be 448created in the 449.Pa /home 450partition with a link to it in 451.Pa /var : 452.Bd -literal 453mkdir -p /home/var.vkernel/boot 454ln -s /home/var.vkernel /var/vkernel 455.Ed 456.Pp 457Next, a filesystem image to be used by the virtual kernel has to be 458created and populated (assuming world has been built previously). 459If the image is created on a UFS filesystem you might want to pre-zero it. 460On a HAMMER filesystem you should just truncate-extend to the image size 461as HAMMER does not re-use data blocks already present in the file. 462.Bd -literal 463vnconfig -c -S 2g -T vn0 /var/vkernel/rootimg.01 464disklabel -r -w vn0s0 auto 465disklabel -e vn0s0 # add `a' partition with fstype `4.2BSD' 466newfs /dev/vn0s0a 467mount /dev/vn0s0a /mnt 468cd /usr/src 469make installworld DESTDIR=/mnt 470cd etc 471make distribution DESTDIR=/mnt 472echo '/dev/vkd0s0a / ufs rw 1 1' >/mnt/etc/fstab 473echo 'proc /proc procfs rw 0 0' >>/mnt/etc/fstab 474.Ed 475.Pp 476Edit 477.Pa /mnt/etc/ttys 478and replace the 479.Li console 480entry with the following line and turn off all other gettys. 481.Bd -literal 482console "/usr/libexec/getty Pc" cons25 on secure 483.Ed 484.Pp 485Replace 486.Li \&Pc 487with 488.Li al.Pc 489if you would like to automatically log in as root. 490.Pp 491Then, unmount the disk. 492.Bd -literal 493umount /mnt 494vnconfig -u vn0 495.Ed 496.Ss Compiling the virtual kernel 497In order to compile a virtual kernel use the 498.Li VKERNEL64 499kernel configuration file residing in 500.Pa /sys/config 501(or a configuration file derived thereof): 502.Bd -literal 503cd /usr/src 504make -DNO_MODULES buildkernel KERNCONF=VKERNEL64 505make -DNO_MODULES installkernel KERNCONF=VKERNEL64 DESTDIR=/var/vkernel 506.Ed 507.Ss Enabling virtual kernel operation 508A special 509.Xr sysctl 8 , 510.Va vm.vkernel_enable , 511must be set to enable 512.Nm 513operation: 514.Bd -literal 515sysctl vm.vkernel_enable=1 516.Ed 517.Ss Configuring the network on the host system 518In order to access a network interface of the host system from the 519.Nm , 520you must add the interface to a 521.Xr bridge 4 522device which will then be passed to the 523.Fl I 524option: 525.Bd -literal 526kldload if_bridge.ko 527kldload if_tap.ko 528ifconfig bridge0 create 529ifconfig bridge0 addm re0 # assuming re0 is the host's interface 530ifconfig bridge0 up 531.Ed 532.Ss Running the kernel 533Finally, the virtual kernel can be run: 534.Bd -literal 535cd /var/vkernel 536\&./boot/kernel/kernel -m 1g -r rootimg.01 -I auto:bridge0 537.Ed 538.Pp 539You can issue the 540.Xr reboot 8 , 541.Xr halt 8 , 542or 543.Xr shutdown 8 544commands from inside a virtual kernel. 545After doing a clean shutdown the 546.Xr reboot 8 547command will re-exec the virtual kernel binary while the other two will 548cause the virtual kernel to exit. 549.Ss Diskless operation (vkernel as a NFS client) 550Booting a 551.Nm 552with a 553.Xr vknetd 8 554network configuration. The line continuation backslashes have been 555omitted. For convenience and to reduce confusion I recommend mounting 556the server's remote vkernel root onto the host running the vkernel binary 557using the same path as the NFS mount. It is assumed that a full system 558install has been made to /var/vkernel/root using a kernel KERNCONF=VKERNEL64 559for the kernel build. 560.Bd -literal 561\&/var/vkernel/root/boot/kernel/kernel 562 -m 1g -n 4 -I /var/run/vknet 563 -e boot.netif.ip=10.100.0.2 564 -e boot.netif.netmask=255.255.0.0 565 -e boot.netif.gateway=10.100.0.1 566 -e boot.netif.name=vke0 567 -e boot.nfsroot.server=10.0.0.55 568 -e boot.nfsroot.path=/var/vkernel/root 569.Ed 570.Pp 571In this example vknetd is assumed to have been started as shown below, before 572running the vkernel, using an unbridged TAP configuration routed through 573the host. 574IP forwarding must be turned on, and in this example the server resides 575on a different network accessible to the host executing the vkernel but not 576directly on the vkernel's subnet. 577.Bd -literal 578kldload if_tap 579sysctl net.inet.ip.forwarding=1 580vknetd -t tap0 10.100.0.1/16 581.Ed 582.Pp 583You can run multiple vkernels trivially with the same NFS root as long as 584you assign each one a different IP on the subnet (2, 3, 4, etc). You 585should also be careful with certain directories, particularly /var/run 586and possibly also /var/db depending on what your vkernels are going to be 587doing. 588This can complicate matters with /var/db/pkg. 589.Sh BUILDING THE WORLD UNDER A VKERNEL 590The virtual kernel platform does not have all the header files expected 591by a world build, so the easiest thing to do right now is to specify a 592pc64 (in a 64 bit vkernel) target when building the world under a virtual 593kernel, like this: 594.Bd -literal 595vkernel# make MACHINE_PLATFORM=pc64 buildworld 596vkernel# make MACHINE_PLATFORM=pc64 installworld 597.Ed 598.Sh SEE ALSO 599.Xr vknet 1 , 600.Xr bridge 4 , 601.Xr ifmedia 4 , 602.Xr tap 4 , 603.Xr vn 4 , 604.Xr sysctl.conf 5 , 605.Xr build 7 , 606.Xr config 8 , 607.Xr disklabel 8 , 608.Xr ifconfig 8 , 609.Xr vknetd 8 , 610.Xr vnconfig 8 611.Rs 612.%A Aggelos Economopoulos 613.%D March 2007 614.%T "A Peek at the DragonFly Virtual Kernel" 615.Re 616.Sh HISTORY 617Virtual kernels were introduced in 618.Dx 1.7 . 619.Sh AUTHORS 620.An -nosplit 621.An Matt Dillon 622thought up and implemented the 623.Nm 624architecture and wrote the 625.Nm vkd 626device driver. 627.An Sepherosa Ziehau 628wrote the 629.Nm vke 630device driver. 631This manual page was written by 632.An Sascha Wildner . 633