man/man4/nvme.4

.\" Copyright (c) 2016 The DragonFly Project.  All rights reserved.
.\"
.\" This code is derived from software contributed to The DragonFly Project
.\" by Matthew Dillon <dillon@backplane.com>
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\"
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in
.\"    the documentation and/or other materials provided with the
.\"    distribution.
.\" 3. Neither the name of The DragonFly Project nor the names of its
.\"    contributors may be used to endorse or promote products derived
.\"    from this software without specific, prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE
.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING,
.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd June 5, 2015
.Dt NVME 4
.Os
.Sh NAME
.Nm nvme
.Nd NVM Express Controller for PCIe-based SSDs
.Sh SYNOPSIS
To compile this driver into the kernel,
place the following line in your
kernel configuration file:
.Bd -ragged -offset indent
.Cd "device nvme"
.Ed
.Pp
Alternatively, to load the driver as a
module at boot time, place the following line in
.Xr loader.conf 5 :
.Bd -literal -offset indent
nvme_load="YES"
.Ed
.Sh DESCRIPTION
The
.Nm
driver provides support for PCIe storage controllers conforming to the
NVM Express Controller Interface specification.
NVMe controllers have a direct PCIe host interface to the controller
which in turn has a direct connection to the underlying non-volatile
(typically flash) storage, yielding a huge reduction in latency and
increase in performance over
.Xr ahci 4 .
.Pp
In addition, NVMe controllers are capable of supporting up to 65535
independent submission and completion queues each able to support upwards
of 16384 queue entries.  Each queue may be assigned its own interrupt
vector out of the controller's pool (up to 2048).
.Pp
Actual controllers typically implement lower limits.  While most controllers
allow a maximal number of queue entries, the total number of queues is often
limited to far less than 65535.  8-32 queues are commonly supported.
Similarly, while up to 2048 MSI-X vectors can be supported by the spec,
actual controllers typically support fewer vectors.  Still, having several
MSI-X vectors allows interrupts to be distributed to multiple CPUs,
reducing bottlenecks and improving performance.  The multiple queues can
be divvied up across available cpu cores by the driver, as well as split-up
based on the type of I/O operation being performed (such as giving read
and write I/O commands their own queues).  This also significantly
reduces bottlenecks and improves performance, particularly in mixed
read-write environments.
.Sh FORM FACTOR
NVMe boards usually come in one of two flavors, either a tiny form-factor
with a M.2 or NGFF connector, supplying 2 or 4 PCIe lanes, or in a larger
form that slips into a normal PCIe slot.  The larger form typically
implements 2, 4, or 8 lanes.  Also note that adapter cards that fit
into normal PCIe slots and can mount the smaller M.2/NGFF NVME cards can
be cheaply purchased.
.Sh PERFORMANCE
Typical performance for a 2-lane (x2) board is in the 700MB/s to 1.5 GByte/s
range.  4-lane (x4) boards typically range from 1.0 GBytes/s to 2.5 GBytes/s.
Full-blown PCIe cards run the whole gamut, 2.5 GBytes/sec is fairly typical
but performance can exceed 5 GBytes/sec in a high-end card.
.Pp
Multi-threaded random-read performance can exceed 300,000 IOPS on an x4 board.
Single-threaded performance is usually in the 40,000 to 100,000 IOPS range.
Sequential submission/completion latencies are typically below 35uS while
random submission/completion latencies are typically below 110uS.
Performance (uncached) through a filesystem will be bottlenecked by additional
factors, particularly if testing is only being done on a single file.
.Pp
The biggest differentiation between boards is usually write performance.
Small boards with only a few flash chips have relatively low write
performance, usually in the 150MByte/sec range.  Higher-end boards will have
significantly better write performance, potentially exceeding 1.0 GByte/sec.
.Pp
For reference, the SATA-III physical interface is limited to 600 MBytes/sec
and the extra phy layer results in higher latencies, and AHCI controllers are
limited to a single 32-entry queue.
.Sh FEATURES
The
.Dx
.Nm
driver automatically selects the best SMP-friendly and
I/O-typing queue configuration possible based on what the controller
supports.
It uses a direct disk device API which bypasses CAM, so kernel code paths
to read and write blocks are SMP-friendly and, depending on the queue
configuration, potentially conflict-free.
The driver is capable of submitting commands and processing responses on
multiple queues simultaniously in a SMP environment.
.Pp
The driver pre-reserves DMA memory for all necessary descriptors, queue
entries, and internal driver structures, and allows for a very generous
number of queue entries (1024 x NQueues) for maximum performance.
.Sh HINTS ON NVME CARDS
So far I've only been able to test one Samsung NVME M.2 card and
an Intel 750 HHHL (half-height / half-length) PCIe card.
.Pp
My recommendation is to go with Samsung.  The firmware is pretty good.
It appears to be implemented reasonably well regardless of the queue
configuration or I/O blocksize employed, giving expected scaling without
any quirky behavior.
.Pp
The intel 750 has very poorly-implemented firmware.
For example, the more queues the driver configures, the poorer
the single-threaded read performance is.  And no matter the queue
configuration it appears that adding a second concurrent reader drops
performance drastically, then it slowly increases as you add more concurrent
readers.  In addition, on the 750, the firmware degrades horribly when
reads use a blocksize of 64KB.  The best performance is at 32KB.  In fact,
performance again degrades horribly if you drop down to 16KB.
And if that weren't bad enough, the 750 takes over 13 seconds to become
ready after a machine power-up or reset.
.Pp
The grand result of all of this is that filesystem performance through an
Intel NVME card is going to be hit-or-miss, depending on inconseqential
differences in blocksize and queue configuration.
Regardless of whatever hacks Intel might be employing in their own drivers,
this is just totally unacceptable driver behavior.
.Pp
I do not recommend rebranders like Plextor or Kingston.  For one thing,
if you do buy these, be very careful to get one that is actually a NVME
card and not a M.2 card with an AHCI controller on it.  Plextor's performance
is particularly bad.  Kingston seems to have done a better job and reading
at 1.0GB/s+ is possible despite the cpu overhead of going through an AHCI
controller (the flash in both cases is directly connected to the controller,
so there is no SATA Phy to get in the way).  Of course, if you actually want
an AHCI card, then these might be the way to go, and you might even be able
to boot from them.
.Sh HINTS ON CONFIGURING MACHINES (BIOS)
If nvme locks up while trying to probe the BIOS did something horrible to
the PCIe card.  If you have enabled your BIOS's FastBoot option, turn it
off, this may fix the issue.
.Pp
Not all BIOSes can boot from a NVMe card.  Those that can typically require
booting via EFI.
.Sh SEE ALSO
.Xr ahci 4 ,
.Xr intro 4 ,
.Xr pci 4 ,
.Xr loader.conf 5 ,
.Xr nvmectl 8
.Sh HISTORY
The
.Nm
driver first appeared in
.Dx 4.5 .
.Sh AUTHORS
.An -nosplit
The
.Nm
driver for
.Dx
was written from scratch by
.An Matthew Dillon Aq Mt dillon@backplane.com
based on the NVM Express 1.2a specification.