ctl_backend_block: Correct value of NVME nsdata nuse field

Reported by: Coverity Scan
CID: 1545043
Sponsored by: Chelsio Communications

ctl_backend_block: Add support for NVMe

Reviewed by: imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44723

ctl_backend_block: Prepare for NVMe support

- Use wrapper routines for access to shared fields between SCSI and
NVMe I/O requests.

- Use protocol-agnostic wrapper routines for I/O completion stat

ctl_backend_block: Prepare for NVMe support

- Use wrapper routines for access to shared fields between SCSI and
NVMe I/O requests.

- Use protocol-agnostic wrapper routines for I/O completion status.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44851

show more ...

ctl: Add CTL_IO_ASSERT wrapper macro

Currently, this pattern is commonly used to assert that a union ctl_io
is a SCSI request. In the future it will be used to assert other
types.

Suggested by: im

ctl: Add CTL_IO_ASSERT wrapper macro

Currently, this pattern is commonly used to assert that a union ctl_io
is a SCSI request. In the future it will be used to assert other
types.

Suggested by: imp
Reviewed by: imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44844

show more ...

CAM: Replace random sbuf_printf() with cheaper cat/putc.

cam: Remove left-over sys/cdefs.h in sys/cam

These weren't removed when $FreeBSD$ was removed. They aren't needed and
now are a style(9) nonconformity.

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd)

CTL: Allow I/Os up to 8MB, depending on maxphys value.

For years CTL block backend limited I/O size to 1MB, splitting larger
requests into sequentially processed chunks. It is sufficient for
most o

CTL: Allow I/Os up to 8MB, depending on maxphys value.

For years CTL block backend limited I/O size to 1MB, splitting larger
requests into sequentially processed chunks. It is sufficient for
most of use cases, since typical initiators rarely use bigger I/Os.

One of known exceptions is VMWare VAAI offload, by default sending up
to 8 4MB EXTENDED COPY requests same time. CTL internally converted
those into 32 1MB READ/WRITE requests, that could overwhelm the block
backend, having finite number of processing threads and making more
important interactive I/Os to wait in its queue. Previously it was
partially covered by CTL core serializing sequential reads to help
ZFS speculative prefetcher, but that serialization was significantly
relaxed after recent ZFS improvements.

With the new settings block backend receives 8 4MB requests, that
should be easier for both CTL itself and the underlying storage.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

show more ...

vfs: remove the unused thread argument from NDINIT*

See b4a58fbf640409a1 ("vfs: remove cn_thread")

Bump __FreeBSD_version to 1400043.

ctl(4): Do hole-punching for UNMAP to file-backed LUNs

This adds support for SCSI UNMAP command to file-backed LUNs, if the
underlying file system has a non-zerofilling VOP_DEALLOCATE
implementation

ctl(4): Do hole-punching for UNMAP to file-backed LUNs

This adds support for SCSI UNMAP command to file-backed LUNs, if the
underlying file system has a non-zerofilling VOP_DEALLOCATE
implementation where some or all parts of the requested operation range
may be deallocated.

Sponsored by: The FreeBSD Foundation
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D31922

show more ...

Add vn_lktype_write()

and remove repetetive code that calculates vnode locking type for write.

Reviewed by: khng, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision:

Add vn_lktype_write()

and remove repetetive code that calculates vnode locking type for write.

Reviewed by: khng, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31405

show more ...

Introduce "soft" serseq variant.

With new ZFS prefetcher improvements it is no longer needed to fully
serialize reads to reach decent prediction hit rate. Softer variant
only creates small time win

Introduce "soft" serseq variant.

With new ZFS prefetcher improvements it is no longer needed to fully
serialize reads to reach decent prediction hit rate. Softer variant
only creates small time window to reduce races instead of completely
blocking following reads while previous is running. It much less
hurts the performance in case of prediction miss.

MFC after: 1 month

show more ...

Do not exit ctl_be_block_worker() prematurely.

Return while there are any I/Os in a queue may result in them stuck
indefinitely, since there is only one taskqueue task for all of them.
I think I've

Do not exit ctl_be_block_worker() prematurely.

Return while there are any I/Os in a queue may result in them stuck
indefinitely, since there is only one taskqueue task for all of them.
I think I've reproduced this by switching ha_role to secondary under
heavy load.

MFC after: 3 days

show more ...

Optimize out few extra memory accesses.

MFC after: 1 week

Bump CTL block backend threads from 14 to 32 per LUN.

This makes random read benchmarks look better on a wide ZFS pools.
I am not sure where the original value goes from, but it is there
for too lon

Bump CTL block backend threads from 14 to 32 per LUN.

This makes random read benchmarks look better on a wide ZFS pools.
I am not sure where the original value goes from, but it is there
for too long now.

MFC after: 1 week

show more ...

Refactor CTL datamove KPI.

- Make frontends call unified CTL core method ctl_datamove_done()
to report move completion. It allows to reduce code duplication
in differerent backends by accounting D

Refactor CTL datamove KPI.

- Make frontends call unified CTL core method ctl_datamove_done()
to report move completion. It allows to reduce code duplication
in differerent backends by accounting DMA time in common code.
- Add to ctl_datamove_done() and be_move_done() callback samethr
argument, reporting whether the callback is called in the same
context as ctl_datamove(). It allows for some cases like iSCSI
write with immediate data or camsim frontend write save one context
switch, since we know that the context is sleepable.
- Remove data_move_done() methods from struct ctl_backend_driver,
unused since forever.

MFC after: 1 month

show more ...

Microoptimize CTL I/O queues.

Switch OOA queue from TAILQ to LIST and change its direction, so that
we traverse it forward, not backward. There is only one place where
we really need other directio

Microoptimize CTL I/O queues.

Switch OOA queue from TAILQ to LIST and change its direction, so that
we traverse it forward, not backward. There is only one place where
we really need other direction, and it is not critical.

Use STAILQ_REMOVE_HEAD() instead of STAILQ_REMOVE() in backends.

Replace few impossible conditions with assertions.

MFC after: 1 month

show more ...

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pag

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

show more ...

Make CTL nicer to increased MAXPHYS.

Before this CTL always allocated MAXPHYS-sized buffers, even for 4KB I/O,
that is even more overkill for MAXPHYS of 1MB. This change limits maximum
allocation t

Make CTL nicer to increased MAXPHYS.

Before this CTL always allocated MAXPHYS-sized buffers, even for 4KB I/O,
that is even more overkill for MAXPHYS of 1MB. This change limits maximum
allocation to 512KB if MAXPHYS is bigger, plus if one is above 128KB, adds
new 128KB UMA zone for smaller I/Os. The patch factors out alloc/free,
so later we could make it use more zones or malloc() if we'd like.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

show more ...

cam: clean up empty lines in .c and .h files

vfs: drop the error parameter from vn_isdisk, introduce vn_isdisk_error

Most consumers pass NULL.

Implement zero-copy iSCSI target transmission/read.

Add ICL_NOCOPY flag to icl_pdu_append_data(), specifying that the method
can just reference the data buffer instead of immediately copying it.

Ex

Implement zero-copy iSCSI target transmission/read.

Add ICL_NOCOPY flag to icl_pdu_append_data(), specifying that the method
can just reference the data buffer instead of immediately copying it.

Extend the offload KPI with optional PDU queue method, allowing to specify
completion callback, called when all the data referenced by above has been
transferred and won't be accessed any more (the buffers can be freed).

Implement the above functionality in software iSCSI driver using mbufs
with external storage and reference counter. Note that some NICs (ixl(4))
may keep the mbuf in TX queue for a long time, so CTL has to be ready.

Add optional method to struct ctl_scsiio for buffer reference counting.
Implement it for CTL block backend, allowing to delay free of the struct
ctl_be_block_io and memory it references as needed. In first reincarnation
of the patch I tried to delay whole I/O as it is done for FibreChannel,
that was cleaner, but due to the above callback delays I had to rewrite
it this way to not leave LUN referenced potentially for hours or more.

All together on sequential read from ZFS ARC this saves about 30% of CPU
time and memory bandwidth by avoiding one of 3 memory copies (the other
two are from ZFS ARC to DMU cache and then from DMU cache to CTL buffers).
On tests with 2x Xeon Silver 4114 this allows to reach full line rate of
100GigE NIC. Tests with Gold CPUs and two 100GigE NICs are stil TBD,
but expectations to saturate them are pretty high. ;)

Discussed with: Chelsio
Sponsored by: iXsystems, Inc.

show more ...

Make struct ctl_be_lun first element of struct ctl_be_*_lun.

It allows to remove some extra pointer dereferences and slightly tightens
up the code by unification.

MFC after: 2 weeks
Sponsored by: i

Make struct ctl_be_lun first element of struct ctl_be_*_lun.

It allows to remove some extra pointer dereferences and slightly tightens
up the code by unification.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

show more ...