usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

Change "ctlstat -P"'s schema

It now reports stats separately for both ports and luns.

Also, prohibit using both "-p" and "-P" at the same time.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by:

Change "ctlstat -P"'s schema

It now reports stats separately for both ports and luns.

Also, prohibit using both "-p" and "-P" at the same time.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D38500

show more ...

ctlstat: add prometheus output

When invoked by inetd, ctlstat -P will now produce output suitable for
ingestion into Prometheus.

It's a drop-in replacement for https://github.com/Gandi/ctld_exporte

ctlstat: add prometheus output

When invoked by inetd, ctlstat -P will now produce output suitable for
ingestion into Prometheus.

It's a drop-in replacement for https://github.com/Gandi/ctld_exporter,
except that it doesn't report the number of initiators per target, and
it does report time and dma_time.

MFC after: 2 weeks
Sponsored by: Axcient
Relnotes: yes
Reviewed by: bapt, bcr
Differential Revision: https://reviews.freebsd.org/D29901

show more ...

Flush stdout after each iteration.

Without this, if output is redirected from the console, it is buffered for
too long, making tool quite unusable.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Remove hard coded number of lun definition

The number of lun exposed is now exposed via sysctl by the kernel.
Use that number in ctlstat instead of the hardcoded version
Add a backward compatibility

Remove hard coded number of lun definition

The number of lun exposed is now exposed via sysctl by the kernel.
Use that number in ctlstat instead of the hardcoded version
Add a backward compatibility in case the sysctl(2) request fails.

This also allows ctlstat -l 1118 to actually work when having more than
1024 luns.

Reviewed by: avg, manu (both before the backward compatibility addition)
Approved by: avg, manu (both before the backward compatibility addition)
MFC after: 2 weeks
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D13446

show more ...

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Make ctlstat -n option work reasonably for sparse LUN list.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix JSON output.

MFC after: 1 week

Rewrite CTL statistics in more simple and scalable way.

Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, col

Rewrite CTL statistics in more simple and scalable way.

Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, collect
separate statistics for every port and every logical unit separately, that
consume only 176 bytes per each single LU/port. This reduces struct
ctl_lun size down to just 6KB.

Also new IOCTL API/ABI does not hardcode number of LUs/ports, and should
allow handling of very large quantities.

MFC after: 2 weeks (probably keeping old API enabled for some time)

show more ...

Bunch of improvements to ctlstat.

Add -p option to allow filtering by ports.
Make -l and -p options work in all modes as filters.
Improve output formatting to better fit columns.

Output times as normal microseconds, rather then in bintime format.

Drop "internal" CTL frontend.

Its idea was to be a simple initiator and execute several commands from
kernel level, but FreeBSD never had consumer for that functionality,
while its implementation po

Drop "internal" CTL frontend.

Its idea was to be a simple initiator and execute several commands from
kernel level, but FreeBSD never had consumer for that functionality,
while its implementation polluted many unrelated places..

show more ...

Fix couple issues in ctlstat header printing.

MFC after: 1 week

Fix bugs in the elapsed time calculation in ctlstat_standard()
pointed out by bde:

- Casting to long double isn't needed.
- The division isn't needed, multiplication can be used.
"When 1 nanosecon

Fix bugs in the elapsed time calculation in ctlstat_standard()
pointed out by bde:

- Casting to long double isn't needed.
- The division isn't needed, multiplication can be used.
"When 1 nanosecond is in a floating point literal, the whole
expression is automatically promoted correctly."
- non-KNF indentation (1 tab) for the newly split line
- different non-KNF indentation (5 spaces) for the previously split
line
- exessive parentheses around the division operation
- bogus blank line which splits up the etime initialization
- general verboseness from the above.

Submitted by: bde
MFC after: 3 days

show more ...

Fix a time calculation error in ctlstat_standard().

ctlstat.c: When converting a timeval to a floating point
number in ctlstat_standard(), cast the nanoseconds
calculation to a long double, so w

Fix a time calculation error in ctlstat_standard().

ctlstat.c: When converting a timeval to a floating point
number in ctlstat_standard(), cast the nanoseconds
calculation to a long double, so we don't lose
precision. Without the cast, we wind up with a
time in whole seconds only.

Sponsored by: Spectra Logic
MFC after: 3 days

show more ...

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variabl

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

show more ...

Fix building ctlstat with clang.

Submitted by: Dan McGregor <danismostlikely@gmail.com>
MFC after: 1 month

Add the CAM Target Layer (CTL).

CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003. It has been shipping in
Copan (now SGI) pro

Add the CAM Target Layer (CTL).

CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003. It has been shipping in
Copan (now SGI) products since 2005.

It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
available under a BSD-style license. The intent behind the agreement was
that Spectra would work to get CTL into the FreeBSD tree.

Some CTL features:

- Disk and processor device emulation.
- Tagged queueing
- SCSI task attribute support (ordered, head of queue, simple tags)
- SCSI implicit command ordering support. (e.g. if a read follows a mode
select, the read will be blocked until the mode select completes.)
- Full task management support (abort, LUN reset, target reset, etc.)
- Support for multiple ports
- Support for multiple simultaneous initiators
- Support for multiple simultaneous backing stores
- Persistent reservation support
- Mode sense/select support
- Error injection support
- High Availability support (1)
- All I/O handled in-kernel, no userland context switch overhead.

(1) HA Support is just an API stub, and needs much more to be fully
functional.

ctl.c: The core of CTL. Command handlers and processing,
character driver, and HA support are here.

ctl.h: Basic function declarations and data structures.

ctl_backend.c,
ctl_backend.h: The basic CTL backend API.

ctl_backend_block.c,
ctl_backend_block.h: The block and file backend. This allows for using
a disk or a file as the backing store for a LUN.
Multiple threads are started to do I/O to the
backing device, primarily because the VFS API
requires that to get any concurrency.

ctl_backend_ramdisk.c: A "fake" ramdisk backend. It only allocates a
small amount of memory to act as a source and sink
for reads and writes from an initiator. Therefore
it cannot be used for any real data, but it can be
used to test for throughput. It can also be used
to test initiators' support for extremely large LUNs.

ctl_cmd_table.c: This is a table with all 256 possible SCSI opcodes,
and command handler functions defined for supported
opcodes.

ctl_debug.h: Debugging support.

ctl_error.c,
ctl_error.h: CTL-specific wrappers around the CAM sense building
functions.

ctl_frontend.c,
ctl_frontend.h: These files define the basic CTL frontend port API.

ctl_frontend_cam_sim.c: This is a CTL frontend port that is also a CAM SIM.
This frontend allows for using CTL without any
target-capable hardware. So any LUNs you create in
CTL are visible in CAM via this port.

ctl_frontend_internal.c,
ctl_frontend_internal.h:
This is a frontend port written for Copan to do
some system-specific tasks that required sending
commands into CTL from inside the kernel. This
isn't entirely relevant to FreeBSD in general,
but can perhaps be repurposed.

ctl_ha.h: This is a stubbed-out High Availability API. Much
more is needed for full HA support. See the
comments in the header and the description of what
is needed in the README.ctl.txt file for more
details.

ctl_io.h: This defines most of the core CTL I/O structures.
union ctl_io is conceptually very similar to CAM's
union ccb.

ctl_ioctl.h: This defines all ioctls available through the CTL
character device, and the data structures needed
for those ioctls.

ctl_mem_pool.c,
ctl_mem_pool.h: Generic memory pool implementation used by the
internal frontend.

ctl_private.h: Private data structres (e.g. CTL softc) and
function prototypes. This also includes the SCSI
vendor and product names used by CTL.

ctl_scsi_all.c,
ctl_scsi_all.h: CTL wrappers around CAM sense printing functions.

ctl_ser_table.c: Command serialization table. This defines what
happens when one type of command is followed by
another type of command.

ctl_util.c,
ctl_util.h: CTL utility functions, primarily designed to be
used from userland. See ctladm for the primary
consumer of these functions. These include CDB
building functions.

scsi_ctl.c: CAM target peripheral driver and CTL frontend port.
This is the path into CTL for commands from
target-capable hardware/SIMs.

README.ctl.txt: CTL code features, roadmap, to-do list.

usr.sbin/Makefile: Add ctladm.

ctladm/Makefile,
ctladm/ctladm.8,
ctladm/ctladm.c,
ctladm/ctladm.h,
ctladm/util.c: ctladm(8) is the CTL management utility.
It fills a role similar to camcontrol(8).
It allow configuring LUNs, issuing commands,
injecting errors and various other control
functions.

usr.bin/Makefile: Add ctlstat.

ctlstat/Makefile
ctlstat/ctlstat.8,
ctlstat/ctlstat.c: ctlstat(8) fills a role similar to iostat(8).
It reports I/O statistics for CTL.

sys/conf/files: Add CTL files.

sys/conf/NOTES: Add device ctl.

sys/cam/scsi_all.h: To conform to more recent specs, the inquiry CDB
length field is now 2 bytes long.

Add several mode page definitions for CTL.

sys/cam/scsi_all.c: Handle the new 2 byte inquiry length.

sys/dev/ciss/ciss.c,
sys/dev/ata/atapi-cam.c,
sys/cam/scsi/scsi_targ_bh.c,
scsi_target/scsi_cmds.c,
mlxcontrol/interface.c: Update for 2 byte inquiry length field.

scsi_da.h: Add versions of the format and rigid disk pages
that are in a more reasonable format for CTL.

amd64/conf/GENERIC,
i386/conf/GENERIC,
ia64/conf/GENERIC,
sparc64/conf/GENERIC: Add device ctl.

i386/conf/PAE: The CTL frontend SIM at least does not compile
cleanly on PAE.

Sponsored by: Copan Systems, SGI and Spectra Logic
MFC after: 1 month

show more ...

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

Change "ctlstat -P"'s schema

It now reports stats separately for both ports and luns.

Also, prohibit using both "-p" and "-P" at the same time.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by:

Change "ctlstat -P"'s schema

It now reports stats separately for both ports and luns.

Also, prohibit using both "-p" and "-P" at the same time.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D38500

show more ...

ctlstat: add prometheus output

When invoked by inetd, ctlstat -P will now produce output suitable for
ingestion into Prometheus.

It's a drop-in replacement for https://github.com/Gandi/ctld_exporte

ctlstat: add prometheus output

When invoked by inetd, ctlstat -P will now produce output suitable for
ingestion into Prometheus.

It's a drop-in replacement for https://github.com/Gandi/ctld_exporter,
except that it doesn't report the number of initiators per target, and
it does report time and dma_time.

MFC after: 2 weeks
Sponsored by: Axcient
Relnotes: yes
Reviewed by: bapt, bcr
Differential Revision: https://reviews.freebsd.org/D29901

show more ...