powerof2: replace loops with fls or ilog2

In several places, a loop tests for powers of two, or iterates through
powers of two. In those places, replace the loop with an invocation
of fls or ilog2

powerof2: replace loops with fls or ilog2

In several places, a loop tests for powers of two, or iterates through
powers of two. In those places, replace the loop with an invocation
of fls or ilog2 without changing the meaning of the code.

Reviewed by: alc, markj, kib, np, erj, avg (previous version)
Differential Revision: https://reviews.freebsd.org/D45494

show more ...

mlx5en: add diagnostic in one more case of failed eeprom read preparation

Sponsored by: Nvidia Networking
MFC after: 1 week

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

mlx5en(4): Don't wait for receive queue to fill up with mbufs during open channels.

Failure to get mbufs may be transient.
Don't permanently fail to open the channels due to lack of mbufs.
This also

mlx5en(4): Don't wait for receive queue to fill up with mbufs during open channels.

Failure to get mbufs may be transient.
Don't permanently fail to open the channels due to lack of mbufs.
This also makes modifying channel parameters faster.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en(4): Explain why CQE zipping is off.

MFC after: 1 week
Sponsored by: NVIDIA Networking

net: replace IFF_KNOWSEPOCH with IFF_NEEDSEPOCH

Expect that drivers call into the network stack with the net epoch
entered. This has already been the fact since early 2020. The net
interrupts, that

net: replace IFF_KNOWSEPOCH with IFF_NEEDSEPOCH

Expect that drivers call into the network stack with the net epoch
entered. This has already been the fact since early 2020. The net
interrupts, that are marked with INTR_TYPE_NET, were entering epoch
since 511d1afb6bf. For the taskqueues there is NET_TASK_INIT() and
all drivers that were known back in 2020 we marked with it in
6c3e93cb5a4. However in e87c4940156 we took conservative approach
and preferred to opt-in rather than opt-out for the epoch.

This change not only reverts e87c4940156 but adds a safety belt to
avoid panicing with INVARIANTS if there is a missed driver. With
INVARIANTS we will run in_epoch() check, print a warning and enter
the net epoch. A driver that prints can be quickly fixed with the
IFF_NEEDSEPOCH flag, but better be augmented to properly enter the
epoch itself.

Note on TCP LRO: it is a backdoor to enter the TCP stack bypassing
some layers of net stack, ignoring either old IFF_KNOWSEPOCH or the
new IFF_NEEDSEPOCH. But the tcp_lro_flush_all() asserts the presence
of network epoch. Indeed, all NIC drivers that support LRO already
provide the epoch, either with help of INTR_TYPE_NET or just running
NET_EPOCH_ENTER() in their code.

Reviewed by: zlei, gallatin, erj
Differential Revision: https://reviews.freebsd.org/D39510

show more ...

Mechanically convert mlx5en(4) to IfAPI

Reviewed by: zlei
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38595

ifcapnv: fix IFCAP2 usage

IFCAP2_XXX constants are integers, they do not need shift for the
definition. But their usage as bitmask for if_capenable2 does require
shift. Add convenience macro IFCAP

ifcapnv: fix IFCAP2 usage

IFCAP2_XXX constants are integers, they do not need shift for the
definition. But their usage as bitmask for if_capenable2 does require
shift. Add convenience macro IFCAP2_BIT() for consumers.

Fix the only existing consumer, mlx5(4) RXTLS enable bits.

Reported by: jhb
Reviewed by: jhb, jhibbits, hselasky
Coverity CID: 1501659
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D37862

show more ...

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we loose a lot of precision. Instead lets change the math that calculates this to separate out
the seconds/nanoseconds and operate on the two values so we don't get overflow instead of just
shifting the value down and loosing precision.

Reviewed by: kib, hselasky
Sponsored by: Netflix Inc
Differential Revision: https://reviews.freebsd.org/D36327

show more ...

mlx5en(4): Allow RX TLS to be enabled and disabled by ifconfig(8).

While at it, fix double initialization of the "drv_ioctl_data_d" structure
and the "mask" variable.

Reviewed by: kib@
MFC after: 1

mlx5en(4): Allow RX TLS to be enabled and disabled by ifconfig(8).

While at it, fix double initialization of the "drv_ioctl_data_d" structure
and the "mask" variable.

Reviewed by: kib@
MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: formally declare supoort for RXTLS

Reviewed by: hselasky
Sponsored by: NVIDIA Networking
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D32551

Convert mlx5_en to SIOCSIFCAPNV

Reviewed by: hselasky, jhb, kp (previous version)
Sponsored by: NVIDIA Networking
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D32551

mlx5en(4): Use hard-coded 4K page size for RQ/SQ/CQ.

The page size specified for RQ, SQ and CQ is always in units of 4KBytes.
Make sure we subtract MLX5_ADAPTER_PAGE_SHIFT, 12, instead of PAGE_SHIFT

mlx5en(4): Use hard-coded 4K page size for RQ/SQ/CQ.

The page size specified for RQ, SQ and CQ is always in units of 4KBytes.
Make sure we subtract MLX5_ADAPTER_PAGE_SHIFT, 12, instead of PAGE_SHIFT
which may vary. This fixes support for using the mlx5en driver on systems
having non-4K page size.

Linux commit:
68cdf5d6e91068c98d6091b193dc7a5ab7dcf5eb

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en(4): Fix a few typos in source code comments

- s/persistant/persistent/

MFC after: 3 days

mlx5en: Improve RX- and TX- TLS refcounting.

Use the send tag refcounting mechanism to refcount the RX- and TX- TLS
send tags. Then it is no longer needed to wait for refcounts to reach
zero when de

mlx5en: Improve RX- and TX- TLS refcounting.

Use the send tag refcounting mechanism to refcount the RX- and TX- TLS
send tags. Then it is no longer needed to wait for refcounts to reach
zero when destroying RX- and TX- TLS send tags as a result of pending
data or WQE commands.

This also ensures that when TX-TLS and rate limiting is used at the same
time, the underlying SQ is not prematurely destroyed.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not T

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not TX traffic.

The TLS RX tag keeps track of past TLS records up to a certain limit,
approximately 1 Gbyte of TCP data. TLS records of same length are joined
into a single database record.

Regularly the HW is queried for TLS RX progress information. The TCP sequence
number gotten from the HW is then matches against the database of TLS TCP
sequence number records and lengths. If a match is found a static params WQE
is queued on the IQ and the hardware should immediately resume decrypting TLS
data until the next non-sequential TCP packet arrives.

Offloading TLS RX data is supported for untagged, prio-tagged, and
regular VLAN traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement one RQT object per channel.

These objects will eventually be used to switch TLS RX traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Create TIRs before flowtables.

Because flowtables may redirect traffic to TIRs.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Create and destroy all flow tables and rules when the network interface attaches and detaches.

Previously flow steering tables and rules were only created and destroyed
at link up and down e

mlx5en: Create and destroy all flow tables and rules when the network interface attaches and detaches.

Previously flow steering tables and rules were only created and destroyed
at link up and down events, respectivly. Due to new requirements for adding
TLS RX flow tables and rules, the main flow steering table must always be
available as there are permanent redirections from the TLS RX flow table
to the vlan flow table.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Force all packets through the indirection table.

All packets must go through the indirection table, RQT,
because it is not possible to modify the RQN of the TIR
for direct dispatchment after

mlx5en: Force all packets through the indirection table.

All packets must go through the indirection table, RQT,
because it is not possible to modify the RQN of the TIR
for direct dispatchment after it is created, typically
when the link goes up and down.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry res

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry resync
information for ongoing TLS RX connections and when changing schedule queues
for rate limited connections.

The internal queue, IQ, code is more or less a stripped down copy
of the existing SQ managing code with exception of:

1) An optional single segment memory buffer which can be read or
written as a whole by the hardware, may be provided.
2) An optional completion callback for all transmit operations, may
be provided.
3) Does not support mbufs.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Make the receive packet indirection table, RQT, static instead of dynamic.

Allocate the RQT once, pointing all initial entries to the drop RQN.
When opening the channels simplify modify the

mlx5en: Make the receive packet indirection table, RQT, static instead of dynamic.

Allocate the RQT once, pointing all initial entries to the drop RQN.
When opening the channels simplify modify the RQT, directing all traffic
to the new RQNs. Similarly when closing the channels point all RQT entries
back to the so-called drop RQN.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Set CQN in RQ parameters for drop RQ.

Else creating the drop RQ fails.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Set channel pointer for drop receive queue.

A valid channel pointer is needed to get the priv pointer during init.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Print error code when opening drop RQ fails.

MFC after: 1 week
Sponsored by: NVIDIA Networking