Have ifmp_ring_enqueue() abdicate instead of switch to a consumer

Move TX out of the enqueue() path. As a result, we need
to have ifmp_ring_check_drainage() pick up from the abdicate state.

We also

Have ifmp_ring_enqueue() abdicate instead of switch to a consumer

Move TX out of the enqueue() path. As a result, we need
to have ifmp_ring_check_drainage() pick up from the abdicate state.

We also need to either enqueue the TX task, or check drainage
after calling ifmp_ring_enqueue() to ensure it's sent.

This change results in a 30% small packet forwarding improvement.

Reviewed by: olivier, sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12439

show more ...

Make the rx budget a tunable

This allows tuning the rx budget for special load profiles
as well as more easily testing to determine sane defaults.

Reviewed by: sbruno
Approved by: sbruno (mentor)
S

Make the rx budget a tunable

This allows tuning the rx budget for special load profiles
as well as more easily testing to determine sane defaults.

Reviewed by: sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12445

show more ...

Chain mbufs before passing to if_input()

Build a list of mbufs to pass to if_input() after LRO. Results in
12% small packet forwarding rate improvement.

Reviewed by: sbruno
Approved by: sbruno (men

Chain mbufs before passing to if_input()

Build a list of mbufs to pass to if_input() after LRO. Results in
12% small packet forwarding rate improvement.

Reviewed by: sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12444

show more ...

Some small packet performance improvements

If the packet is smaller than MTU, disable the TSO flags.
Move TCP header parsing inside the IS_TSO?() test.
Add a new IFLIB_NEED_ZERO_CSUM flag to indicat

Some small packet performance improvements

If the packet is smaller than MTU, disable the TSO flags.
Move TCP header parsing inside the IS_TSO?() test.
Add a new IFLIB_NEED_ZERO_CSUM flag to indicate the checksums need to be zeroed before TX.

Reviewed by: sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12442

show more ...

Fix iflib netmap RX

RXQ setup for netmap was broken because netmap_rxq_init was getting called
before IFDI_INIT - thus we ended up with ring tail pointer being reset to zero.

Reviewed by: sbruno
Ap

Fix iflib netmap RX

RXQ setup for netmap was broken because netmap_rxq_init was getting called
before IFDI_INIT - thus we ended up with ring tail pointer being reset to zero.

Reviewed by: sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12140

show more ...

Revert r323516 (iflib rollup)

This was really too big of a commit even if everything worked, but there
are multiple new issues introduced in the one huge commit, so it's not
worth keeping this until

Revert r323516 (iflib rollup)

This was really too big of a commit even if everything worked, but there
are multiple new issues introduced in the one huge commit, so it's not
worth keeping this until it's fixed.

I'll work on splitting this up into logical chunks and introduce them one
at a time over the next week or two.

Approved by: sbruno (mentor)
Sponsored by: Limelight Networks

show more ...

Roll up iflib commits from github. This pulls in most of the work done
by Matt Macy as well as other changes which he has accepted via pull
request to his github repo at https://github.com/mattmacy/

Roll up iflib commits from github. This pulls in most of the work done
by Matt Macy as well as other changes which he has accepted via pull
request to his github repo at https://github.com/mattmacy/networking/

This should bring -CURRENT and the github repo into close enough sync to
allow small feature branches rather than a large chain of interdependant
patches being developed out of tree. The reset of the synchronization
should be able to be completed on github by splitting the remaining
changes that are not yet ready into short feature branches for later
review as smaller commits.

Here is a summary of changes included in this patch:

1) More checks when INVARIANTS are enabled for eariler problem
detection
2) Group Task Queue cleanups
- Fix use of duplicate shortdesc for gtaskqueue malloc type.
Some interfaces such as memguard(9) use the short description to
identify malloc types, so duplicates should be avoided.
3) Allow gtaskqueues to use ithreads in addition to taskqueues
- In some cases, this can improve performance
4) Better logging when taskqgroup_attach*() fails to set interrupt
affinity.
5) Do not start gtaskqueues until they're needed
6) Have mp_ring enqueue function enter the ABDICATED rather than BUSY
state. This moves the TX to the gtaskq and allows processing to
continue faster as well as make TX batching more likely.
7) Add an ift_txd_errata function to struct if_txrx. This allows
drivers to inspect/modify mbufs before transmission.
8) Add a new IFLIB_NEED_ZERO_CSUM for drivers to indicate they need
checksums zeroed for checksum offload to work. This avoids modifying
packet data in the TX path when possible.
9) Use ithreads for iflib I/O instead of taskqueues
10) Clean up ioctl and support async ioctl functions
11) Prefetch two cachlines from each mbuf instead of one up to 128B. We
often need to parse packet header info beyond 64B.
12) Fix potential memory corruption due to fence post error in
bit_nclear() usage.
13) Improved hang detection and handling
14) If the packet is smaller than MTU, disable the TSO flags.
This avoids extra packet parsing when not needed.
15) Move TCP header parsing inside the IS_TSO?() test.
This avoids extra packet parsing when not needed.
16) Pass chains of mbufs that are not consumed by lro to if_input()
rather call if_input() for each mbuf.
17) Re-arrange packet header loads to get as much work as possible done
before a cache stall.
18) Lock the context when calling IFDI_ATTACH_PRE()/IFDI_ATTACH_POST()/
IFDI_DETACH();
19) Attempt to distribute RX/TX tasks across cores more sensibly,
especially when RX and TX share an interrupt. RX will attempt to
take the first threads on a core, and TX will attempt to take
successive threads.
20) Allow iflib_softirq_alloc_generic() to request affinity to the same
cpus an interrupt has affinity with. This allows TX queues to
ensure they are serviced by the socket the device is on.
21) Add new iflib sysctls to net.iflib:
- timer_int - interval at which to run per-queue timers in ticks
- force_busdma
22) Add new per-device iflib sysctls to dev.X.Y.iflib
- rx_budget allows tuning the batch size on the RX path
- watchdog_events Count of watchdog events seen since load
23) Fix error where netmap_rxq_init() could get called before
IFDI_INIT()
24) e1000: Fixed version of r323008: post-cold sleep instead of DELAY
when waiting for firmware
- After interrupts are enabled, convert all waits to sleeps
- Eliminates e1000 software/firmware synchronization busy waits after
startup
25) e1000: Remove special case for budget=1 in em_txrx.c
- Premature optimization which may actually be incorrect with
multi-segment packets
26) e1000: Split out TX interrupt rather than share an interrupt for
RX and TX.
- Allows better performance by keeping RX and TX paths separate
27) e1000: Separate igb from em code where suitable
Much easier to understand separate functions and "if (is_igb)" than
previous tests like "if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC))"

#blamebruno

Reviewed by: sbruno
Approved by: sbruno (mentor)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12235

show more ...

Do not abuse flag that is clearly marked as unused.
This creates conflicts with FreeBSD variations that may use it. The
usage of the flag M_TOOBIG is limited to iflib queue, thus using
one of M_PROT

Do not abuse flag that is clearly marked as unused.
This creates conflicts with FreeBSD variations that may use it. The
usage of the flag M_TOOBIG is limited to iflib queue, thus using
one of M_PROTO flags is fine. There is no need to grab global flag.

Silence from: kmacy, sbruno (2 weeks)

show more ...

Revert r323008 and its conversion of e1000/iflib to using SX locks.

This seems to be missing something on the 82574L causing NFS root mounts
to hang.

Reported by: kib

Continuation of lock cleanup in e1000.

Post-cold sleep instead of DELAY when waiting for firmware.

Convert softc mutex to an SX lock. Change all waits to sleeps
once interrupts are enabled (and it

Continuation of lock cleanup in e1000.

Post-cold sleep instead of DELAY when waiting for firmware.

Convert softc mutex to an SX lock. Change all waits to sleeps
once interrupts are enabled (and it is safe to sleep).

Submitted by: Matt Macy <matt@mattmacy.io>
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12101

show more ...

iflib: call device's if_init function during vlan initialization.

Submitted by: bhargava.marreddy@broadcom.com
Reviewed by: shurd
Sponsored by: Broadcom
Differential Revision: https://reviews.freeb

iflib: call device's if_init function during vlan initialization.

Submitted by: bhargava.marreddy@broadcom.com
Reviewed by: shurd
Sponsored by: Broadcom
Differential Revision: https://reviews.freebsd.org/D12098

show more ...

Don't leak mbufs if clusers exceeds the number of segments. This would
leak mbufs over time causing crashes.

PR: 221202
Submitted by: Matt Macy <matt@mattmacy.io>
Reported by: gergely.czuczy@harml

Don't leak mbufs if clusers exceeds the number of segments. This would
leak mbufs over time causing crashes.

PR: 221202
Submitted by: Matt Macy <matt@mattmacy.io>
Reported by: gergely.czuczy@harmless.hu
Sponsored by: Limelight Networks

show more ...

Export IFCAP_HWSTATS so that we don't experience double stats counting
on iflib enabled devices.

PR: 220198
Submitted by: Matt Macy <matt@mattmacy.io>
Reported by: Ben Woods <woodsb02@freebsd.org>

Export IFCAP_HWSTATS so that we don't experience double stats counting
on iflib enabled devices.

PR: 220198
Submitted by: Matt Macy <matt@mattmacy.io>
Reported by: Ben Woods <woodsb02@freebsd.org>
Sponsored by: Limelight Networks

show more ...

Slight restructure of iflib_busdma_load_mbuf_sg() to fix accounting
when m_collapse() fails.

Submitted by: krzystof.galazka@intel.com
Reviewed by: Jeb Cramer <cramerj@intel.com>
Sponsored by: Intel

Slight restructure of iflib_busdma_load_mbuf_sg() to fix accounting
when m_collapse() fails.

Submitted by: krzystof.galazka@intel.com
Reviewed by: Jeb Cramer <cramerj@intel.com>
Sponsored by: Intel Corporation
Differential Revision: https://reviews.freebsd.org/D11476

show more ...

Fix printf format warning in iflib.c

Clang 5.0.0 got better warnings about printf format strings using %zd,
and this leads to the following -Werror warning on e.g. arm:

sys/net/iflib.c:1517:8:

Fix printf format warning in iflib.c

Clang 5.0.0 got better warnings about printf format strings using %zd,
and this leads to the following -Werror warning on e.g. arm:

sys/net/iflib.c:1517:8: error: format specifies type 'ssize_t' (aka 'int') but the argument has type 'bus_size_t' (aka 'unsigned long') [-Werror,-Wformat]
sctx->isc_tx_maxsize, nsegments, sctx->isc_tx_maxsegsize);
^~~~~~~~~~~~~~~~~~~~
sys/net/iflib.c:1517:41: error: format specifies type 'ssize_t' (aka 'int') but the argument has type 'bus_size_t' (aka 'unsigned long') [-Werror,-Wformat]
sctx->isc_tx_maxsize, nsegments, sctx->isc_tx_maxsegsize);
^~~~~~~~~~~~~~~~~~~~~~~

Fix this by casting bus_size_t arguments to uintmax_t, and using %ju
instead.

Reviewed by: emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D11679

show more ...

Don't cache mbuf pointers if the number of descriptors is greater than
the number of buffers.

Submitted by: Matt Macy <mmacy@mattmacy.io>
Sponsored by: Limelight Networks

iflib - flib_busdma_load_mbuf_sg used isc_tx_maxsize as max semgent size.

Submitted by: krzysztof.galazka@intel.com
Differential Revision: https://reviews.freebsd.org/D11403

bnxt(4) Enable LRO support, redux

iflib - reset fl-ifl_fragidx to 0 on iflib_fl_bufs_free(). This caused the
panic in em/igb when adding it to a bridge device.

iflib - Handle out of order packet d

bnxt(4) Enable LRO support, redux

iflib - reset fl-ifl_fragidx to 0 on iflib_fl_bufs_free(). This caused the
panic in em/igb when adding it to a bridge device.

iflib - Handle out of order packet delivery from hardware in support of LRO

Out of order updates to rxd's is fixed in r315217. However, it is not
completely fixed. While refilling the buffers, iflib is not considering
the out of order descriptors. Hence, it is refilling sequentially.
"idx" variable in _iflib_fl_refill routine is incremented sequentially.
By doing refilling sequentially, it will override the SGEs that
are *IN USE* by other connections. Fix is to maintain a bitmap of
rx descriptors and differentiate the used one with unused one and
refill only at the unused indices. This patch also fixes a
few bugs in bnxt, related to the same feature.

Submitted by: bhargava.marreddy@broadcom.com
Reviewed by: venkatkumar.duvvuru@broadcom.com shurd
Differential Revision: https://reviews.freebsd.org/D10681

show more ...

Revert r319989 "bnxt(4) Enable LRO support"

This generates startup LORs and panics when adding elements to bridge
devices. I will document further in https://reviews.freebsd.org/D10681

PR: 220073
S

Revert r319989 "bnxt(4) Enable LRO support"

This generates startup LORs and panics when adding elements to bridge
devices. I will document further in https://reviews.freebsd.org/D10681

PR: 220073
Submitted by: dchagin
Reported by: db

show more ...

bnxt(4) Enable LRO support

iflib - Handle out of order packet delivery from hardware in support of LRO

Out of order updates to rxd's is fixed in r315217. However, it is not
completely fixed. While

bnxt(4) Enable LRO support

iflib - Handle out of order packet delivery from hardware in support of LRO

Out of order updates to rxd's is fixed in r315217. However, it is not
completely fixed. While refilling the buffers, iflib is not considering
the out of order descriptors. Hence, it is refilling sequentially.
"idx" variable in _iflib_fl_refill routine is incremented sequentially.
By doing refilling sequentially, it will override the SGEs that
are *IN USE* by other connections. Fix is to maintain a bitmap of
rx descriptors and differentiate the used one with unused one and
refill only at the unused indices. This patch also fixes a
few bugs in bnxt, related to the same feature.

Submitted by: bhargava.marreddy@broadcom.com
Reviewed by: shurd@
Differential Revision: https://reviews.freebsd.org/D10681

show more ...

Revert r319921 which seems to cause NFS booting assertion panics in
various configurations.

Reported by: pho@

Add new sysctl to allow changing of timing of the txq timers.

Add new sysctl to override use of busdma in the driver.

Submitted by: Drew Gallitin <gallatin@netflix.com>

Plug mbuf leak in the busdma path of iflib.

Submitted by: Michael Tuexen <tuexen@freebsd.org>
Reported by: Drew Gallitin <gallatin@netflix.com>

Adjust a comment.

MFC after: 3 days

Move pause frame counter out of struct if_ctx and into struct if_softc_ctx_t
so that we can use it in iflib to detect pause frames.

The igb(4) driver definitely used to use this in its old timer fun

Move pause frame counter out of struct if_ctx and into struct if_softc_ctx_t
so that we can use it in iflib to detect pause frames.

The igb(4) driver definitely used to use this in its old timer function and
I see no reason to restrict it to that driver only.

Sponsored by: Limelight Networks

show more ...