<net/if.h>: Do not include <net/if_var.h> for _KERNEL

* Clean up an ancient leftover: do not include <net/if_var.h> from <net/if.h>
for kernel stuffs.

* Adjust various files to include the necess

<net/if.h>: Do not include <net/if_var.h> for _KERNEL

* Clean up an ancient leftover: do not include <net/if_var.h> from <net/if.h>
for kernel stuffs.

* Adjust various files to include the necessary <net/if_var.h> header.

NOTE:
I have also tested removing the inclusion of <net/if.h> from <net/if_var.h>,
therefore add <net/if.h> inclusion for those files that need it but only
included <net/if_var.h>. For some files, the header inclusion orderings are
also adjusted.

show more ...

polling: Utilize netisr_domsg_global

polling: No need to explicitly align io context and systimer context

polling: Adjust comment

polling: Don't do direct input in critical section.

polling: Implement direct input support.

When "direct input" is enabled by driver, driver's RX polling handler
will run ethernet/ip/tcp processing directly, which avoids cache-miss
on mbufs themselv

polling: Implement direct input support.

When "direct input" is enabled by driver, driver's RX polling handler
will run ethernet/ip/tcp processing directly, which avoids cache-miss
on mbufs themselves. Currently it is enabled on ix(4) by default.

The normal IP forwarding performance is improved by %12, while the fast
IP forwarding performance is improved by 10%. 13.2Mpps is achieved for
dual side IP forwarding!

1 request/connection HTTP/1.1 performance and avg-latency stay same,
but the latency is further stablized:
1K 5.20ms -> 4.60ms
8K 6.43ms -> 5.76ms
16K 16.30ms -> 14.90ms

show more ...

netisr: Simplify assertion related bits

polling: Simplify the code by using netsr_*msg functions.

ifpoll: Use netisr_ncpus.

ifpoll: Use SYSINIT to initialize per-cpu ifpoll context.

net: Factor out ASSERT_IN_NETISR and ASSERT_IS_NETISR

polling: Utilize microtime_pcpu

msgport: Add putport_oncpu; helps scheduling netisr locally for spin port

Background:
High rate (actually same rate as polling(4)) IPIs on random CPUs are
observed when polling(4) is enabled and the

msgport: Add putport_oncpu; helps scheduling netisr locally for spin port

Background:
High rate (actually same rate as polling(4)) IPIs on random CPUs are
observed when polling(4) is enabled and there is virtually no network
activity.

After polling(4) activities are traced using ktr(9), it turns out that the
high rate IPIs are actually from the wakeup() on netisr's msgport. Since
the sleep queue cpumask is indexed by the hash of ident, there are chances
that the netisr's msgport ident has the same hash value as other idents
that certain threads on other CPUs are waiting on. If this ever happens
(well, it does happen), the netisr's msgport wakeup will trigger "wakeup"
IPIs to other CPUs. However, these "wakeup" IPIs are actually useless,
since only netisr will wait on its msgport.

putport_oncpu() msgport method is added to call wakeup_mycpu() for spin
msgport, if we know that this port is only accessed by one thread on the
current CPU, e.g. polling(4). This is also the case for other network
code, e.g. syncache timeout, TCP timeout, fastforward flow cache timeout
etc. However, these network code's running rate is too low to unveil the
extra "wakeup" IPIs problem. lwkt_sendmsg_oncpu() is added as wrapper to
putport_oncpu() msgport method.

Currently, only polling(4) is using lwkt_sendmsg_oncpu(). Others will
be converted soon.

show more ...

polling: Add preliminary KTR support

ifpoll: Use tsc_invariant consistently

net: Use tsc_invariant when it is necessary; mainly in time measure

polling: Fix comment

netisr: Inline netisr_cpuport() and netisr_curport()

These two functions do nothing more than just return pointer to the
element in the array.

Per our header file naming convention, put these two f

netisr: Inline netisr_cpuport() and netisr_curport()

These two functions do nothing more than just return pointer to the
element in the array.

Per our header file naming convention, put these two functions in
net/netisr2.h

show more ...

netisr: Function renaming; no functional changes

This cleans up code for keeping input packets' hash instead of masking
the hash with ncpus2_mask. netisr_hashport(), which maps packet hash
to netis

netisr: Function renaming; no functional changes

This cleans up code for keeping input packets' hash instead of masking
the hash with ncpus2_mask. netisr_hashport(), which maps packet hash
to netisr port, will be added soon.

show more ...

if: Move if_cpuid into ifaltq; prepare multiple TX queues support

if_cpuid and if_npoll_cpuid are merged and moved into ifaltq as
altq_cpuid, which indicates the owner CPU of the tx queue. Since
we

if: Move if_cpuid into ifaltq; prepare multiple TX queues support

if_cpuid and if_npoll_cpuid are merged and moved into ifaltq as
altq_cpuid, which indicates the owner CPU of the tx queue. Since
we already have code in if_start_dispatch() to catching tx queue
owner CPU changes, this merging is quite safe.

show more ...

polling: Perform RX polling before TX polling

polling: Increase default polling rate to 6000Hz

This increases the bidirational normal IP forwarding rate by 30~40Kpps

polling: Increase default rx.each_burst to 50

With this default the CPU usage still could be throttled to the desired
value (rx.user_frac), it gives reasonable burst for modern systems and
number of

polling: Increase default rx.each_burst to 50

With this default the CPU usage still could be throttled to the desired
value (rx.user_frac), it gives reasonable burst for modern systems and
number of empty RX polling is reduced.

show more ...

polling: Diverge each CPU's polling frequency a little bit (within 50Hz)

This avoids possible thunder hurd effect on ifnet.if_snd's serializer.

polling: Add tunable for net.ifpoll.X.rx.user_frac