net - Simplify net/altq TSC timer code, get rid of its TSC calibration code.

* There is no use for the custom TSC calibration code in the ALTQ code.
If our bootup TSC calibration would ever fail,

net - Simplify net/altq TSC timer code, get rid of its TSC calibration code.

* There is no use for the custom TSC calibration code in the ALTQ code.
If our bootup TSC calibration would ever fail, we have bigger problems
than being a little bit slower in the ALTQ code.

show more ...

kernel - Refactor smp collision statistics

* Add an indefinite wait timing API (sys/indefinite.h,
sys/indefinite2.h). This interface uses the TSC and will
record lock latencies to our pcpu stat

kernel - Refactor smp collision statistics

* Add an indefinite wait timing API (sys/indefinite.h,
sys/indefinite2.h). This interface uses the TSC and will
record lock latencies to our pcpu stats in microseconds.
The systat -pv 1 display shows this under smpcoll.

Note that latencies generated by tokens, lockmgr, and mutex
locks do not necessarily reflect actual lost cpu time as the
kernel will schedule other threads while those are blocked,
if other threads are available.

* Formalize TSC operations more, supply a type (tsc_uclock_t and
tsc_sclock_t).

* Reinstrument lockmgr, mutex, token, and spinlocks to use the new
indefinite timing interface.

show more ...

netisr: Simplify assertion related bits

altq: Use microuptime() instead of microtime() - boottime.

i386 removal, part 34/x: Remove some i386 traces from the kernel.

net: Factor out ASSERT_IN_NETISR and ASSERT_IS_NETISR

ifnet: Make ifnet and ifindex2ifnet MPSAFE

- Accessing to these two global variables from non-netisr threads uses
ifnet lock. This kind of accessing is from
- Accessing to ifindex2ifnet from neti

ifnet: Make ifnet and ifindex2ifnet MPSAFE

- Accessing to these two global variables from non-netisr threads uses
ifnet lock. This kind of accessing is from
- Accessing to ifindex2ifnet from netisrs are lockless MPSAFE.
- Netisrs no longer access ifnet, instead they access ifnet array as of
this commit, which is lockless MPSAFE.

Rules for accessing ifnet and ifindex2ifnet is commented near the
declaration of the related global variables/functions in net/if_var.h.

show more ...

altq: Dispatch tbr_timeout to netisr0 to run

This function now access ifnet global variable in netisr.

altq: Use tsc_mpsync to detect whether TSC could be used or not

net: Use tsc_invariant when it is necessary; mainly in time measure

altq: Clean up the code for PCC usage detection

ifsubque: Cut ties with ifqueue

altq: Remove the unused parameter 'mpolled' from dequeue method

ifq: Remove the unused parameter 'mpolled' from ifq dequeue interface

The ifq_poll() -> ifq_dequeue() model is not MPSAFE, and mpolled has
not been used, i.e. set to NULL, for years; time to let it

ifq: Remove the unused parameter 'mpolled' from ifq dequeue interface

The ifq_poll() -> ifq_dequeue() model is not MPSAFE, and mpolled has
not been used, i.e. set to NULL, for years; time to let it go.

show more ...

ifsq: Let ifaltq_subque know its related hardware TX queue's serializer

This avoids following operations on packet transmission hot path:
- Dereferening device driver supplied serialize function poi

ifsq: Let ifaltq_subque know its related hardware TX queue's serializer

This avoids following operations on packet transmission hot path:
- Dereferening device driver supplied serialize function pointers
- Locating hardware TX queue's serializer

Comparing to the lwkt_serialize functions, the above two operations are
costful.

Driver changes:
- For device drivers which use the default ifnet serializer, no additional
code will be needed, if_attach() will assign ifnet serializer to
ifaltq_subque.
- For device drivers which use independent serializers for main function,
RX queues and TX queues, ifsq_set_hw_serialize() must be called to
properly assign the hardware TX queue's serializer to ifaltq_subque.
Drivers in this category are bce(4), emx(4), igb(4) and jme(4).

show more ...

if: Multiple TX queue support step 3 of 3; map CPUID to subqueue

Add CPUID to subqueue mapping method to ifaltq. Driver could provide
its own CPUID to subqueue mapping method through ifnet.if_mapsu

if: Multiple TX queue support step 3 of 3; map CPUID to subqueue

Add CPUID to subqueue mapping method to ifaltq. Driver could provide
its own CPUID to subqueue mapping method through ifnet.if_mapsubq,
which is used when ALTQ's packet scheduler is not enabled. ALTQ's
packet schedulers always map CPUID to the default subqueue.

show more ...

if: Multiple TX queue support step 2 of many

ifnet_{serialize,deserialize,tryserialize}_tx and hardware TX serializer
asserion macros now require subqueue, so the proper hardware TX queue's
serializ

if: Multiple TX queue support step 2 of many

ifnet_{serialize,deserialize,tryserialize}_tx and hardware TX serializer
asserion macros now require subqueue, so the proper hardware TX queue's
serializer could be held, released and asserted.

show more ...

if: Multiple TX queue support step 1 of many; introduce ifaltq subqueue

Put the plain queue information, e.g. queue header and tail, serializer,
packet staging scoreboard and ifnet.if_start schedule

if: Multiple TX queue support step 1 of many; introduce ifaltq subqueue

Put the plain queue information, e.g. queue header and tail, serializer,
packet staging scoreboard and ifnet.if_start schedule netmsg etc. into
its own structure (subqueue). ifaltq structure could have multiple of
subqueues based on the count that drivers can specify.

Subqueue's enqueue, dequeue, purging and states updating are protected
by the subqueue's serializer, so for hardwares supporting multiple TX
queues, contention on queuing operation could be greatly reduced.

The subqueue is passed to if_start to let the driver know which hardware
TX queue to work on. Only the related driver's TX queue serializer will
be held, so for hardwares supporting multiple TX queues, contention on
driver's TX queue serializer could be greatly reduced.

Bunch of ifsq_ prefixed functions are added, which is used to perform
various operations on subqueues. Commonly used ifq_ prefixed functions
are still kept mainly for the drivers which do not support multiple TX
queues (well, these functions also ease the netif/ convertion in this
step :).

All of the pseudo network devices under sys/net are converted to use the
new subqueue operation. netproto/802_11 is converted too. igb(4) is
converted to use the new subqueue operation, the rest of the network
drivers are only changed for the if_start interface modification.

For ALTQs which have packet scheduler enabled, only the first subqueue
is used (*).

(*) Whether we should utilize multiple TX queues if ALTQ's packet scheduler
is enabled is quite questionable. Mainly because hardware's multiple TX
queue packet dequeue mechanism could have negative impact on ALTQ's packet
scheduler's decision.

show more ...

ifq: Add ifq_purge_all(); currently it's same as ifq_purge()

This paves the way for the multiple transmission queues support.

ifq_purge_all() is used when ALTQ packet scheduler is changed,
interfac

ifq: Add ifq_purge_all(); currently it's same as ifq_purge()

This paves the way for the multiple transmission queues support.

ifq_purge_all() is used when ALTQ packet scheduler is changed,
interface routing is changed and pseudo devices are closed.

show more ...

kernel: Make SMP support default (and non-optional).

The 'SMP' kernel option gets removed with this commit, so it has to
be removed from everybody's configs.

Reviewed-by: sjg
Approved-by: many

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

Split ifnet serialize step 1/many: Add if_{serialize,deserialize,tryserialize}()
function pointers to ifnet.

These three function pointers accept ifnet struct and ifnet_serialize
enumeration.

The i

Split ifnet serialize step 1/many: Add if_{serialize,deserialize,tryserialize}()
function pointers to ifnet.

These three function pointers accept ifnet struct and ifnet_serialize
enumeration.

The ifnet_serialize enumeration indicates the serialization type:
IFNET_SERIALIZE_ALL:
All of the serializers should be held. Except for if_start and if_input,
this enumeration must be used when call ifnet function pointers.
IFNET_SERIALIZE_TX:
Only transmit serializer should be held. This enumeration could be used
when calling ifnet.if_start.
IFNET_SERIALIZE_RX:
Only receive serializer should be held. This enumeration could be used
when calling ifnet.if_input.

If the NIC driver does not set these three function pointer, then if_attach()
will set them to the default ones: only one serializer (if_serializer) is used
and ifnet_serialize parameter is ignored.

Following several inline functions are added which are sheer wrappers of the
three ifnet serialize function pointers:
ifnet_serialize_{all,tx,rx}()
ifnet_deserialize_{all,tx,rx}()
ifnet_tryserialize_{all,tx,rx}()

All of the protocol layers and most of the pseudo drivers are converted.

Discussed-with: dillon@

show more ...

Reduce ifnet.if_serializer contention on output path:
- Push ifnet.if_serializer holding down into each ifnet.if_output implementation
- Add a serializer into ifaltq, which is used to protect send qu

Reduce ifnet.if_serializer contention on output path:
- Push ifnet.if_serializer holding down into each ifnet.if_output implementation
- Add a serializer into ifaltq, which is used to protect send queue instead of
its parent's if_serializer. This change has following implication:
o On output path, enqueueing packets and calling ifnet.if_start are decoupled
o In device drivers, poll->dev_encap_ok->dequeue operation sequence is no
longer safe, instead dequeue->dev_encap_fail->prepend should be used
This serializer will be held by using lwkt_serialize_adaptive_enter()
- Add altq_started field into ifaltq, which is used to interlock the calling
of its parent's if_start, to reduce ifnet.if_serializer contention.
if_devstart(), a helper function which utilizes ifaltq.altq_started, is added
to reduce code duplication in ethernet device drivers.
- Add if_cpuid into ifnet. This field indicates on which CPU device driver's
interrupt will happen.
- Add ifq_dispatch(). This function will try to hold ifnet.if_serializer in
order to call ifnet.if_start. If this attempt fails, this function will
schedule ifnet.if_start to be called on CPU located by ifnet.if_start_cpuid
if_start_nmsg, which is per-CPU netmsg, is added to ifnet to facilitate
ifnet.if_start scheduling. ifq_dispatch() is called by ether_output_frame()
currently
- Use ifq_classic_ functions, if altq is not enabled
- Fix various device drivers bugs in their if_start implementation
- Add ktr for ifq classic enqueue and dequeue
- Add ktr for ifnet.if_start

show more ...

Finish moving the kernel from tsc_freq (32 bits) to tsc_frequency (64 bits).

Add sysctls to access hw.tsc_present and hw.tsc_frequency

Implement TSC support in the vkernel.

Add fairq to altq. Fairq is a fair queueing algorithm with bandwidth
prioritization and a bandwidth delimiter (hogs) to allow low bandwidth
buckets to jump the round robin. This fairq algorithm is

Add fairq to altq. Fairq is a fair queueing algorithm with bandwidth
prioritization and a bandwidth delimiter (hogs) to allow low bandwidth
buckets to jump the round robin. This fairq algorithm is currently unweighted
but traffic can still be classified with the global priority model. For
each queue traffic is normally round robined by taking a packet from each
bucket in turn.

This feature is primarily intended for edge routers and egress points with
bandwidth constrictions.

* Hogs feature allows low bandwidth buckets to burst. Low bandwidth can
mean, e.g. an interactive shell or even simply ack traffic, without
the need to explicitly classify it. Bandwidth is managed on a per-bucket
basis.

* Prioritization feature allows minimum guaranteed bandwidths based on
service classifications. e.g. VOIP, web, mail, PtP, etc.

* Weighted fairq not implemented (beyond using classification into priority
queues), but the circular bucket design makes it a fairly easy task if
someone wants to do it.

* Add ALTQ_MBUF_STATE_HASHED and generate a hash of the connection state
in the mbuf header for any packet that you have set 'keep state' for
in pf. This is done in PF and is needed by fairq to bucketize
'connections'.

* Add the fairq implementation and a new ALTQ_FAIRQ kernel build option.

* Simple example included below.

ports="{ 25, 80 }"
altq on vke0 fairq bandwidth 500Kb queue { normal, bulk }
queue bulk priority 1 bandwidth 100Kb \
fairq(buckets 64, hogs 25Kb) qlimit 50
queue normal priority 2 bandwidth 400Kb \
fairq(buckets 64, hogs 25Kb, default) qlimit 50

pass out on vke0 inet proto tcp from any to any \
keep state queue normal
pass out on vke0 inet proto tcp from any to any port $ports \
keep state queue bulk

show more ...