<sys/serialize.h>: Limit prototypes to kernel only.

The "struct lwkt_serialize" must be left userland visible for if_net.

kernel/lwkt_serialize: Take serializer bookkeeping out of INVARIANTS.

IS_SERIALIZED() is no longer only used in [K]KASSERT(), which disrupts
compiling without INVARIANTS rather regularly.

This comm

kernel/lwkt_serialize: Take serializer bookkeeping out of INVARIANTS.

IS_SERIALIZED() is no longer only used in [K]KASSERT(), which disrupts
compiling without INVARIANTS rather regularly.

This commit makes the needed bits available without INVARIANTS too.

show more ...

kernel: Make SMP support default (and non-optional).

The 'SMP' kernel option gets removed with this commit, so it has to
be removed from everybody's configs.

Reviewed-by: sjg
Approved-by: many

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

serializer: Revoke PROFILE_SERIALIZER kernel option

This kernel is added by me to do preliminary serializer contention
profiling. It is kinda invasive and expands struct lwkt_serialize
considerably

serializer: Revoke PROFILE_SERIALIZER kernel option

This kernel is added by me to do preliminary serializer contention
profiling. It is kinda invasive and expands struct lwkt_serialize
considerably. Need to find a better way...

show more ...

Put lwkt_serialize_array_{enter,try,exit}() into its own header file;
name it sys/serialize2.h according to our convention.

Factor out lwkt_serialize_array_{enter,try,exit}()

Unbreak kernel/module building without INVARIANTS

Put last_td accessing back into INVARIANTS

Reduce ifnet.if_serializer contention on output path:
- Push ifnet.if_serializer holding down into each ifnet.if_output implementation
- Add a serializer into ifaltq, which is used to protect send qu

Reduce ifnet.if_serializer contention on output path:
- Push ifnet.if_serializer holding down into each ifnet.if_output implementation
- Add a serializer into ifaltq, which is used to protect send queue instead of
its parent's if_serializer. This change has following implication:
o On output path, enqueueing packets and calling ifnet.if_start are decoupled
o In device drivers, poll->dev_encap_ok->dequeue operation sequence is no
longer safe, instead dequeue->dev_encap_fail->prepend should be used
This serializer will be held by using lwkt_serialize_adaptive_enter()
- Add altq_started field into ifaltq, which is used to interlock the calling
of its parent's if_start, to reduce ifnet.if_serializer contention.
if_devstart(), a helper function which utilizes ifaltq.altq_started, is added
to reduce code duplication in ethernet device drivers.
- Add if_cpuid into ifnet. This field indicates on which CPU device driver's
interrupt will happen.
- Add ifq_dispatch(). This function will try to hold ifnet.if_serializer in
order to call ifnet.if_start. If this attempt fails, this function will
schedule ifnet.if_start to be called on CPU located by ifnet.if_start_cpuid
if_start_nmsg, which is per-CPU netmsg, is added to ifnet to facilitate
ifnet.if_start scheduling. ifq_dispatch() is called by ether_output_frame()
currently
- Use ifq_classic_ functions, if altq is not enabled
- Fix various device drivers bugs in their if_start implementation
- Add ktr for ifq classic enqueue and dequeue
- Add ktr for ifnet.if_start

show more ...

- Add lwkt_serialize_adaptive_enter(9), it is same as lwkt_serialize_enter(9)
except that it spins a little bit before sleeping.
- Under debug sysctl tree, add sysctl nodes to tune various backoff

- Add lwkt_serialize_adaptive_enter(9), it is same as lwkt_serialize_enter(9)
except that it spins a little bit before sleeping.
- Under debug sysctl tree, add sysctl nodes to tune various backoff related
parameter for lwkt_serialize_adaptive_enter(9).
- Add ktr for serializer enter end, exit begin, spin backoff and spin backoff
failure.

Reviewed-by: corecode@

show more ...

Add counters for serializer enter/try

Suggested-by: aggelos@

- Add two fields in lwkt_serialize to profile serializer contention.
- Expose serializer contention data through em's private sysctl tree.

Add ASSERT_NOT_SERIALIZED

Move atomic_intr_t to machine/stdint.h and predent __ to reduce
namespace pollution. Include that file in sys/serialize.h instead
of machine/atomic.h to dramatically reduce namespace pollution of
use

Move atomic_intr_t to machine/stdint.h and predent __ to reduce
namespace pollution. Include that file in sys/serialize.h instead
of machine/atomic.h to dramatically reduce namespace pollution of
userland applications.

show more ...

Add a lwkt_serialize_try() API function.

Major cleanup of the interrupt registration subsystem.

* Collapse the separate registrations in the kernel interrupt thread and
i386 layers into a single machine-independant kernel interrupt threa

Major cleanup of the interrupt registration subsystem.

* Collapse the separate registrations in the kernel interrupt thread and
i386 layers into a single machine-independant kernel interrupt thread layer
in kern/kern_intr.c. Get rid of the i386 layer's 'MUX' code entirely.

* Have the interrupt vector assembly code (icu_vector.s and apic_vector.s)
call a machine-independant function in the kernel interrupt thread
layer to figure out how to process an interrupt.

* Move a lot of assembly into the new C interrupt processing function.

* Add support for INTR_MPSAFE. If a device driver registers an interrupt
as being MPSAFE, the Big Giant Lock will not be obtained or required.

* Temporarily just schedule the ithread if a FAST interrupt cannot be executed
due to its serializer being locked.

* Add LWKT serialization support for a non-blocking 'try' function.

* Get rid of ointhand2_t and adjust all old ISA code to use inthand2_t.

* Supply a frame pointer as a pointer rather then embedding it on th stack.

* Allow FAST and SLOW interrupts to be mixed on the same IRQ, though this
will not necessarily result in optimal operation.

* Remove direct APIC/ICU vector calls from the apic/icu vector assembly code.
Everything goes through the new routine in kern/kern_intr.c now.

* Add a new flag, INTR_NOPOLL. Interrupts registered with the flag will
not be polled by the upcoming emergency general interrupt polling
sysctl (e.g. ATA cannot be safely polled due to the way ATA register
access interferes with ATA DMA).

* Remove most of the distinction in the i386 assembly layers between FAST
and SLOW interrupts (part 1/2).

* Revamp the interrupt name array returned to userland to list multiple
drivers associated with the same IRQ.

show more ...

Get rid of bus_{disable,enable}_intr(), it wasn't generic enough for
our needs.

Implement some generic atomic.h functions to aid in the implementation of
a low level mutex.

Implement a generic low

Get rid of bus_{disable,enable}_intr(), it wasn't generic enough for
our needs.

Implement some generic atomic.h functions to aid in the implementation of
a low level mutex.

Implement a generic low level sleep-mutex serializer, kern/lwkt_serialize.c.
The serializer is designed to be a replacement for SPL calls but may also
be used for other very low level work (e.g. lockmgr interlocks).

Add a serializer argument to BUS_SETUP_INTR(). When non-NULL, the interrupt
handler will no longer be protected by an SPL so e.g. spl*() will no
longer protect against that device's interrupts.

The IF queueing and dequeueing mechanisms may no longer depend on outside
SPL state because network driver interrupt handlers are no longer required to
enter splnet(). Use critical sections for the moment. The IFQ and
IFF_OACTIVE interactions are not yet MP safe.

show more ...

Remove useless belt and suspenders include guards in some of our headers.

For these headers:

/usr/include/machine/atomic.h
/usr/include/machine/bus_dma.h
/usr/include/machine/coredump.h
/usr/includ

Remove useless belt and suspenders include guards in some of our headers.

For these headers:

/usr/include/machine/atomic.h
/usr/include/machine/bus_dma.h
/usr/include/machine/coredump.h
/usr/include/machine/cpufunc.h
/usr/include/machine/db_machdep.h
/usr/include/machine/elf.h
/usr/include/machine/endian.h
/usr/include/machine/frame.h
/usr/include/machine/limits.h
/usr/include/machine/npx.h
/usr/include/machine/profile.h
/usr/include/machine/segments.h
/usr/include/machine/stdarg.h
/usr/include/machine/stdint.h
/usr/include/machine/trap.h
/usr/include/machine/tss.h
/usr/include/machine/ucontext.h
/usr/include/machine/vframe.h
/usr/include/machine/vm86.h

All these headers #define _CPU_... and not _MACHINE_... even though they
are in /usr/include/machine. And the headers themselves have include
guards already. So there's little point in having them around the actual
#include additionally.

show more ...

kernel - Add LWKT_SERIALIZE_INITIALIZER

* Add an initializer for serializer static/global declarations.