<sys/msgport.h>: For boolean_t, it is enough to include <sys/stdint.h>.

It was moved there from <sys/types.h> in 6b2b0a8b5c1c774a5c3cbb4c79bdc150.

Adjust headers for <machine/stdint.h> visibility.

This also reduces namespace pollution a bit. Include <machine/stdint.h>
where <stdint.h> is used too. External compiler under -ffreestanding
(_

Adjust headers for <machine/stdint.h> visibility.

This also reduces namespace pollution a bit. Include <machine/stdint.h>
where <stdint.h> is used too. External compiler under -ffreestanding
(__STDC_HOSTED__ == 0) will use their own <stdint.h> version and will
not include <machine/stdint.h>.

show more ...

<sys/msgport.h>: Add <sys/types.h> for boolean_t (kernel only).

kernel: Move M_LWKTMSG declaration to <sys/msgport2.h>.

Including <sys/malloc.h> has side effects, its introduction in this
header was done in 6aad077d69b6a2d39405e25c987bb090ff78a724.
The msgpor

kernel: Move M_LWKTMSG declaration to <sys/msgport2.h>.

Including <sys/malloc.h> has side effects, its introduction in this
header was done in 6aad077d69b6a2d39405e25c987bb090ff78a724.
The msgport2.h is a better place for M_LWKTMSG since all users already
include this header directly or indirectly through <net/netmsg2.h>.

Also, this moves <sys/malloc.h> inclusion after <sys/globaldata.h> down
the chain when including <sys/systm.h> (TBA later).

Fix a single case where malloc.h where not included before <sys/mount.h>
for M_MOUNT.

show more ...

uipc: Use sendmsg_oncpu, if possible.

This further reduces unnecessary IPIs (5500/s -> 5000/s).

msgport: Add message put done receipt support

The receipt callback is called immediately after the message is put on to
the target msgport. Complex operation is NOT allowed in this callback,
since

msgport: Add message put done receipt support

The receipt callback is called immediately after the message is put on to
the target msgport. Complex operation is NOT allowed in this callback,
since it could be called w/ spinlock for spin msgports.

It will be used to fix the race between the upper half (socket code)
and the so_port switching.

show more ...

<sys/msgport.h>: Extend the #ifdef _KERNEL to cover the lwkt_* protos too.

This unbreaks buildworld after the previous commit to <sys/msgport.h>.

Reported-by: Ed Berger <edwberger@gmail.com>

msgport: Add putport_oncpu; helps scheduling netisr locally for spin port

Background:
High rate (actually same rate as polling(4)) IPIs on random CPUs are
observed when polling(4) is enabled and the

msgport: Add putport_oncpu; helps scheduling netisr locally for spin port

Background:
High rate (actually same rate as polling(4)) IPIs on random CPUs are
observed when polling(4) is enabled and there is virtually no network
activity.

After polling(4) activities are traced using ktr(9), it turns out that the
high rate IPIs are actually from the wakeup() on netisr's msgport. Since
the sleep queue cpumask is indexed by the hash of ident, there are chances
that the netisr's msgport ident has the same hash value as other idents
that certain threads on other CPUs are waiting on. If this ever happens
(well, it does happen), the netisr's msgport wakeup will trigger "wakeup"
IPIs to other CPUs. However, these "wakeup" IPIs are actually useless,
since only netisr will wait on its msgport.

putport_oncpu() msgport method is added to call wakeup_mycpu() for spin
msgport, if we know that this port is only accessed by one thread on the
current CPU, e.g. polling(4). This is also the case for other network
code, e.g. syncache timeout, TCP timeout, fastforward flow cache timeout
etc. However, these network code's running rate is too low to unveil the
extra "wakeup" IPIs problem. lwkt_sendmsg_oncpu() is added as wrapper to
putport_oncpu() msgport method.

Currently, only polling(4) is using lwkt_sendmsg_oncpu(). Others will
be converted soon.

show more ...

msgport: Merge several sendmsg functions

sendmsg_stage1 and sendmsg_stage2 are actually copy and paste of part of
sendmsg. Make the functionality inline and let sendmsg call them
sequentially.

Whi

msgport: Merge several sendmsg functions

sendmsg_stage1 and sendmsg_stage2 are actually copy and paste of part of
sendmsg. Make the functionality inline and let sendmsg call them
sequentially.

While I am here, rename "stage1" to "prepare" and "stage2" to "start"

show more ...

kernel - Beef up lwkt_dropmsg() API and fix deadlock in so_async_rcvd*()

* Beef up the lwkt_dropmsg() API. The API now conditionally returns
success (0) or an error (ENOENT).

* so_pru_rcvd_async

kernel - Beef up lwkt_dropmsg() API and fix deadlock in so_async_rcvd*()

* Beef up the lwkt_dropmsg() API. The API now conditionally returns
success (0) or an error (ENOENT).

* so_pru_rcvd_async() improperly calls lwkt_sendmsg() with a spinlock
held. This is not legal. Hack up lwkt_sendmsg() a bit to resolve.

show more ...

msgport: Always save owner thread for threads' msgports

This unbreaks the assertion in dropmsg for spin msgport. Also for shared
spin msgport don't allow dropmsg.

While im here, add comment for mp

msgport: Always save owner thread for threads' msgports

This unbreaks the assertion in dropmsg for spin msgport. Also for shared
spin msgport don't allow dropmsg.

While im here, add comment for mp_dropmsg and adjust comment about mpu_td

Reported-by: pavalos@

show more ...

kernel/lwkt_msgport: Improve comments a bit.

Reviewed-by: Sacha Wildner <swildner@>

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

kernel - Fix spin-based msgports

* LWKT threads can use thread/IPI or spin-based message ports. The
default is thread-based. Spin-based ports had numerous problems which
would result in panics

kernel - Fix spin-based msgports

* LWKT threads can use thread/IPI or spin-based message ports. The
default is thread-based. Spin-based ports had numerous problems which
would result in panics. This commit fixes those panics and makes the
spinlock version viable.

* However, currently there is no performance improvement so the default
is staying as it was.

show more ...

kernel - Major vm_page, lwkt thread, and other changes

* Remove the rest of the LWKT fairq code, it may be added back in a different
form later. Go back to the strict priority model with round-ro

kernel - Major vm_page, lwkt thread, and other changes

* Remove the rest of the LWKT fairq code, it may be added back in a different
form later. Go back to the strict priority model with round-robining
of same-priority LWKT threads.

Currently the model scans gd_tdrunq for sort insertion, which is probably
a bit too inefficient.

* Refactor the LWKT scheduler clock. The round-robining is now based on
the head of gd->gd_tdrunq and the lwkt_schedulerclock() function will
move it. When a thread not on the head is selected to run (because
the head is contending on a token), the round-robin tick will force a
resched on the next tick. As before, we never reschedule-ahead the
kernel scheduler helper thread or threads that have already dropped
to a user priority.

* The token code now tries a little harder to acquire the token before
giving up, controllable with lwkt.token_spin and lwkt.token_delay
(token_spin is the number of times to try and token_delay is the delay
between tries, in nanoseconds).

* Fix a serious bug in usched_bsd4.c which improperly reassigned the 'dd'
variable and caused the scheduler helper to monitor the wrong dd
structure.

* Refactor the vm_page coloring code. On SMP systems we now use the
coloring code to implement cpu localization when allocating pages.
The pages are still 'twisted' based on their physical address so both
functions are served, but cpu localization is now the more important
function.

* Implement NON-OBJECT vm_page allocations. NULL may now be passed, which
allocates a VM page unassociated with any VM object. This will be
used by the pmap code.

* Implement cpu localization for zalloc() and friends. This removes a major
contention point when handling concurrent VM faults. The only major
contention point left is the PQ_INACTIVE vm_page_queues[] queue.

* Temporarily remove the VM_ALLOC_ZERO request. This will probably be
reenabled in a later commit.

* Remove MSGF_NORESCHED (it is not being used) and simplify related
lwkt scheduler functions.

* schedcpu_stats() and schedcpu_resource() no longer stall the callout
kernel threads when scanning allproc, if they are unable to acquire
proc->p_token.

* Move the need_lwkt_resched() from hardclock() to lwkt_schedulerclock()
(which hardclock() calls).

show more ...

Resurrect priority message support; there is always +/-5~10K measurement error.

Remove MSGF_PRIORITY support. The flag testing and message queue selection
on the hot code path introduce noticeable performance regression during ip
forwarding (from 667Kpps to 655Kpps w/ 64bytes p

Remove MSGF_PRIORITY support. The flag testing and message queue selection
on the hot code path introduce noticeable performance regression during ip
forwarding (from 667Kpps to 655Kpps w/ 64bytes packet and fastforwarding
enabled on Phenom 9550).

show more ...

- Add priority message queue to msgport. Send a message with MSGF_PRIORITY
flag will queue the message into the priority message queue of the target
port. The priority message queue takes prece

- Add priority message queue to msgport. Send a message with MSGF_PRIORITY
flag will queue the message into the priority message queue of the target
port. The priority message queue takes precendence over normal message
queue, so the messages with MSGF_PRIORITY flag will be processed before
other messages on the same target port. This could be used by defering
callout or operation that should not be delayed too long on the target
port.
- Add dropmsg function to msgport. Message must be marked with MSGF_DROPABLE,
else dropmsg operation is not allowed. Message marked with MSGF_DROPABLE
is not waitable, i.e. you could not call domsg on this kind of message.
Currently only thread msgport supports this operation and this operation
must be performed in the same thread of the msgport's owner thread.

Discussed-with: dillon@

show more ...

Add a MSGF_NORESCHED feature for lwkt thread-based message ports. The
idea is to use it to allow certain async messages to be queued to higher
priority system threads and schedule those threads with

Add a MSGF_NORESCHED feature for lwkt thread-based message ports. The
idea is to use it to allow certain async messages to be queued to higher
priority system threads and schedule those threads without forcing an
immediate reschedule.

The feature will be used by the new socket code to prevent cavitation
between a user process and system protocol thread when the user process
is write()ing a lot of data over the network.

show more ...

AMD64 Support:

* Move amd64-specific signal.h defs to machine/signal.h

* Change size field in tls_info from 'int' to 'long' to accomodate
64 bit architectures.

Submitted-by: Matthew Dillon
Obtai

AMD64 Support:

* Move amd64-specific signal.h defs to machine/signal.h

* Change size field in tls_info from 'int' to 'long' to accomodate
64 bit architectures.

Submitted-by: Matthew Dillon
Obtained-from: FreeBSD, Jordan Gordeev <jgordeev@dir.bg>, and Matt

show more ...

Add serializer port backend. Implementation is mainly based on spin
port backend, but unlike spin port backend: users of this port backend
are assumed to have the port's serializer held.

Proper fun

Add serializer port backend. Implementation is mainly based on spin
port backend, but unlike spin port backend: users of this port backend
are assumed to have the port's serializer held.

Proper function name is suggested by corecode@, aggelos@, tgen@ and
dillon@

Reviewed-by: tgen@, dillon@

show more ...

Add lwkt_sleep() to formalize a shortcut numerous bits of code have been
using for a while, which is to directly deschedule oneself and switch away.
This method of blocking requires a direct lwkt_sch

Add lwkt_sleep() to formalize a shortcut numerous bits of code have been
using for a while, which is to directly deschedule oneself and switch away.
This method of blocking requires a direct lwkt_schedule() call to reschedule
the thread and is primarily used by the message port abstraction.

Change the psignal code to check TDF_SINTR in the thread flags instead
of checking MSGPORTF_WAITING in the thread's private message port.

The lwkt_waitmsg() and lwkt_waitport() functions use the same msgport
backend function (mp_waitport). Separate the backend into two functions,
mp_waitport and mp_waitmsg, and allow tsleep flags to be passed in instead
of flagging interruptability in the lwkt_msg flags.

Optimize the lwkt_waitmsg() backends - in the fully synchronous critical
path case no critical sections or spinlocks are required at all.

show more ...

LWKT message ports contain a number of function pointers which abstract
their backend operation.

* Add a new function, mp_getport(), which takes over the functionality
of lwkt_getport().

* Formal

LWKT message ports contain a number of function pointers which abstract
their backend operation.

* Add a new function, mp_getport(), which takes over the functionality
of lwkt_getport().

* Formalize the default backend and rename it the 'thread' port backend, used
when a message port will only be drained by a single thread. This backend
is able to use critical sections and IPI messages to handle races.

* Fix a small timing window in the thread port backend where replying a
synchronous message request from a different cpu may fail to wake up
the originator who is waiting for the message completion.

* Abstract-out the message port initialization code and clean up related
code pollution.

* Add a new backend called the 'spin' port backend. This backend can be
used if a message port might be drained by several different threads.
For example, this would allow us to use a message port as part of a
file pointer / file descriptor construct.

* Add a boot-time tunable, lwkt.use_spin_port (defaults to off) which
forces spin ports to be used instead of thread ports for the per-thread
message port. This is used only for debugging.

show more ...

* Greatly reduce the complexity of the LWKT messaging and port abstraction.
Significantly reduce the overhead of the subsystem.

* The message abort algorithm has been rewritten. It now sends a

* Greatly reduce the complexity of the LWKT messaging and port abstraction.
Significantly reduce the overhead of the subsystem.

* The message abort algorithm has been rewritten. It now sends a
separate message to issue the abort instead of trying to requeue
the original message. This also means the TAILQ embedded in the
lwkt_msg structure can be used by unrelated code during processing
of the message.

* Numerous MSGF_ flags have been removed, and all the LWKT msg/port
algorithms have been rewritten and simplified. The message structure
is now only touched by the current owner in all situations.

* Numerous structural fields have been removed. In particular, the
fields used for message abort sequencing have been simplified and
we do not try to embed a 'command' field in the base LWKT message
any more.

* Clean up the netmsg abstraction, which is used all over the network stack.
Instead of trying to overload fields in lwkt_msg we now simply extend
the base lwkt_msg into struct netmsg. The function dispatch now takes
a netmsg and returns void (before we had to return EASYNC), and we no
longer need weird casts.

Accept/connect message aborts are now greatly simplified.

show more ...

Simplify the lwkt_msg structure by removing two unused fields and a number
of obsolete preprocessor defines.