No need to set pkthdr fields to 0 that are already 0. MGETHDR() does that.
OK dlg@

spelling; ok mvs@

For consistency with other similar sysctl(2) variables use
atomic_load_int(9) while loading `somaxconn' and `sominconn'.

ok bluhm

Run socket splice idle timeout without kernel lock.

OK mvs@

Remove `sb_sel' from sobuf_print() output, no sense to print "...".

ok bluhm

Unlock udp(4) somove().

Socket splicing belongs to sockets buffers. udp(4) sockets are fully
switched to fine-grained buffers locks, so use them instead of exclusive
solock().

Always schedule somov

Unlock udp(4) somove().

Socket splicing belongs to sockets buffers. udp(4) sockets are fully
switched to fine-grained buffers locks, so use them instead of exclusive
solock().

Always schedule somove() thread to run as we do for tcp(4) case. This
brings delay to packet processing, but it is comparable wit non splicing
case where soreceive() threads are always scheduled.

So, now spliced udp(4) sockets rely on sb_lock() of `so_rcv' buffer
together with `sb_mtx' mutexes of both buffers. Shared solock() only
required around pru_send() call, so the most of somove() thread runs
simultaneously with network stack.

Also document 'sosplice' structure locking.

Feedback, tests and OK from bluhm.

show more ...

Fix source and drain confusion in socket splicing somove().

If a large mbuf in the source socket buffer does not fit into the
drain buffer, split the mbuf. But if the drain buffer still has
some da

Fix source and drain confusion in socket splicing somove().

If a large mbuf in the source socket buffer does not fit into the
drain buffer, split the mbuf. But if the drain buffer still has
some data in it, stop moving data and try again later. This skips
a potentially expensive mbuf operation.
When looking which socket buffer has to be locked, I found that the
length of the source send buffer was checked. Change it to drain.
As this is a performance optimization for a special corner case,
noone noticed the bug.

OK sashan@

show more ...

Switch `so_snd' of udp(4) sockets to the new locking scheme.

udp_send() and following udp{,6}_output() do not append packets to
`so_snd' socket buffer. This mean the sosend() and sosplice() sending

Switch `so_snd' of udp(4) sockets to the new locking scheme.

udp_send() and following udp{,6}_output() do not append packets to
`so_snd' socket buffer. This mean the sosend() and sosplice() sending
paths are dummy pru_send() and there is no problems to simultaneously
run them on the same socket.

Push shared solock() deep down to sesend() and take it only around
pru_send(), but keep somove() running unedr exclusive solock(). Since
sosend() doesn't modify `so_snd' the unlocked `so_snd' space checks
within somove() are safe. Corresponding `sb_state' and `sb_flags'
modifications are protected by `sb_mtx' mutex(9).

Tested and OK bluhm.

show more ...

Switch AF_ROUTE sockets to the new locking scheme.

At sockets layer only mark buffers as SB_MTXLOCK. At PCB layer only
protect `so_rcv' with corresponding `sb_mtx' mutex(9).

SS_ISCONNECTED and SS_C

Switch AF_ROUTE sockets to the new locking scheme.

At sockets layer only mark buffers as SB_MTXLOCK. At PCB layer only
protect `so_rcv' with corresponding `sb_mtx' mutex(9).

SS_ISCONNECTED and SS_CANTRCVMORE bits are redundant for AF_ROUTE
sockets. Since SS_CANTRCVMORE modifications performed with both solock()
and `sb_mtx' held, the 'unlocked' SS_CANTRCVMORE check in
rtm_senddesync() is safe.

ok bluhm

show more ...

Turn sblock() to `sb_lock' rwlock(9) wrapper for all sockets.

Unify behaviour to all sockets. Now sblock() should be always
taken before solock() in all involved paths as sosend(), soreceive(),
sorf

Turn sblock() to `sb_lock' rwlock(9) wrapper for all sockets.

Unify behaviour to all sockets. Now sblock() should be always
taken before solock() in all involved paths as sosend(), soreceive(),
sorflush() and sosplice(). sblock() is fine-grained lock which
serializes socket send and receive routines on `so_rcv' or `so_snd'
buffers. There is no big problem to wait netlock while holding sblock().

This unification removes a lot of temporary "sb_flags & SB_MTXLOCK" code
from sockets layer. This unification makes straight "solock()" and
"sblock()" lock order, no more solock() -> sblock() -> sounlock() ->
solock() -> sbunlock() -> sounlock() chains in sosend() and soreceive()
paths. This unification brings witness(4) support for sblock(), include
NFS involved sockets, which is useful.

Since the witness(4) support was introduced to sblock() with this diff,
some new witness reports appeared.

bulk(1) tests by tb, ok bluhm

show more ...

Switch AF_KEY sockets to the new locking scheme.

The simplest case. Nothing to change in sockets layer, only set
SB_MTXLOCK on socket buffers.

ok bluhm

Push solock() down to sosend() and remove it from soreceive() paths fro
unix(4) sockets.

Push solock() deep down to sosend() and remove it from soreceive() paths
for unix(4) sockets.

The transmissi

Push solock() down to sosend() and remove it from soreceive() paths fro
unix(4) sockets.

Push solock() deep down to sosend() and remove it from soreceive() paths
for unix(4) sockets.

The transmission of unix(4) sockets already half-unlocked because
connected peer is not locked by solock() during sbappend*() call. Use
`sb_mtx' mutex(9) and `sb_lock' rwlock(9) to protect both `so_snd' and
`so_rcv'.

Since the `so_snd' is protected by `sb_mtx' mutex(9) the re-locking
is not required in uipc_rcvd().

Do direct `so_rcv' dispose and cleanup in sofree(). This sockets is
almost dead and unlinked from everywhere include spliced peer, so
concurrent sotask() thread will just exit. This required to keep locks
order between `i_lock' and `sb_lock'. Also this removes re-locking from
sofree() for all sockets.

SB_OWNLOCK became redundant with SB_MTXLOCK, so remove it. SB_MTXLOCK
was kept because checks against SB_MTXLOCK within sb*() routines are mor
consistent.

Feedback and ok bluhm

show more ...

Pass `sosp' instead of `so' to sblock() when locking `so_snd' within
sosplice().

ok bluhm

Push solock() down to sosend() for SOCK_RAW sockets.

Raw sockets are the simplest inet sockets, so use them to start landing
`sb_mtx' mutex(9) protection for `so_snd' buffer. Now solock() is taken

Push solock() down to sosend() for SOCK_RAW sockets.

Raw sockets are the simplest inet sockets, so use them to start landing
`sb_mtx' mutex(9) protection for `so_snd' buffer. Now solock() is taken
only around pru_send*(), the rest of sosend() serialized by sblock() and
`sb_mtx'. The unlocked SS_ISCONNECTED check is fine, because
rip{,6}_send() check it. Also, previously the SS_ISCONNECTED could be
lost due to solock() release around following m_getuio().

ok bluhm

show more ...

Don't take solock() in soreceive() for udp(4) sockets.

These sockets are not connection oriented, they don't call pru_rcvd(),
but they have splicing ability and they set `so_error'.

Splicing abilit

Don't take solock() in soreceive() for udp(4) sockets.

These sockets are not connection oriented, they don't call pru_rcvd(),
but they have splicing ability and they set `so_error'.

Splicing ability is the most problem. However, we can hold `sb_mtx'
around `ssp_socket' modifications together with solock(). So the
`sb_mtx' is pretty enough to isspiced() check in soreceive(). The
unlocked `so_sp' dereference is fine, because we set it only once for
the whole socket life-time and we do this before `ssp_socket'
assignment.

We also need to take sblock() before splice sockets, so the sosplice()
and soreceive() are both serialized. Since `sb_mtx' required to unsplice
sockets too, it also serializes somove() with soreceive() regardless on
somove() caller.

The sosplice() was reworked to accept standalone sblock() for udp(4)
sockets.

soreceive() performs unlocked `so_error' check and modification.
Previously, we have no ability to predict which concurrent soreceive()
or sosend() thread will fail and clean `so_error'. With this unlocked
access we could have sosend() and soreceive() threads which fails
together.

`so_error' stored to local `error2' variable because `so_error' could be
overwritten by concurrent sosend() thread.

Tested and ok bluhm

show more ...

Don't take solock() in soreceive() for SOCK_RAW inet sockets.

For inet sockets solock() is the netlock wrapper, so soreceive() could
be performed simultaneous with exclusively locked code paths.

Th

Don't take solock() in soreceive() for SOCK_RAW inet sockets.

For inet sockets solock() is the netlock wrapper, so soreceive() could
be performed simultaneous with exclusively locked code paths.

These sockets are not connection oriented, they don't call pru_rcvd(),
they can't be spliced, they don't set `so_error'. Nothing to protect
with solock() in soreceive() path.

`so_rcv' buffer protected by `sb_mtx' mutex(9), but since it released,
sblock() required to serialize concurrent soreceive() and sorflush()
threads. Current sblock() is some kind of rwlock(9) implementation, so
introduce `sb_lock' rwlock(9) and use it directly for that purpose.

The sorflush() and callers were refactored to avoid solock() for raw
inet sockets. This was done to avoid packet processing stop.

Tested and ok bluhm.

show more ...

Remove `head' socket re-locking in sonewconn().

uipc_attach() releases solock() because it should be taken after
`unp_gc_lock' rwlock(9) which protects the `unp_link' list. For this
reason, the list

Remove `head' socket re-locking in sonewconn().

uipc_attach() releases solock() because it should be taken after
`unp_gc_lock' rwlock(9) which protects the `unp_link' list. For this
reason, the listening `head' socket should be unlocked too while
sonewconn() calls uipc_attach(). This could be reworked because now
`so_rcv' sockbuf relies on `sb_mtx' mutex(9).

The last one `unp_link' foreach loop within unp_gc() discards sockets
previously marked as UNP_GCDEAD. These sockets are not accessed from the
userland. The only exception is the sosend() threads of connected
sending peers, but they only sbappend*() mbuf(9) to `so_rcv'. So it's
enough to unlink mbuf(9) chain with `sb_mtx' held and discard lockless.

Please note, the existing SS_NEWCONN_WAIT logic was never used because
the listening unix(4) socket protected from concurrent unp_detach() by
vnode(9) lock, however `head' re-locked all times.

ok bluhm

show more ...

Implement SO_ACCEPTCONN in getsockopt(2)
Requested by robert@
OK mvs@ millert@ deraadt@

Remove wrong "temporary udp error" comment in filt_so{read,write}(). Not
only udp(4) sockets set and check `so_error'.

No functional changes.

ok bluhm

Allow listen(2) only on sockets of type SOCK_STREAM or SOCK_SEQPACKET.
listen(2) man(1) page clearly prohibits sockets of other types.

Reported-by: syzbot+00450333592fcd38c6fe@syzkaller.appspotmail.

Allow listen(2) only on sockets of type SOCK_STREAM or SOCK_SEQPACKET.
listen(2) man(1) page clearly prohibits sockets of other types.

Reported-by: syzbot+00450333592fcd38c6fe@syzkaller.appspotmail.com

ok bluhm

show more ...

Mark `so_rcv' sockbuf of udp(4) sockets as SB_OWNLOCK.

sbappend*() and soreceive() of SB_MTXLOCK marked sockets uses `sb_mtx'
mutex(9) for protection, meanwhile buffer usage check and corresponding

Mark `so_rcv' sockbuf of udp(4) sockets as SB_OWNLOCK.

sbappend*() and soreceive() of SB_MTXLOCK marked sockets uses `sb_mtx'
mutex(9) for protection, meanwhile buffer usage check and corresponding
sbwait() sleep still serialized by solock(). Mark udp(4) as SB_OWNLOCK
to avoid solock() serialization and rely to `sb_mtx' mutex(9). The
`sb_state' and `sb_flags' modifications must be protected by `sb_mtx'
too.

ok bluhm

show more ...

Introduce SB_OWNLOCK to mark sockets which `so_rcv' buffer modified
outside socket lock.

`sb_mtx' mutex(9) used for this case and it should not be released between
`so_rcv' usage check and correspon

Introduce SB_OWNLOCK to mark sockets which `so_rcv' buffer modified
outside socket lock.

`sb_mtx' mutex(9) used for this case and it should not be released between
`so_rcv' usage check and corresponding sbwait() sleep. Otherwise wakeup()
could be lost sometimes.

ok bluhm

show more ...

Use `sb_mtx' to protect `so_rcv' receive buffer of unix(4) sockets.

This makes re-locking unnecessary in the uipc_*send() paths, because
it's enough to lock one socket to prevent peer from concurren

Use `sb_mtx' to protect `so_rcv' receive buffer of unix(4) sockets.

This makes re-locking unnecessary in the uipc_*send() paths, because
it's enough to lock one socket to prevent peer from concurrent
disconnection. As the little bonus, one unix(4) socket can perform
simultaneous transmission and reception with one exception for
uipc_rcvd(), which still requires the re-lock for connection oriented
sockets.

The socket lock is not held while filt_soread() and filt_soexcept()
called from uipc_*send() through sorwakeup(). However, the unlocked
access to the `so_options', `so_state' and `so_error' is fine.

The receiving socket can't be or became listening socket. It also can't
be disconnected concurrently. This makes immutable SO_ACCEPTCONN,
SS_ISDISCONNECTED and SS_ISCONNECTED bits which are clean and set
respectively.

`so_error' is set on the peer sockets only by unp_detach(), which also
can't be called concurrently on sending socket.

This is also true for filt_fiforead() and filt_fifoexcept(). For other
callers like kevent(2) or doaccept() the socket lock is still held.

ok bluhm

show more ...

Use sorflush() instead of direct unp_scan(..., unp_discard) to discard
dead unix(4) sockets.

The difference in direct unp_scan() and sorflush() is the mbuf(9) chain.
For the first case it is still l

Use sorflush() instead of direct unp_scan(..., unp_discard) to discard
dead unix(4) sockets.

The difference in direct unp_scan() and sorflush() is the mbuf(9) chain.
For the first case it is still linked to the `so_rcv', for the second it
is not. This is required to make `sb_mtx' mutex(9) the only `so_rcv'
sockbuf protection and remove socket re-locking from the most of
uipc_*send() paths. The unlinked mbuf(9) chain doesn't require any
protection, so this allows to perform sleeping unp_discard() lockless.

Also, the mbuf(9) chain of the discarded socket still contains addresses
of file descriptors and it is much safer to unlink it before FRELE()
them. This is the reason to commit this diff standalone.

ok bluhm

show more ...

Pass protosw instead of domain structure to soalloc() to get real
`pr_type'. The corresponding domain is referenced as `pr_domain'.
Otherwise dp->dom_protosw->pr_type of inet sockets always points
to

Pass protosw instead of domain structure to soalloc() to get real
`pr_type'. The corresponding domain is referenced as `pr_domain'.
Otherwise dp->dom_protosw->pr_type of inet sockets always points
to inetsw[0].

ok bluhm

show more ...