domain: Nuke pfslowtimo.

domain: Nuke pffasttimo

tcp: Make setsockopt(2) TCP_{NODELAY,NOPUSH,NOOPT,FASTKEEP} asynchronous

This makes all network system calls used by nginx asynchronous and greatly
improves nginx performance.

For 1 requests/connec

tcp: Make setsockopt(2) TCP_{NODELAY,NOPUSH,NOOPT,FASTKEEP} asynchronous

This makes all network system calls used by nginx asynchronous and greatly
improves nginx performance.

For 1 requests/connection (30K concurrent connections), this commit gives
12% improvement for 16KB web object (was 16.8Gbps, now 19.2Gbps, which
maxes out 2x10Ge). And it gives 19% improvement for 1KB web object (was
156Kreqs/s, now 186Kreqs/s).

show more ...

Remove advertising header from sys/

Correct BSD License clause numbering from 1-2-4 to 1-2-3.

Some less clear cases taken as it was done of FreeBSD.

socket/tcp: Implement asynchronized pru_attach for TCP

This commit mainly splits TCP pru_attach into two parts:
- First part operates on socket buffer, so it could run directly in
the caller threa

socket/tcp: Implement asynchronized pru_attach for TCP

This commit mainly splits TCP pru_attach into two parts:
- First part operates on socket buffer, so it could run directly in
the caller thread.
- The second part creates and initializes tcpcb, which still runs in
netisr. But we don't wait for the result of this operation
(lwkt_sendmsg() is used instead of lwkt_domsg()).

This removes the last lwkt_domsg on commonly/mostly used socket APIs.
This is enabled by default and could be disabled by setting sysctl
kern.ipc.socreate_fast to 0.

The measured effect of this change on 2-ways E5-2600v2 with Intel 82599
(10Gbe) using tools/kq_connect_client:
- Connect rate increases by ~10Kconns/s; we are now doing 395Kconns/s.
- Idle time on the CPUs not running netisrs increases (55% -> 65%).
- IPIs rate to the CPUs not running netisrs reduces (40Kipis/s ->
23Kipis/s).

show more ...

kernel: Remove our ancient SCTP support.

It was based on an early stage of FreeBSD's support and never got
beyond that point, due to lack of usage and lack of maintenance.

It is also kind of standi

kernel: Remove our ancient SCTP support.

It was based on an early stage of FreeBSD's support and never got
beyond that point, due to lack of usage and lack of maintenance.

It is also kind of standing the the way of ongoing work in the
IPv6 stack by sephe.

If anyone ever wants to start again on upgrading and maintaining it,
they can decide then if reverting this commit again makes sense as a
base.

Requested-by: sephe
Approved-by: dillon

show more ...

icmp: Asynchonize ctlinput/mtudisc/redirect processing

Supportive facilities added:
- Embed netmsg_ctlinput into mbuf (used by input ICMP datagram) so that
we don't need to do additional memory al

icmp: Asynchonize ctlinput/mtudisc/redirect processing

Supportive facilities added:
- Embed netmsg_ctlinput into mbuf (used by input ICMP datagram) so that
we don't need to do additional memory allocation to forward the
pr_ctlinput to all netisrs for certain types of ICMP datagrams.
- Add/Expose so_pr_ctlport().
- Return target netisr's cpuid from pr_ctlport. If the returned cpuid
equals to ncpus, it means that pr_ctlinput needs to be run in all
netisrs.
- Support 'direct' mode for TCP/UDP's pr_ctlinput. The caller of the
'direct' mode pr_ctlinput will do the message forwarding.

ICMP pr_ctlinput/mtudisc/redirect processing changes:
- For mtudisc and redirect we unconditionally forward the message
embedded in input ICMP datagram to all netisrs, and call pr_ctlinput
in 'direct' mode.
- For pr_ctlinput, we first call so_pr_ctlport() to see whether we need
to forward the message embedded in input ICMP datagram to all netisrs,
and act accordingly.
- Once pr_ctlinput/mtudisc/redirect processing is done, the message
embedded in the input ICMP datagram is forwarded back to the netisr0
to run rip_input().

After this commit, netisr0 will not be blocked for ICMP pr_ctlinput/
mtudisc/redirect processing. This also fixes the mtudisc (originally
only cpu0's routes get mtu updates).

show more ...

udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

MAJOR CHANGES:

- Add token to protect pcbinfo's inpcb list and wildcard hash table.
Currently only udp per-cpu pcbinfo sets

udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

MAJOR CHANGES:

- Add token to protect pcbinfo's inpcb list and wildcard hash table.
Currently only udp per-cpu pcbinfo sets this token. udp serializer
and netisr barrier are nuked.

o udp inpcb list:

Under most cases, udp inpcb list is operated in its owner netisr.
However, it is also accessed and modified (no effiective udp inpcb
will be unlinked though) in netisr0 to adjust multicast options if
one interface is to be detached. So protecting udp inpcb list
accessing and modification w/ token is necessary.

At udp inpcb detach time, the udp inpcb is first removed from the
udp inpcb list, then a message will go through all netisrs, which
makes sure that no netisrs are using or can find this udp inpcb
from the udp inpcb list. After all these, this udp inpcb is
destroyed in its owner netisr.

In netisrs, it is MP safe to find a udp inpcb from udp inpcb list,
then release the token and process the found udp inpcb.

In other threads, it is MP safe to find a udp inpcb from udp inpcb
list, then release the token and process the found udp inpcb in
non-blocking fashion.

See also the usage of inpcb marker.

o udp wildcard hash table:

On input path, udp wildcard hash table is searched in its owner
netisr. In order to ease implicit binding (bind during send),
connect after binding, and disconnect, udp inpcb are inserted
into and removed from other udp pcbinfos' wildcard hash table in
its owner netisr. Thus the udp wildcard hash table must be
protected w/ token.

At udp inpcb detach time, a message will go through all netisrs,
and this udp inpcb will be removed from the udp wildcard hash
table belonging to the current netisr. This makes sure that once
the current netisr runs the message handler, this udp inpcb will
not be used and be found in the current netisr. When the message
reaches the last netisr, this udp inpcb is redispatched to its
owner netisr to be destroyed.

In netisrs, it is MP safe to find a udp inpcb from udp wildcard
hash table, then release the token and process the found udp inpcb,
e.g. use udp inpcb found by in_pcblookuphash().

In other threads, it is MP safe to find a udp inpcb from udp
wildcard hash table, then release the token and process the found
udp inpcb in non-blocking fashion.

See also the usage of inpcb container marker.

o udp connect hash table:

It is lockless MP safe, and only accessed and modified in its owner
netisr.

- During inpcb iteration through inpcb list, use inpcb marker when
calling functions, which may block, e.g. in_pcbpurgeif0(), so the
inpcb iteration will not stop prematurely, if the inpcb being
processed is removed from the inpcb list.

- Use udp inpcb wildcard table and udp inpcb connect hash table to
dispatch input multicast and broadcast udp datagrams. Using udp inpcb
list could be time consume, since we need to check udp inpcb lists on
all cpus; and secondly, once udp inpcb has a local port, it will be in
either udp wildcard hash table or udp connect hash table.

Since the socket buffer operation on input path may block, inpcb
container marker is used when iterating inpcbs from udp inpcb wildcard
hash table. in_pcblookup_pkthash() is adjusted to skip inpcb
container marker.

- udp socket so_port is no longer fixed to netisr0 msgport
o Initial udp socket so_port is the current cpu's netisr msgport.
o Bound but unconnected udp socket so_port is selected according to
local port hash.
o Connected udp socket so_port is selected according to the udp hash,
i.e. laddr/faddr toeplitz hash (exception: multicast laddr or
multicast faddr, is hashed to netisr0).
o Multicast socket options are forced to be handled in netisr0, since
udp socket so_port may not be netisr0 msgport.

- In order to support asynchronized udp inpcb detach:
o EJUSTRETURN from pru_detach method now means protocol will call
sodiscard() and sofree() for soclose(). udp pru_detach method
returns EJUSTRETURN as of this commit.
o SS_ISCLOSING socket state is set before calling pru_detach method,
so protocol could avoid certain expensive, unnecessary or
disallowed operation in pru_disconnect or pru_detach method, e.g.
udp pru_disconnect method avoids putting udp inpcb back to udp
wildcard hash table, if SS_ISCLOSING is set.

MISC CHANGES:

- pcbinfo's cpu id must be set now; -1 is disallowed.
- udp pru_abort method should never be called; it panicks now.
- Restore traditional BSD behaviour, if unbound udp socket connect
fails: if local port of the udp socket has been selected, its inpcb
should be in wildcard hash table, i.e. the udp inpcb should be visible
on udp datagrams input path.
- Make sure multicast stuffs are adjusted only in netisr0 for inet6, if
one interface is about to be detached.

PERFORMANCE IMPROVEMENT:

For 'kq_connect_client -u' test, this commit gives 400% performance
improvement (31Kconns/s -> 160Kconns/s).

show more ...

<sys/protosw.h>: Use netmsg_t.

net: Add initport protosw method to init socket's default protocol port

Currently only tcp implements this method to distribute pru_attach and
pru_connect workload.

Nuke no longer necessary protosw

net: Add initport protosw method to init socket's default protocol port

Currently only tcp implements this method to distribute pru_attach and
pru_connect workload.

Nuke no longer necessary protosw flag PR_RAND_INITPORT and sysctl
kern.ipc.rand_initport.

show more ...

tcp: Implement random initial msgport

After e368a6e95e2cd9556a3e0fc43167d2dcf3a8253f, it is found that the
left bottle neck of nonblocking TCP connect(2) performance is that all
socket(2) and initia

tcp: Implement random initial msgport

After e368a6e95e2cd9556a3e0fc43167d2dcf3a8253f, it is found that the
left bottle neck of nonblocking TCP connect(2) performance is that all
socket(2) and initial TCP connect operation (bind laddr and lport) are
all carried out in netisr0; CPU0 is 100% busy during test.

The idea of random initial msgport for TCP is that instead of using
netisr0's msgport as initial msgport, we could use any of the available
netisr msgport to carry out socket(2) and initial TCP connect operation.

Most of parts of TCP are already ready for random initial msgport, only
TCP pru_listen requires trivial modification to fix the socket msgport
to netisr0's msgport (which is required to perform global wild hashtable
updating).

As of this commit, the current CPU's netisr msgport will be selected as
TCP socket's initial msgport, if random initial msgport is enabled.

Sysctl node kern.ipc.rand_initport is added to disable this optimization.
It is enabled by default.

This commit improves both nonblocking TCP connect(2) and blocking TCP
connect(2) performance.

Nonblocking connect(2) performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/kq_connect_client:

kq_connect_client -4 SERVADDR -p SERVPORT -i 8 -c 32 -l 30
(8 processes, each creates 32 connections simultaniously)

16 run average:
random initial msgport netisr0 msgport
263915.17 conns/s 220979.89 conns/s

This commit gives ~19% performance improvement for nonblocking connect(2)

Blocking connect(2) performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/connect_client:

connect_client -4 SERVADDR -p SERVPORT -i 256 -l 30
(256 processes)

16 run average:
random initial msgport netisr0 msgport
240235.23 conns/s 198312.87 conns/s

This commit gives ~21% performance improvement for blocking connect(2)

show more ...

proto: Fix comment

tcp: Implement asynchronous pru_connect

This is mainly used to improve TCP nonblocking connect(2) performance.

Before this commit the user space thread uses nonblocking connect(2)
will have to wait

tcp: Implement asynchronous pru_connect

This is mainly used to improve TCP nonblocking connect(2) performance.

Before this commit the user space thread uses nonblocking connect(2)
will have to wait for the netisr completes the SYN output. This could
be performance hit for nonblocking connect(2). First, the user space
thread is put into sleep, even if the connect(2) is nonblocking.
Second, it does not make too much sense for nonblocking connect(2) to
wait for the SYN output.

TCP's asynchronous pru_connect implementation will set ISCONNECTING
before dispatching netmsg to netisr0. The errors like EADDRNOTAVAIL,
i.e. out of local port space, will be notified through kevent(2) or
getsockopt(2) SOL_SOCKET/SO_ERROR.

NFS and other kernel code still use old synchronized pru_connect. This
commit only affects connect(2) syscall.

Sysctl node kern.ipc.soconnect_async is added to enable and disable
asynchronous pru_connect. It is enabled by default.

The performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/kq_connect_client:

kq_connect_client -4 SERVADDR -p SERVPORT -i 8 -c 32 -l 30
(8 processes, each creates 32 connections simultaniously, run 30 secs)

16 runs average:

asynchronous pru_connect synchronized pru_connect
220979.89 conns/s 189106.88 conns/s

This commit gives ~16% performance improvement for nonblocking connect(2)

show more ...

uipc: Reference the originator thread if asynchronized pru_send is used

It is required for UDP asynchronized pru_send, which could access the
originator's thread.

proto: Fix copy-n-paste error; PR_ASYNC_RCVD should be 0x0800.

It was defined to the same value as PR_ASYNC_SEND. However, in the
current code base, it does not hurt, since only TCP has this flag s

proto: Fix copy-n-paste error; PR_ASYNC_RCVD should be 0x0800.

It was defined to the same value as PR_ASYNC_SEND. However, in the
current code base, it does not hurt, since only TCP has this flag set.

show more ...

tcp: Implement asynchronized pru_rcvd

This mainly avoids extra scheduling cost on the reception path due to
lwkt_domsg(). lwkt_sendmsg() is now used to carry out TCP pru_rcvd.

Since TCP's pru_rcvd

tcp: Implement asynchronized pru_rcvd

This mainly avoids extra scheduling cost on the reception path due to
lwkt_domsg(). lwkt_sendmsg() is now used to carry out TCP pru_rcvd.

Since TCP's pru_rcvd could be batched, one pru_rcvd netmsg is embedded
into struct socket to avoid pru_rcvd netmsg allocation for each pru_rcvd,
and this netmsg will be used by lwkt_sendmsg(). Whether this embedded
pcu_rcvd netmsg should be sent or not is determined by its MSG_DONE bit.
Since user thread and netisr thread could be on different CPUs, the
embedded pru_rcvd netmsg's MSG_DONE bit is protected by a spinlock.

To cope with the following race that could drop window updates,
tcp_usr_rcvd() replies asynchronized rcvd netmsg before tcp_output():

netisr thread user thread

tcp_usr_rcvd() sorcvtcp()
{ {
tcp_output() :
: :
: sbunlinkmbuf()
: if (rcvd & MSG_DONE) (2)
: lwkt_sendmsg(rvcd)
: :
lwkt_replymsg(rcvd) (1)
}

At (2) window update is dropped, since rcvd netmsg is not replied yet at (1)

The result:
On i7-2600 (4C/8T, 3.4GHz):
32 parallel netperf -H 127.0.0.1 -t TCP_STREAM -P0 -l 30 (4 runs, unit: Mbps)

old 30253.88 30242.58 30162.55 30101.51
new 33962.74 33798.70 33499.92 33482.35

This gives ~12% performance improvement.

show more ...

accept: Call pru_accept in protocol thread

This avoids socket.so_pcb use-after-check race against protocol thread.

There is no performance impact on the mostly used sockets:
- IPv4/IPv6 TCP impleme

accept: Call pru_accept in protocol thread

This avoids socket.so_pcb use-after-check race against protocol thread.

There is no performance impact on the mostly used sockets:
- IPv4/IPv6 TCP implemented pru_savefaddr, so their pru_accept will not
be called at all
- UNIX domain socket uses sync msgport, so no protocol thread dispatching

show more ...

accept: Save foreign address earlier, if protocol supports it

- Add so_faddr into socket, which records the accepted socket's foreign
address. If it is set, kern_accept() will use it directly ins

accept: Save foreign address earlier, if protocol supports it

- Add so_faddr into socket, which records the accepted socket's foreign
address. If it is set, kern_accept() will use it directly instead of
calling protocol specific method to extract the foreign address.
- Add protocol specific method, pru_safefaddr, which will save the
foreign address into socket.so_faddr if the necessary information is
supplied. This protocol method will only be called in protocol
thread.
- Pass the foreign address to sonewconn() if possible, so the foreign
address could be saved before the accepted socket is put onto the
complete list.

Currently only IPv4/TCP implemented pru_savefaddr

This intends to address the following problems:
- Calling pru_accept directly from user context is not MPSAFE, we
always races the socket.so_pcb check->use against protocol thread
clear/free socket.so_pcb, though the race window is too tiny to
be hit. To make it mpsafe, we should dispatch pru_accept to
protocol thread.
If socket.so_faddr is set here, we are race against nothing and
nothing expensive like put the current user thread into sleep will
happen. However, if the socket is dropped when it still sits
on the complete list, the error will not be timely delivered, i.e.
accept(2) will not return error, but the later on read(2)/write(2)
on the socket will deliver the error.
- Calling pru_accept directly races against the inpcb.inp_f{addr,port}
setting up in the protocol thread, since inpcb.inp_f{addr,port} is
setup _after_ the accepted socket was put onto the complete list.

user thread proto thread
: :
: accepted socket -> comp
: (inpcb.inp_f{addr,port} are 0 here)
comp -> socket :
pru_accept :
: setup inpcb.inp_f{addr,port}

Returning of 0.0.0.0:0 from accept(2) was observed on heavily loaded
web servers.

show more ...

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

protosw: Add PR_ASYNC_SEND, mainly to make sure async pru_send is supported

Currently on IP/TCP and IPv6/TCP set this flag

tcp_{debug,timer}.h & protosw.h: Constify some strings.

Deal with the netgraph NULL function dereference on shutdown()
vulnerability. Use a solution similar to FreeBSD's.

pr_ctlinput is usually called when certains types of ICMP packets are received.
However, the processing of ICMP packets happens in netisr0, which means the
thread context, in which pr_ctlinput is cal

pr_ctlinput is usually called when certains types of ICMP packets are received.
However, the processing of ICMP packets happens in netisr0, which means the
thread context, in which pr_ctlinput is called, is not correct. To handle this
following two fixes are applied:
- Add pr_ctlport to protosw and ip6protosw, which could be used to locate
correct msgport to call pr_ctlinput for specific protocol
- All necessary information needed by pr_ctlinput are gather into one netmsg,
and this netmsg is delivered synchronously (some information is on the stack)

Note for new protocol implementation:
pr_ctlinput and pr_ctlport should be both NULL or both non-NULL.

Obtained-from: dillon@
Tested-by: pavalos@

show more ...

Add following three network protocol threads running mode:
1) BGL (default)
2) Adaptive BGL. Protocol threads run without BGL by default. BGL will be
held if the received msg does not have MSGF_

Add following three network protocol threads running mode:
1) BGL (default)
2) Adaptive BGL. Protocol threads run without BGL by default. BGL will be
held if the received msg does not have MSGF_MPSAFE turned on the ms_flags
field
3) No BGL (experimental)

The code on the main path is done by dillon@

Following three sysctls and tunables are added to adjust the "mode":
net.netisr.mpsafe_thread
net.inet.tcp.mpsafe_thread
net.inet.udp.mpsafe_thread

They have same set of values,
0 (default) -- BGL
1 -- Adaptive BGL
2 -- No BGL



NETISR_FLAG_MPSAFE is added (netisr.ni_flags), so that:
- netisr_queue() and schednetisr() could set MSGF_MPSAFE during msg
initialization
- netisr_run() (called by ether_input_oncpu()) could hold BGL based on this flag
before calling netisr's handler



PR_MPSAFE is added (protosw.pr_flags), so that tranport_processing_oncpu() could
hold BGL before calling protocol's input handler



Kernel API changes:
- The thread parameter to netmsg_service_loop() must be supplied (running mode)
and it must have the type of "int *"
- netisr_register() takes additional flags parameter to indicate whether its
handler is MPSAFE (NETISR_FLAG_MPSAFE) or not

Reviewed-by: dillon@

show more ...

do early copyin / delayed copyout for socket options