Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into st

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

show more ...

Mark the context switch counters volatile (because preemption).

Minor scheduler cleanup:

- Adapt to cpu_need_resched() changes. Avoid lost & duplicate IPIs and ASTs.
sched_resched_cpu() and sched_resched_lwp() contain the logic for this.
- Changes for LSIDL to

Minor scheduler cleanup:

- Adapt to cpu_need_resched() changes. Avoid lost & duplicate IPIs and ASTs.
sched_resched_cpu() and sched_resched_lwp() contain the logic for this.
- Changes for LSIDL to make the locking scheme match the intended design.
- Reduce lock contention and false sharing further.
- Numerous small bugfixes, including some corrections for SCHED_FIFO/RT.
- Use setrunnable() in more places, and merge cut & pasted code.

show more ...

lwp_setlock(): return pointer to the kmutex_t that we replaced

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

show more ...

Define LW_DBGSUSPEND in sys/lwp.h

This flag specifies the suspended by debugger property, as used by
PT_SUSPEND in ptrace(2).

Add an explicit node that lwp status numbers are set in stone

Requested by <mrg>

Stop defining in lwp.h: LSDEAD

Not used since NetBSD-5.0.

It had a confusing comment that suggested that all entries after LSDEAD
are removed, which wasn't true.

Correct use-after-free issue in vfork(2)

In the previous behavior vforking parent was keeping pointer to a child
and checking whether it clears a PL_PPWAIT in its bitfield p_lflag. However
a child c

Correct use-after-free issue in vfork(2)

In the previous behavior vforking parent was keeping pointer to a child
and checking whether it clears a PL_PPWAIT in its bitfield p_lflag. However
a child can go invalid between exec/exit event from child and waking up
vforked parent and this can cause invalid pointer read and in the worst
scenario kernel crash.

In the new behavior vforked child keeps a reference to vforked parent LWP
and sets a value l_vforkwaiting to false. This means that vforked child
can finish its work, exec/exit and be terminated and once parent will be
woken up it will read its own field whether its child is still blocking.

Add new field in struct lwp: l_vforkwaiting protected by proc_lock.
In future it should be refactored and all PL_PPWAIT users transformed to
l_vforkwaiting and next l_vforkwaiting probably transformed into a bit
field.

This is another attempt of fixing this bug after <rmind> from 2012 in
commit:

Author: rmind <rmind@NetBSD.org>
Date: Sun Jul 22 22:40:18 2012 +0000

fork1: fix use-after-free problems. Addresses PR/46128 from Andrew Doran.
Note: PL_PPWAIT should be fully replaced and modificaiton of l_pflag by
other LWP is undesirable, but this is enough for netbsd-6.

The new version no longer performs unsafe access in l_lflag changing the
LP_VFORKWAIT bit.

Verified with ATF t_vfork and t_ptrace* tests and they are no longer
causing any issues in my local setup.

Fixes PR/46128 by Andrew Doran

show more ...

Implement an aggressive psref leak detector

It is yet another psref leak detector that enables to tell where a leak occurs
while a simpler version that is already committed just tells an occurrence

Implement an aggressive psref leak detector

It is yet another psref leak detector that enables to tell where a leak occurs
while a simpler version that is already committed just tells an occurrence of a
leak.

Investigating of psref leaks is hard because once a leak occurs a percpu list of
psref that tracks references can be corrupted. A reference to a tracking object
is memorized in the list via an intermediate object (struct psref) that is
normally allocated on a stack of a thread. Thus, the intermediate object can be
overwritten on a leak resulting in corruption of the list.

The tracker makes a shadow entry to an intermediate object and stores some hints
into it (currently it's a caller address of psref_acquire). We can detect a
leak by checking the entries on certain points where any references should be
released such as the return point of syscalls and the end of each softint
handler.

The feature is expensive and enabled only if the kernel is built with
PSREF_DEBUG.

Proposed on tech-kern

show more ...

Implement a simple psref leak detector

It detects leaks by counting up the number of held psref by an LWP and checking
its zeroness at the end of syscalls and softint handlers. For the counter, a
u

Implement a simple psref leak detector

It detects leaks by counting up the number of held psref by an LWP and checking
its zeroness at the end of syscalls and softint handlers. For the counter, a
unused field of struct lwp is reused.

The detector runs only if DIAGNOSTIC is turned on.

show more ...

Move pointer to fstrans private data into "struct lwp".

Ride NetBSD 8.99.35

Move counting involuntary switches into mi_switch. preempt() passes that
information by setting a new LWP flag.

While here, don't even try to switch when the scheduler has no other LWP
to run. This

Move counting involuntary switches into mi_switch. preempt() passes that
information by setting a new LWP flag.

While here, don't even try to switch when the scheduler has no other LWP
to run. This check is currently spread over all callers of preempt()
and will be removed there.

ok mrg@.

show more ...

s/static inline/static __inline/g for consistency with other include
headers.

Add missing barriers to curlwp_bind and curlwp_bindx

The barriers prevent the instruction of setting/clearing the LP_BOUND flag
from reordering over where we want to prevent LWP migrations.

Note th

Add missing barriers to curlwp_bind and curlwp_bindx

The barriers prevent the instruction of setting/clearing the LP_BOUND flag
from reordering over where we want to prevent LWP migrations.

Note that the fix doesn't mean that there was a race condition. For now the API
is used only for psref and the combination use of them doesn't need the
barriers(*).

(*) https://mail-index.netbsd.org/tech-kern/2018/02/15/msg023101.html

Pointed out by Mateusz Guzik

show more ...

typos

Check LP_BOUND is surely set in curlwp_bindx

This may find an extra call of curlwp_bindx.

allow cv_signal() immediately followed by cv_destroy().
this sequence is used by ZFS in a couple places and by supporting it
natively we can undo our local ZFS changes that avoided it.
note that this

allow cv_signal() immediately followed by cv_destroy().
this sequence is used by ZFS in a couple places and by supporting it
natively we can undo our local ZFS changes that avoided it.
note that this is only legal when all of the waiters use cv_wait()
and not any of the other variations, and lockdebug will catch
any violations of this rule.

show more ...

- Propagate the signal mask from the ucontext_t to the newly created thread
as specified by _lwp_create(2)
- Reset the signal stack for threads created with _lwp_create(2)

Add new ptrace(2) API: PT_SETSTEP & PT_CLEARSTEP

These operations allow to mark thread as a single-stepping one.

This allows to i.a.:
- single step and emit a signal (PT_SETSTEP & PT_CONTINUE)
-

Add new ptrace(2) API: PT_SETSTEP & PT_CLEARSTEP

These operations allow to mark thread as a single-stepping one.

This allows to i.a.:
- single step and emit a signal (PT_SETSTEP & PT_CONTINUE)
- single step and trace syscall entry and exit (PT_SETSTEP & PT_SYSCALL)

The former is useful for debuggers like GDB or LLDB. The latter can be used
to singlestep a usermode kernel. These examples don't limit use-cases of
this interface.

Define PT_*STEP only for platforms defining PT_STEP.

Add new ATF tests setstep[1234].

These ptrace(2) operations first appeared in FreeBSD.

Sponsored by <The NetBSD Foundation>

show more ...

GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is d

GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

show more ...

Introduce curlwp_bind and curlwp_bindx

The API prevents the current LWP from migrating between CPUs during
the critical section (between curlwp_bind and curlwp_bindx). One use
case of it is psref(9)

Introduce curlwp_bind and curlwp_bindx

The API prevents the current LWP from migrating between CPUs during
the critical section (between curlwp_bind and curlwp_bindx). One use
case of it is psref(9) that has a contract that forbids such migrations.

Proposed at http://mail-index.netbsd.org/tech-kern/2016/06/13/msg020710.html
(check it out if you want to know why the function names are chosen)

show more ...

Provide struct cpu_info *lwp_getcpu(struct lwp *) inline for <machine/cpu.h>
<machine/cpu.h> is include by <sys/lwp.h> before struct lwp is defined so
it can't access members inside it. This provide

Provide struct cpu_info *lwp_getcpu(struct lwp *) inline for <machine/cpu.h>
<machine/cpu.h> is include by <sys/lwp.h> before struct lwp is defined so
it can't access members inside it. This provides an accessor which is defined
after struct lwp is defined.

show more ...

It is now lwp_t::l_pcu_valid for the PCU changes (missed in the
previous commit).

prototype adjustments for lwp_park, welcome to 6.99.19