arm32: Optimization: Omit needless membar when triggering softint.

When we are triggering a softint, it can't already hold any mutexes.
So any path to mutex_exit(mtx) must go via mutex_enter(mtx), w

arm32: Optimization: Omit needless membar when triggering softint.

When we are triggering a softint, it can't already hold any mutexes.
So any path to mutex_exit(mtx) must go via mutex_enter(mtx), which is
always done with atomic r/m/w, and we need not issue any explicit
barrier between ci->ci_curlwp = softlwp and a potential load of
mtx->mtx_owner in mutex_exit.

PR kern/57240

XXX pullup-8
XXX pullup-9
XXX pullup-10

show more ...

arm32: Add missing barriers in cpu_switchto.

Details in comments.

PR kern/57240

XXX pullup-8
XXX pullup-9
XXX pullup-10

typo in comment

Ensure that r5 contains curlwp before DO_AST_AND_RESTORE_ALIGNMENT_FAULTS
in lwp_trampoline as required by the move to make ASTs operate per-LWP
rather than per-CPU.

Thanks to martin@ for bisecting

Ensure that r5 contains curlwp before DO_AST_AND_RESTORE_ALIGNMENT_FAULTS
in lwp_trampoline as required by the move to make ASTs operate per-LWP
rather than per-CPU.

Thanks to martin@ for bisecting the amap corruption he was seeing and
testing this fix.

show more ...

#ifdef _ARM_ARCH_7 the dmbs

Mirror the changes to aarch64 and

- Switch to TPIDRPRW_IS_CURLWP, because curlwp is accessed much more often
by MI code. It also makes curlwp preemption safe,

- Make ASTs operate per-LWP rather

Mirror the changes to aarch64 and

- Switch to TPIDRPRW_IS_CURLWP, because curlwp is accessed much more often
by MI code. It also makes curlwp preemption safe,

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Make sure ASTs are always set on the same CPU as the target LWP, and
delivered via IPI if posted from a remote CPU so that they are resolved
quickly.

- Add some cache line padding to struct cpu_info.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit().

show more ...

Add support for KASAN on ARMv[67]

Thanks to maxv for many pointers and reviews.

Whitespace

KNF (sort #includes)

G/C

oldlwp is always non-NULL in cpu_switchto so remove the test for NULL.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

show more ...

Explicitly annotate FPU requirements for LLVM MC.

When using GCC, this annotations change the global state, but there is
no push/pop functionality for .fpu to avoid this problem. The state is
local

Explicitly annotate FPU requirements for LLVM MC.

When using GCC, this annotations change the global state, but there is
no push/pop functionality for .fpu to avoid this problem. The state is
local to each inline assembler block with LLVM MC.

show more ...

Typo in comment

Typo in comment

Whitespace (align comments)

Trailing whitespace

Make TPIDRPRW_IS_CURLWP work for MULTIPROCESSOR.
get curcpu() from new lwp.
don't set lwp l_cpu (already done).
Remove support for __HAVE_UNNESTED_INTRS
don't set curlwp until after we are done savin

Make TPIDRPRW_IS_CURLWP work for MULTIPROCESSOR.
get curcpu() from new lwp.
don't set lwp l_cpu (already done).
Remove support for __HAVE_UNNESTED_INTRS
don't set curlwp until after we are done saving the oldlwp.
disable interrupts when setting curlwp/kernel stack pointer.
Overall, these changes simplify cpu_switchto even more.

show more ...

There is no need to save/restore l_private in softint_switch now that
cpu_switchto is fixed

Rework register usage in cpu_switchto so curcpu() is preserved across
ras_lookup. Only set vfp & tpid registers and do ras lookups if new lwp
is not LW_SYSTEM. (tested on RPI and atf tests on BPI b

Rework register usage in cpu_switchto so curcpu() is preserved across
ras_lookup. Only set vfp & tpid registers and do ras lookups if new lwp
is not LW_SYSTEM. (tested on RPI and atf tests on BPI by skrll).

show more ...

Fix register usage in softint_switch. load / restore l_private across
softint_dispatch

Make sure to save the user thread point in softint_switch in case it was
set just before we got an interrupt. Otherwise if the softint blocks, the
old value would be restored and change lost.

src is too big these days to tolerate superfluous apostrophes. It's
"its", people!

Fix wrong instruction; mcr => mrc

ARM_MMU_EXTENDED support.