The fp_ex_[st]w struct savefpu members were inherited from NetBSD where
they're used in the 32bit-compat support, which we dropped years ago.
Bye bye!

ok deraadt@

Start to add annotations to the cpu_info members, doing I/a/o for
immutable/atomic/owned ala <sys/proc.h>. Move CPUF_USERSEGS and
CPUF_USERXSTATE, which really are private to the CPU, into a new
ci_

Start to add annotations to the cpu_info members, doing I/a/o for
immutable/atomic/owned ala <sys/proc.h>. Move CPUF_USERSEGS and
CPUF_USERXSTATE, which really are private to the CPU, into a new
ci_pflags and rename s/CPUF_/CPUPF_/. Make all (remaining) ci_flags
alterations via atomic_{set,clear}bits_int(), so its annotation
isn't a lie. Delete ci_info member as unused all the way from
rev 1.1

ok jsg@ mlarkin@

show more ...

reset fpu in fpu_kernel_enter() even when there is no xsave state

Avoids the situation where FP code ran out of the amdgpu attachhook
would sometimes trigger SSE FP exception traps because precision

reset fpu in fpu_kernel_enter() even when there is no xsave state

Avoids the situation where FP code ran out of the amdgpu attachhook
would sometimes trigger SSE FP exception traps because precision
exceptions were not masked in mxcsr.

feedback from and ok kettenis@

show more ...

Move signal generation from fputrap() to where it's called in trap()

Switch from lazy FPU switching to semi-eager FPU switching: track whether
curproc's xstate ("extended state") is loaded in the CPU or not.
- context switch, sendsig(), vmm, and doing CPU crypto in t

Switch from lazy FPU switching to semi-eager FPU switching: track whether
curproc's xstate ("extended state") is loaded in the CPU or not.
- context switch, sendsig(), vmm, and doing CPU crypto in the kernel all
check the flag and, if set, save the old thread's state to the PCB,
clear the flag, and then load the _blank_ state
- when returning to userspace, if the flag is clear then set it and restore
the thread's state

This simpler tracking also fixes the restoring of FPU state after nested
signal handlers.

With this, %cr0's TS flag is never set, the FPU #DNA trap can no
longer happen, and IPIs are no longer necessary for flushing or
syncing FPU state; on the other hand, restoring xstate while returning
to userspace means we have to handle xrstor faulting if we could
be loading an altered state. If that happens, reset the state,
fake a #GP fault (SIGBUS), and recheck for ASTs.

While here, regularize fxsave/fxrstor vs xsave/xrstor handling, by
using codepatching to switch to xsave/xrstor when present in the
CPU. In addition, code patch in use of xsaveopt in most places
when the CPU supports that. Use the 64bit-wide variants of the
instructions in all cases so that x87 instruction fault IPs are
reported correctly.

This change has three motivations:
1) with modern clang, SSE registers are used even in rcrt0.o, making
lazy FPU switching a smaller benefit vs trap costs
2) the Intel SDM warns that lazy FPU switching may increase power costs
3) post-Spectre rumors suggest that the %cr0 TS flag might not block
speculation, permitting leaking of information about FPU state
(AES keys?) across protection boundaries.

tested by many in snaps; prodding from deraadt@

show more ...

Initialize processor extended state in fpu_kernel_enter

Tested by a few; OK visa, sthen

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Follow the pattern set by copy*/pcb_onfault: when xrstor faults, return
from the trap to a 'resume' address to effectively make xrstor_user()
return an error indication, then do the FPU cleanup and t

Follow the pattern set by copy*/pcb_onfault: when xrstor faults, return
from the trap to a 'resume' address to effectively make xrstor_user()
return an error indication, then do the FPU cleanup and trap generation
from there where we can get access to the original, userspace trapframe.

The original fix tried to handle the trap while on the wrong trapframe,
leaking kernel addresses and possibly leading to double faults.
Problem pointed out by abluhm@
ok deraadt@ mikeb@

show more ...

The xrstor instruction will fault if the provided xstate data, which
is under userspace control via sigreturn, fails various consistency
checks. Rather than trying to replicate the CPU's hardwired c

The xrstor instruction will fault if the provided xstate data, which
is under userspace control via sigreturn, fails various consistency
checks. Rather than trying to replicate the CPU's hardwired checks
in C code, handle it like iretq: check in trap() whether a fault
is from the problem instruction and handle it there.

CPU behavior and the potential issue pointed out on Linux kernel-hardening
ok mikeb@ deraadt@

show more ...

Kill SPINLOCK_SPIN_HOOK, use CPU_BUSY_CYCLE() instead.

ok visa@, kettenis@

vmm(4): proper save/restore of FPU context during entry/exit.

tested by reyk, dcoppa, and a few others.

ok kettenis@ on the fpu bits
ok deraadt@ on the vmm bits

KNF

Save/restore AVX registers and other XSAVE-managed state information when
entering/leaving a signal handler like we already do the the FPU and SSE
state. This should make it possible to use AVX inst

Save/restore AVX registers and other XSAVE-managed state information when
entering/leaving a signal handler like we already do the the FPU and SSE
state. This should make it possible to use AVX instructions in signal
handlers.

ok mlarkin@

show more ...

Add support for saving/restoring FPU state using the XSAVE/XRSTOR. Limit
support to the X87, SSE and AVX state.

This gives us (almost) full AVX support. The AVX state isn't saved by
signal handler

Add support for saving/restoring FPU state using the XSAVE/XRSTOR. Limit
support to the X87, SSE and AVX state.

This gives us (almost) full AVX support. The AVX state isn't saved by
signal handlers yet, and ptrace(2) support is still missing.

ok guenther@

show more ...

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

Include <sys/vmmeter.h> directly instead of relying on it being
pulled by <uvm/uvm_extern.h> and turn uvm_total() into a private
function.

The preferred way to get memory stats is through the VM_UVM

Include <sys/vmmeter.h> directly instead of relying on it being
pulled by <uvm/uvm_extern.h> and turn uvm_total() into a private
function.

The preferred way to get memory stats is through the VM_UVMEXP
sysctl(3) since VM_METER is just a wrapper on top of it. In the
kernel, use `uvmexp' directly instead of uvm_total().

This change does not remove <sys/vmmeter.h> from <uvm/uvm_extern.h>
to give some more time to port maintainers to fix their ports.

ok guenther@ as part of a larger diff.

show more ...

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Use an explicit suffix for the "fld" instruction to shut up clang. The correct
instruction is fldl sinze we try to load a double-precision value.
GCC actually gets it wrong and emits "flds" (which i

Use an explicit suffix for the "fld" instruction to shut up clang. The correct
instruction is fldl sinze we try to load a double-precision value.
GCC actually gets it wrong and emits "flds" (which is harmless).

Pointed out by Brad. ok guenther@

show more ...

Revert art@'s moving around of the KERNEL_LOCK()/KERNEL_UNLOCK() calls,
as it causes hangs in some ports, including libsigsegv's configure script

confirmed by krw@, landry@

remove irrelevant old comment

There is a bunch of places in the kernel entry points where we don't
hold the kernel lock, but still need call one function that needs it.

Instead of grabbing the lock all over the place, move the l

There is a bunch of places in the kernel entry points where we don't
hold the kernel lock, but still need call one function that needs it.

Instead of grabbing the lock all over the place, move the locks into
the affected functions: trapsignal, scdebug*, ktrsyscall, ktrsysret,
systrace_redirect and ADDUPROF. In the cases we already hold the biglock
we'll just recurse.

kettenis@, beck@ ok

show more ...

Clean up after P_BIGLOCK removal.
KERNEL_PROC_LOCK -> KERNEL_LOCK
KERNEL_PROC_UNLOCK -> KERNEL_UNLOCK

oga@ ok

When reading MXCSR from userland sigcontext or a ptrace request,
mask out invalid bits to prevent a protect fault.

Original diff by joshe@; further feedback and ok kettenis@

Back out previous, it appears to be broken.

When reading MXCSR from userland sigcontext, mask out invalid bits.

This prevents a protection fault if a userland signal handler
scribbles all over it's struct sigcontext

Help from and ok guenther

When reading MXCSR from userland sigcontext, mask out invalid bits.

This prevents a protection fault if a userland signal handler
scribbles all over it's struct sigcontext

Help from and ok guenther@ kettenis@

show more ...