Ensure the userland thread and floating-point state has been saved before
copying the pcb. These values may have been changed just before the call
to fork and without a call to cpu_switch, where they

Ensure the userland thread and floating-point state has been saved before
copying the pcb. These values may have been changed just before the call
to fork and without a call to cpu_switch, where they would have been saved.

Sponsored by: The FreeBSD Foundation

show more ...

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a fu

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation

show more ...

rtld: Clear x29 and lr for thread_start to conform Aarch64 ABI pcs

Strictly speaking, SYS V Aarch64 ABI does not states that lr should be
cleared, however gdb relies on lr and tryes to unwind stack

rtld: Clear x29 and lr for thread_start to conform Aarch64 ABI pcs

Strictly speaking, SYS V Aarch64 ABI does not states that lr should be
cleared, however gdb relies on lr and tryes to unwind stack further.

Reviewed by:
Differential Revision: https://reviews.freebsd.org/D40841

show more ...

Remove unused registes from the arm pcb

These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.

Reviewed by: jhb
Sponsored by: Arm

Remove unused registes from the arm pcb

These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.

Reviewed by: jhb
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35378

show more ...

Add macros for arm64 pcb register offsets

Add macros for offsets of macros we set in the arm64 pcb pcb_x array.
This will simplift reducing the size of this array in a later change.

Sponsored by: A

Add macros for arm64 pcb register offsets

Add macros for offsets of macros we set in the arm64 pcb pcb_x array.
This will simplift reducing the size of this array in a later change.

Sponsored by: Arm Ltd

show more ...

Split out vfp_new_thread

To keep the vfp thread creation code in one place move into vfp.c. This
will also help with adding SVE support as it depends on VFP.

Reviewed by: markj
Sponsored by: The Fr

Split out vfp_new_thread

To keep the vfp thread creation code in one place move into vfp.c. This
will also help with adding SVE support as it depends on VFP.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35615

show more ...

Save only callee-saved registers in pcb

On AArch64, registers x9-x18 are not callee-saved, yet they are
preserved at many placed in swtch.S. This patch removes code that
preserves these registers.

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier an

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

show more ...

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi ha

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi has been tied to a "virtual memory" swi
(swi_vm). However, all of the swi_vm implementations are the same and
consist of checking a flag (busdma_swi_pending) which is always true
and if set calling busdma_swi. I suspect this dates back to the
pre-SMPng days and that the intention was for swi_vm to serve as a
mux. However, in the current scheme there's no need for the mux.

Instead, remove swi_vm and vm_ih. Each bus_dma implementation that
uses bounce pages is responsible for creating its own swi (busdma_ih)
which it now schedules directly. This swi invokes busdma_swi directly
removing the need for busdma_swi_pending.

One consequence is that the swi now works on RISC-V which had previously
failed to invoke busdma_swi from swi_vm.

Reviewed by: imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33447

show more ...

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is un

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

show more ...

arm64: Handle thumb2 thread entry point.

In cpu_set_upcall(), if the thread startup routine is a thumb routine, make
sure to set PSR_T, so that the CPU will run in thumb mode.

MFC After: 1 week

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsor

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477

show more ...

arm64: clear debug register state on fork

Following the analogous change for amd64 and i386 in 8223717ce62c,
ensure that new processes start with these registers inactive.

PR: 254661
Reported by:

arm64: clear debug register state on fork

Following the analogous change for amd64 and i386 in 8223717ce62c,
ensure that new processes start with these registers inactive.

PR: 254661
Reported by: Michał Górny
Reviewed by: kib, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29630

show more ...

Replace the arm64 initial_fpcr with a macro

This value was never changed from its default value. Replace it with a
macro.

Sponsored by: Innovate UK

Arch64: Clear VFP state on execve()

I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being

Arch64: Clear VFP state on execve()

I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being reset when starting a new process. This change resets the
VFP inside exec_setregs() to ensure no VFP register state is leaked from
parent processes to children.

This commit also moves the clearing of fpcr that was added in 65618fdda0f27
from fork() to execve() since that makes more sense: fork() can retain
current register values, but execve() should result in a well-defined
clean state.

Reviewed By: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29060

show more ...

AArch64: Don't set flush-subnormals-to-zero flag on startup

This flag has been set on startup since 65618fdda0f272a823e6701966421bdca0efa301.
However, This causes some of the math-related tests to f

AArch64: Don't set flush-subnormals-to-zero flag on startup

This flag has been set on startup since 65618fdda0f272a823e6701966421bdca0efa301.
However, This causes some of the math-related tests to fail as they report
zero instead of a tiny number. This fixes at least
/usr/tests/lib/msun/ldexp_test and possibly others.
Additionally, setting this flag prevents printf() from printing subnormal
numbers in decimal form.
See also https://www.openwall.com/lists/musl/2021/02/26/1

PR: 253847
Reviewed By: mmel
Differential Revision: https://reviews.freebsd.org/D28938

show more ...

arm64: optimize set_syscall_retval()

Microoptimize set_syscall_retval() for arm64 by predicting
the return value to be zero. This is similar to what has
been done for other architectures

Reviewed

arm64: optimize set_syscall_retval()

Microoptimize set_syscall_retval() for arm64 by predicting
the return value to be zero. This is similar to what has
been done for other architectures

Reviewed By: emaste, mhorne
Differential Revision: https://reviews.freebsd.org/D26991

show more ...

arm64: Initialize VFP control register.

The RW fields in this register reset to architecturally unknown values,
so initialize these to the proper rounding and denormal mode.
MFC after: 1 week

arm64: Clear FPU flags in the pcb in cpu_copy_thread().

New threads start off with clean FPU state instead of inheriting state
from the parent thread.

PR: 247945
Sponsored by: Netflix

arm64: Trim duplicate code from cpu_fork_kthread_handler().

cpu_fork_kthread_handler() is always called after either cpu_fork() or
cpu_copy_thread(). The arm64 version was duplicating some of the w

arm64: Trim duplicate code from cpu_fork_kthread_handler().

cpu_fork_kthread_handler() is always called after either cpu_fork() or
cpu_copy_thread(). The arm64 version was duplicating some of the work
already done by both of those functions.

Sponsored by: Netflix

show more ...

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus.

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus. Rather than writing to this field in
more places, it can be removed in favor of using the value in the link
register.

To make this clearer, pcb->pcb_x[30] is renamed to pcb->pcb_lr, similar
to what already exists in struct trapframe. Also, prefer lr to x30 in
assembly, as it better conveys intention.

This improves PC_REGS() for kdb_thread != curthread. It is required for
a functional gdb(4) stub, fixing the output of `info threads`, in
particular.

The space occupied by pcb_pc is retained, for compatibility with kgdb.

Reviewed by: markj, jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27720

show more ...

Move SV_ABI_ERRNO translation into linux-specific code, to simplify
the syscall path and declutter it a bit. No functional changes intended.

Reviewed by: kib (earlier version)
MFC after: 2 weeks
Sp

Move SV_ABI_ERRNO translation into linux-specific code, to simplify
the syscall path and declutter it a bit. No functional changes intended.

Reviewed by: kib (earlier version)
MFC after: 2 weeks
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26378

show more ...

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

show more ...

Allow exceptions to be masked when in userspace

We may want to mask exceptions when in userspace. This was previously
impossible as threads are created with all exceptions unmasked and
signals expec

Allow exceptions to be masked when in userspace

We may want to mask exceptions when in userspace. This was previously
impossible as threads are created with all exceptions unmasked and
signals expected userspace to mask any. Fix these by copying the
mask state on thread creation and allow exceptions to be masked on
signal return, as long as they don't change.

Sponsored by: DARPA, AFRL

show more ...

amd64 KPTI: add control from procctl(2).

Add the infrastructure to allow MD procctl(2) commands, and use it to
introduce amd64 PTI control and reporting. PTI mode cannot be
modified for existing pm

amd64 KPTI: add control from procctl(2).

Add the infrastructure to allow MD procctl(2) commands, and use it to
introduce amd64 PTI control and reporting. PTI mode cannot be
modified for existing pmap, the knob controls PTI of the new vmspace
created on exec.

Requested by: jhb
Reviewed by: jhb, markj (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D19514

show more ...