arm64: Add a CPU reset hook instead of expecting PSCI

Some SoCs do not include a PSCI for power management and defer it to
something else instead. Add a CPU reset hook to account for this, and
use i

arm64: Add a CPU reset hook instead of expecting PSCI

Some SoCs do not include a PSCI for power management and defer it to
something else instead. Add a CPU reset hook to account for this, and
use it in the psci driver.

Reviewed by: andrew
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D44535

show more ...

thread: Add a return value to cpu_set_upcall()

Some implementations copy data to userspace, an operation which can in
principle fail. In preparation for adding a __result_use_check
annotation to co

thread: Add a return value to cpu_set_upcall()

Some implementations copy data to userspace, an operation which can in
principle fail. In preparation for adding a __result_use_check
annotation to copyin() and related functions, let implementations of
cpu_set_upcall() return an error, and check for errors when copying data
to user memory.

Reviewed by: kib, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43100

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

arm64: Fix long lines in vm_machdep.c

Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41828

Add cpu_sync_core()

Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32360

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

rtld: Clear x29 and lr for thread_start to conform Aarch64 ABI pcs

Strictly speaking, SYS V Aarch64 ABI does not states that lr should be
cleared, however gdb relies on lr and tryes to unwind stack

rtld: Clear x29 and lr for thread_start to conform Aarch64 ABI pcs

Strictly speaking, SYS V Aarch64 ABI does not states that lr should be
cleared, however gdb relies on lr and tryes to unwind stack further.

Reviewed by:
Differential Revision: https://reviews.freebsd.org/D40841

show more ...

Remove unused registes from the arm pcb

These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.

Reviewed by: jhb
Sponsored by: Arm

Remove unused registes from the arm pcb

These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.

Reviewed by: jhb
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35378

show more ...

Add macros for arm64 pcb register offsets

Add macros for offsets of macros we set in the arm64 pcb pcb_x array.
This will simplift reducing the size of this array in a later change.

Sponsored by: A

Add macros for arm64 pcb register offsets

Add macros for offsets of macros we set in the arm64 pcb pcb_x array.
This will simplift reducing the size of this array in a later change.

Sponsored by: Arm Ltd

show more ...

Split out vfp_new_thread

To keep the vfp thread creation code in one place move into vfp.c. This
will also help with adding SVE support as it depends on VFP.

Reviewed by: markj
Sponsored by: The Fr

Split out vfp_new_thread

To keep the vfp thread creation code in one place move into vfp.c. This
will also help with adding SVE support as it depends on VFP.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35615

show more ...

Save only callee-saved registers in pcb

On AArch64, registers x9-x18 are not callee-saved, yet they are
preserved at many placed in swtch.S. This patch removes code that
preserves these registers.

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier an

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

show more ...

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi ha

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi has been tied to a "virtual memory" swi
(swi_vm). However, all of the swi_vm implementations are the same and
consist of checking a flag (busdma_swi_pending) which is always true
and if set calling busdma_swi. I suspect this dates back to the
pre-SMPng days and that the intention was for swi_vm to serve as a
mux. However, in the current scheme there's no need for the mux.

Instead, remove swi_vm and vm_ih. Each bus_dma implementation that
uses bounce pages is responsible for creating its own swi (busdma_ih)
which it now schedules directly. This swi invokes busdma_swi directly
removing the need for busdma_swi_pending.

One consequence is that the swi now works on RISC-V which had previously
failed to invoke busdma_swi from swi_vm.

Reviewed by: imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33447

show more ...

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is un

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

show more ...

arm64: Handle thumb2 thread entry point.

In cpu_set_upcall(), if the thread startup routine is a thumb routine, make
sure to set PSR_T, so that the CPU will run in thumb mode.

MFC After: 1 week

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsor

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477

show more ...

arm64: clear debug register state on fork

Following the analogous change for amd64 and i386 in 8223717ce62c,
ensure that new processes start with these registers inactive.

PR: 254661
Reported by:

arm64: clear debug register state on fork

Following the analogous change for amd64 and i386 in 8223717ce62c,
ensure that new processes start with these registers inactive.

PR: 254661
Reported by: Michał Górny
Reviewed by: kib, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29630

show more ...

Replace the arm64 initial_fpcr with a macro

This value was never changed from its default value. Replace it with a
macro.

Sponsored by: Innovate UK

Arch64: Clear VFP state on execve()

I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being

Arch64: Clear VFP state on execve()

I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being reset when starting a new process. This change resets the
VFP inside exec_setregs() to ensure no VFP register state is leaked from
parent processes to children.

This commit also moves the clearing of fpcr that was added in 65618fdda0f27
from fork() to execve() since that makes more sense: fork() can retain
current register values, but execve() should result in a well-defined
clean state.

Reviewed By: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29060

show more ...

AArch64: Don't set flush-subnormals-to-zero flag on startup

This flag has been set on startup since 65618fdda0f272a823e6701966421bdca0efa301.
However, This causes some of the math-related tests to f

AArch64: Don't set flush-subnormals-to-zero flag on startup

This flag has been set on startup since 65618fdda0f272a823e6701966421bdca0efa301.
However, This causes some of the math-related tests to fail as they report
zero instead of a tiny number. This fixes at least
/usr/tests/lib/msun/ldexp_test and possibly others.
Additionally, setting this flag prevents printf() from printing subnormal
numbers in decimal form.
See also https://www.openwall.com/lists/musl/2021/02/26/1

PR: 253847
Reviewed By: mmel
Differential Revision: https://reviews.freebsd.org/D28938

show more ...

arm64: optimize set_syscall_retval()

Microoptimize set_syscall_retval() for arm64 by predicting
the return value to be zero. This is similar to what has
been done for other architectures

Reviewed

arm64: optimize set_syscall_retval()

Microoptimize set_syscall_retval() for arm64 by predicting
the return value to be zero. This is similar to what has
been done for other architectures

Reviewed By: emaste, mhorne
Differential Revision: https://reviews.freebsd.org/D26991

show more ...

arm64: Initialize VFP control register.

The RW fields in this register reset to architecturally unknown values,
so initialize these to the proper rounding and denormal mode.
MFC after: 1 week

arm64: Clear FPU flags in the pcb in cpu_copy_thread().

New threads start off with clean FPU state instead of inheriting state
from the parent thread.

PR: 247945
Sponsored by: Netflix

arm64: Trim duplicate code from cpu_fork_kthread_handler().

cpu_fork_kthread_handler() is always called after either cpu_fork() or
cpu_copy_thread(). The arm64 version was duplicating some of the w

arm64: Trim duplicate code from cpu_fork_kthread_handler().

cpu_fork_kthread_handler() is always called after either cpu_fork() or
cpu_copy_thread(). The arm64 version was duplicating some of the work
already done by both of those functions.

Sponsored by: Netflix

show more ...

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus.

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus. Rather than writing to this field in
more places, it can be removed in favor of using the value in the link
register.

To make this clearer, pcb->pcb_x[30] is renamed to pcb->pcb_lr, similar
to what already exists in struct trapframe. Also, prefer lr to x30 in
assembly, as it better conveys intention.

This improves PC_REGS() for kdb_thread != curthread. It is required for
a functional gdb(4) stub, fixing the output of `info threads`, in
particular.

The space occupied by pcb_pc is retained, for compatibility with kgdb.

Reviewed by: markj, jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27720

show more ...