arm64: Add EL1 hardware breakpoint exceptions

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44353

arm64: Use a switch to decide when to enable debug

Use a switch statement to decide which exceptions we need to call
dbg_enable for. This simplifies adding more esceptions to the list in
the future.

arm64: Use a switch to decide when to enable debug

Use a switch statement to decide which exceptions we need to call
dbg_enable for. This simplifies adding more esceptions to the list in
the future.

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44352

show more ...

arm64: Make KMSAN aware of exceptions

- Call kmsan_intr_enter() when an exception occurs. This ensures that
code running in the exception context does not clobber thread-local
KMSAN state.
- En

arm64: Make KMSAN aware of exceptions

- Call kmsan_intr_enter() when an exception occurs. This ensures that
code running in the exception context does not clobber thread-local
KMSAN state.
- Ensure that stack memory containing trap frames is treated as
initialized.

Co-authored-by: Alexander Stetsenko <alex.stetsenko@klarasystems.com>
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43155

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

arm64: Raise a SIGBUS on a user external abort

When userspace triggers an external abort allow it to handle the abort
by raising a SIGBUS.

Reviewed by: markj, kib
Sponsored by: Arm Ltd
Differential

arm64: Raise a SIGBUS on a user external abort

When userspace triggers an external abort allow it to handle the abort
by raising a SIGBUS.

Reviewed by: markj, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42018

show more ...

syscalls: fix missing SIGSYS for several ENOSYS errors

In particular, when the syscall number is too large, or when syscall is
dynamic. For that, add nosys_sysent structure to pass fake sysent to
s

syscalls: fix missing SIGSYS for several ENOSYS errors

In particular, when the syscall number is too large, or when syscall is
dynamic. For that, add nosys_sysent structure to pass fake sysent to
syscall top code.

Reviewed by: dchagin, markj
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41976

show more ...

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential R

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450

show more ...

arm64: initialize pcb in the TBI/PAC/etc. fault case

After 2c10be9e06d, we may jump to the bad_far label without `pcb` being
set, resulting in a follow-up fault as we may dereference it immediately

arm64: initialize pcb in the TBI/PAC/etc. fault case

After 2c10be9e06d, we may jump to the bad_far label without `pcb` being
set, resulting in a follow-up fault as we may dereference it immediately
after the jump if td_intr_nesting_level == 0. In this branch, it should
be safe to dereference `td` as we're not handling the special case
mentioned below of accessing it during promotion/demotion.

This seems to fix a null ptr deref I hit during my most recent pkgbase
build attempt on the Windows DevKit, though that was admittedly
encountered while we were on the way to a panic from an apparent
use-after-free in ZFS bits.

Reviewed by: andrew, markj
Fixes: 2c10be9e06d ("arm64: Handle translation faults for thread [..]")
Differential Revision: https://reviews.freebsd.org/D41677

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64: improve KTR_TRAP trace entries

Follow-up of 7756232199ad ("riscv: improve KTR_TRAP trace entries").

Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
D

arm64: improve KTR_TRAP trace entries

Follow-up of 7756232199ad ("riscv: improve KTR_TRAP trace entries").

Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40696

show more ...

arm64: print '0x' before hex values

Not making it explicit that we're printing values in hex can be
misleading when the number doesn't contain hex-only symbols (a-f). A
good example of this is print

arm64: print '0x' before hex values

Not making it explicit that we're printing values in hex can be
misleading when the number doesn't contain hex-only symbols (a-f). A
good example of this is print_gp_register(), where we print "(func +
offset)"; if the offset doesn't contain a-f symbols, it's not
immediately clear if that value is in decimal or hex. Using '%#' instead
of '0x%' also isn't a better option, it doesn't print '0x' if the value
is 0, and it also messes up column alignment.

Reviewed by: imp, markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40583

show more ...

arm64: Print the spinlock count on panic

When the spinlock count is non-zero while taking a data abort we panic.
Print this count to help debugging.

Sponsored by: Arm Ltd

Print registers on an arm64 spinlock data abort

When checking if the spinlock count is correct in a data abort we can
panic without printing the registers. These are useful to debug the
abort, e.g.

Print registers on an arm64 spinlock data abort

When checking if the spinlock count is correct in a data abort we can
panic without printing the registers. These are useful to debug the
abort, e.g. by giving the fault address register.

Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40228

show more ...

Don't print leading 0's in the arm64 esr

We don't print leading zeros for other registers, so do the same with
ESR_EL1.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D401

Don't print leading 0's in the arm64 esr

We don't print leading zeros for other registers, so do the same with
ESR_EL1.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40129

show more ...

Handle the arm64 unknown exception separately

Rather than falling through to the default case handle the unknown
exception with its own panic message. As ESR_EL1 is zero for this
exception stop prin

Handle the arm64 unknown exception separately

Rather than falling through to the default case handle the unknown
exception with its own panic message. As ESR_EL1 is zero for this
exception stop printing it.

Sponsored by: Arm Ltd

show more ...

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow m

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

show more ...

arm64: add fault address to trapframe

It was previously possible for the fault address register to get
clobbered before it was saved. This small window occurred when an
additional exception was enco

arm64: add fault address to trapframe

It was previously possible for the fault address register to get
clobbered before it was saved. This small window occurred when an
additional exception was encountered inside the exception handler,
overwriting the previous value.

Commit f29942229d24 ("Read the arm64 far early in el0 exceptions")
patched this issue, but avoided changing the trapframe since this could
be considered a KBI change in FreeBSD 13.

Revert the above fix and save the fault address in the trapframe
instead. This saves the fault address even earlier in the exception
handling process, and is a more robust and simple fix.

Reviewed by: andrew, jhb, jrtc27
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38984

show more ...

arm64: extend ESR/SPSR registers to 64b

For the Exception Syndrome Register, ESR_ELx, the upper 32b were
previously unused, but now may contain additional exception info as of
Armv8.7 (FEAT_LS64).

arm64: extend ESR/SPSR registers to 64b

For the Exception Syndrome Register, ESR_ELx, the upper 32b were
previously unused, but now may contain additional exception info as of
Armv8.7 (FEAT_LS64).

Extend ESR from u32->u64 in exception handling code to support this. In
addition, also extend Saved Program Status Register SPSR_ELx in the same
way to allow for future extensions.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38983

show more ...

Read the arm64 far early in el0 exceptions

When handling userspace exceptions on arm64 we need to dereference the
current thread pointer. If this is being promoted/demoted there is a
small window wh

Read the arm64 far early in el0 exceptions

When handling userspace exceptions on arm64 we need to dereference the
current thread pointer. If this is being promoted/demoted there is a
small window where it will cause another exception to be hit. As this
second exception will set the fault address register we will read the
incorrect value in the userspace exception handler.

Fix this be always reading the fault address before dereferencing the
current thread pointer.

Reported by: olivier@
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38196

show more ...

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value.

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value. The value is
stored in td->td_md.md_canary; the sp_el0 register always contains a
pointer to that value, and certain functions selected by the compiler
will store the canary value on the stack as a part of the function
prologue (and will verify the copy as part of the epilogue). In
particular, the thread structure may be accessed.

This happens to occur in data_abort(), which leads to the same problem
addressed by commit 2c10be9e06d4 ("arm64: Handle translation faults for
thread structures"). This commit fixes that directly, by disabling SSP
in data_abort() and a couple of related functions by using a function
attribute. It also moves the update of sp_el0 out of C code in case
the compiler decides to start checking the canary in pmap_switch()
someday.

A different solution might be to move the canary value to the PCB, which
currently lives on the kernel stack and isn't subject to the same
problem as thread structures (if only because guard pages inhibit
superpage promotion). However, there isn't any particular reason the
PCB has to live on the stack today; on amd64 it is embedded in struct
thread, reintroducing the same problem. Keeping the reference canary
value at the top of the stack is also rather dubious since it could be
clobbered by a sufficiently large stack overflow.

A third solution could be to go back to the approach of commit
5aa5420ff2e8, and modify UMA to use the direct map for thread structures
even if KASAN is enabled. But, transient promotions and demotions in
the direct map are possible too.

Reviewed by: alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37255

show more ...

arm64: Handle translation faults for thread structures

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translati

arm64: Handle translation faults for thread structures

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translation fault while accessing curthread, and data_abort() accesses
the thread again before pmap_fault() can translate the address and
return.

Normally this isn't a problem because we have a hack to ensure that
slabs used by the thread zone are always accessed via the direct map,
where promotions and demotions are rare. However, this hack doesn't
work properly with UMA_MD_SMALL_ALLOC disabled, as is the case with
KASAN configured (since our KASAN implementation does not shadow the
direct map and so tries to force the use of the kernel map wherever
possible).

Fix the problem by modifying data_abort() to handle translation faults
in the kernel map without dereferencing "td", i.e., curthread, and
without enabling interrupts. pmap_klookup() has special handling for
translation faults which makes it safe to call in this context. Then,
revert the aforementioned hack.

Reviewed by: kevans, alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37231

show more ...

Trap SVE instructions until we have SVE support

When running on hardware that supports SVE send the correct signal when
an SVE instruction is run.

Sponsored by: The FreeBSD Foundation

arm64: Enable the floating-point exception traps

To enable it user-space needs to call feenableexcept().

FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.

Reviewed

arm64: Enable the floating-point exception traps

To enable it user-space needs to call feenableexcept().

FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.

Reviewed by: andrew@
Differential revision: https://reviews.freebsd.org/D35247
MFC after: 2 weeks

show more ...

syscallarg_t: Add a type for system call arguments

This more clearly differentiates system call arguments from integer
registers and return values. On current architectures it has no effect,
but on

syscallarg_t: Add a type for system call arguments

This more clearly differentiates system call arguments from integer
registers and return values. On current architectures it has no effect,
but on architectures where pointers are not integers (CHERI) and may
not even share registers (CHERI-MIPS) it is necessiary to differentiate
between system call arguments (syscallarg_t) and integer register values
(register_t).

Obtained from: CheriBSD

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D33780

show more ...

Treat cache write as a read in arm64 data faults

On arm64 we can ask the hardware to perform cache operations from
userspace. These require read permission however when the memory is
unmapped the ke

Treat cache write as a read in arm64 data faults

On arm64 we can ask the hardware to perform cache operations from
userspace. These require read permission however when the memory is
unmapped the kernel will receive a write exception. Add a check to
see if the cause of the exception is from the cache and pass a memory
read fault type to the vm subsystem.

PR: 262836
Reported by: dch
Sponsored by: The FreeBSD Foundation

show more ...