arm64: Support BTI checking in most of the kernel

LLD has the -zbti-report=error argument to check if the BTI note is
present when linking. To allow for this to be used when linking the
kernel and m

arm64: Support BTI checking in most of the kernel

LLD has the -zbti-report=error argument to check if the BTI note is
present when linking. To allow for this to be used when linking the
kernel and modules:
- Add the BTI note to the remaining assembly files
- Mark ptrauth.c as protected by BTI
- Disable -zbti-report for vmm hypervisor switching code as it's not
used there.

The linux64 module doesn't build with the flag as it includes vdso code
that doesn't include the note.

Reviewed by: imp, kib, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45466

show more ...

arm64: map kernel using large pages when page size is 16K

When the page size is 16K, use ATTR_CONTIGUOUS to map the kernel code
and data sections using 2M pages. Previously, they were mapped using

arm64: map kernel using large pages when page size is 16K

When the page size is 16K, use ATTR_CONTIGUOUS to map the kernel code
and data sections using 2M pages. Previously, they were mapped using
16K pages.

Reviewed by: markj
Tested by: markj
Differential Revision: https://reviews.freebsd.org/D45162

show more ...

arm64: Update the page table list in locore

The comment describing the page tables was out of date. Update it with
the current list.

Sponsored by: Arm Ltd

arm64: Merge common page table creation code

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45061

arm64: Support a shared release for spin-table

When releasing multiple CPUs that share a release address we need them
to wait for their turn to boot. Add a mechanism to do this by booting
them until

arm64: Support a shared release for spin-table

When releasing multiple CPUs that share a release address we need them
to wait for their turn to boot. Add a mechanism to do this by booting
them until they enable the TLB before waiting their turn to enter
init_secondary.

Reviewed by: jhibbits, kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45082

show more ...

arm64: Rename drop_to_el1 to enter_kernel_el

In the future we may not drop to EL1, e.g. when we support FEAT_VHE
where the kernel runs in EL2.

Reviewed by: emaste, imp
Sponsored by: Arm Ltd
Differe

arm64: Rename drop_to_el1 to enter_kernel_el

In the future we may not drop to EL1, e.g. when we support FEAT_VHE
where the kernel runs in EL2.

Reviewed by: emaste, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43976

show more ...

arm64: Fix typo in pagetable_l0_ttbr0_bootstrap symbol name

Obtained from: CheriBSD

arm64: Use the new CurrentEL macros

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43972

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreove

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403

show more ...

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a l

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a location without a correct landing pad instruction to
raise an exception.

This should help mitigate some attacks where a function pointer is
changed to point somewhere other than the start of the function,
however it doesn't stop an attacker pointing it to an unintended
function.

Reviewed by: alc, scottph (both earlier version), markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42080

show more ...

arm64: Remove an old comment

This was missed when removing kern_delta

Sponsored by: Arm Ltd

arm64: Correct a comment in locore.S

We now use the physical address of get_load_phys_addr. Use it in a
comment rather than the old symbol.

Sponsored by: Arm Ltd

arm64: Clean up finding our load address

Use the linker to pre-calculate the offset of a known symbol from
KERNBASE, and use this to find the physical address KERNBASE should
map to.

Sponsored by:

arm64: Clean up finding our load address

Use the linker to pre-calculate the offset of a known symbol from
KERNBASE, and use this to find the physical address KERNBASE should
map to.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42568

show more ...

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.

While here remove an unneeded printf that indirectly used it or was
related to the previous printf.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567

show more ...

arm64: Use adrp + :lo12: to load globals from asm

When loading a global variable we can use a pseudo-instruction similar
to "ldr, xn, =global" to load the address of the symbol. As this is
unlikely

arm64: Use adrp + :lo12: to load globals from asm

When loading a global variable we can use a pseudo-instruction similar
to "ldr, xn, =global" to load the address of the symbol. As this is
unlikely to be supported by a mov instruction a pc-relative load is
used, with the absolute address written at the end of the function so
it will be loaded.

This load can be partially replaced with an adrp instruction. This
generates the address, aligned to a 4k boundary, using a pc-relative
addition. Because the address is 4k-aligned we then update reading the
global variable using a load with the offset of the load the low
12-bits of the global. Arm64 assemblers have :lo12: to support this,
e.g. "ldr xn, [xn, :lo12:global]".

The only remaining users of "ldr, xn, =global" that I can find are
executed from the physical address space the kernel was loaded in and
need an address in the kernels virtual address space. Because of this
they can't use adrp.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42565

show more ...

arm64: Initialize x18 for APs earlier during boot

When KMSAN is configured, the instrumentation inserts calls to
__msan_get_context_state() into all function prologues. The
implementation dereferen

arm64: Initialize x18 for APs earlier during boot

When KMSAN is configured, the instrumentation inserts calls to
__msan_get_context_state() into all function prologues. The
implementation dereferences curthread and thus assumes that x18 points
to the PCPU area. This applies in particular to init_secondary(), which
currently is responsible for initializing x18 for APs.

Move initialization into locore to avoid this problem. No functional
change intended.

Reviewed by: kib, andrew
MFC after: 2 weeks
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D42533

show more ...

arm64: Save an instruction in locore.S

We can move and sub at the same time, so let's do that.

Reviewed by: andrew, kevans
Pull Request: https://github.com/freebsd/freebsd-src/pull/794

arm64 locore.S: fix typos

Reviewed by: jhb, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42143

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pa

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.

There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction

Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.

There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.

Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078

show more ...

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential R

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64: use PSR_DAIF instead of each individual flag

No functional change intended.

Reviewed by: mhorne, andrew
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D40165

Move the arm64 sigcode to .rodata

The kernel doesn't execute this code, it's only ever copied to
userspace. Move it to .rodata as we don't need to modify it.

Sponsored by: Arm Ltd
Differential Revi

Move the arm64 sigcode to .rodata

The kernel doesn't execute this code, it's only ever copied to
userspace. Move it to .rodata as we don't need to modify it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39399

show more ...

arm64: Move the initial kernel stack out of the init_pagetables section

init_pagetables is mapped into the segment containing the BSS, but does
not get zeroed by locore. It is used for bootstrap pa

arm64: Move the initial kernel stack out of the init_pagetables section

init_pagetables is mapped into the segment containing the BSS, but does
not get zeroed by locore. It is used for bootstrap page table pages.

It happens that the bootstrap kernel stack is also placed in that
section, but there's no reason it shouldn't live in the BSS, so move it
there. No functional change intended.

Reviewed by: andrew
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D39367

show more ...

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow m

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

show more ...