pmap: Convert boolean_t to bool.

Reviewed by: kib (older version)
Differential Revision: https://reviews.freebsd.org/D39921

powerpc_pmap: include proper vm_radix file

Like the pmap.h files for amd64, arm64, i386 and riscv, the one for powerpc
should #include vm/_vm_radix.h, not vm/vm_radix.h, to get a definition for
stru

powerpc_pmap: include proper vm_radix file

Like the pmap.h files for amd64, arm64, i386 and riscv, the one for powerpc
should #include vm/_vm_radix.h, not vm/vm_radix.h, to get a definition for
struct vm_radix.

Fixes a problem introduced with 429c871ddddac4bbf6abf1eb9e2e6603f87c2ef5.

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a va

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a variant of pmap_remove(), pmap_map_delete(), to
allow pmap to distinguish between real removes of the UVA mappings
and any other internal removes, e.g. swapout.

For non-amd64, pmap_map_delete() is stubbed by define to pmap_remove().

Reported by: andrew
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39556

show more ...

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.f

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548

show more ...

Define _NPCM and the last PC_FREEn constant in terms of _NPCPV.

This applies one of the changes from
5567d6b4419b02a2099527228b1a51cc55a5b47d to other architectures
besides arm64.

Reviewed by: kib

Define _NPCM and the last PC_FREEn constant in terms of _NPCPV.

This applies one of the changes from
5567d6b4419b02a2099527228b1a51cc55a5b47d to other architectures
besides arm64.

Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36263

show more ...

powerpc64: fix OFWFB with Radix MMU

Current implementation of Radix MMU doesn't support mapping
arbitrary virtual addresses, such as the ones generated by
"direct mapping" I/O addresses. This caused

powerpc64: fix OFWFB with Radix MMU

Current implementation of Radix MMU doesn't support mapping
arbitrary virtual addresses, such as the ones generated by
"direct mapping" I/O addresses. This caused the system to hang, when
early I/O addresses, such as those used by OpenFirmware Frame Buffer,
were remapped after the MMU was up.

To avoid having to modify mmu_radix_kenter_attr just to support this
use case, this change makes early I/O map use virtual addresses from
KVA area instead (similar to what mmu_radix_mapdev_attr does), as
these can be safely remapped later.

Reviewed by: alfredo (earlier version), jhibbits (in irc)
MFC after: 2 weeks
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D31232

show more ...

Add pmap_vm_page_alloc_check()

which is the place to put MD asserts about allocated pages.

On amd64, verify that allocated page does not belong to the kernel
(text, data) or early allocated pages.

Add pmap_vm_page_alloc_check()

which is the place to put MD asserts about allocated pages.

On amd64, verify that allocated page does not belong to the kernel
(text, data) or early allocated pages.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31121

show more ...

[PowerPC] Don't overwrite vm.pmap sysctl node

After r367417, both mmu_oea64 and mmu_radix were defining the vm.pmap
sysctl node, resulting in the later definition hiding the properties of
the previo

[PowerPC] Don't overwrite vm.pmap sysctl node

After r367417, both mmu_oea64 and mmu_radix were defining the vm.pmap
sysctl node, resulting in the later definition hiding the properties of
the previous one. Avoid this issue by defining vm.pmap in a common
source file and declaring it where needed.

This change also standardizes the tunable name used to enable superpages
and change its default to disabled on radix MMU, because it still has some
issues with superpages.

Reviewed by: bdragon, jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D27156

show more ...

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes wer

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes were inspired by RISC-V implementation of superpages,
by @markj (r344106), but heavily adapted to fit PPC64 HPT architecture
and existing MMU OEA64 code.

While these changes are not better tested, superpages support is disabled by
default. To enable it, use vm.pmap.superpages_enabled=1.

In this initial implementation, when superpages are disabled, system
performance stays at the same level as without these changes. When
superpages are enabled, buildworld time increases a bit (~2%). However,
for workloads that put a heavy pressure on the TLB the performance boost
is much bigger (see HPC Challenge and pgbench on D25237).

Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D25237

show more ...

powerpc: clean up empty lines in .c and .h files

powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobj

With IFUNC support in the kernel, we can finally get rid of our poor-man's
ifunc for pmap, utilizing kobj. Since moea64 uses a second ti

powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobj

With IFUNC support in the kernel, we can finally get rid of our poor-man's
ifunc for pmap, utilizing kobj. Since moea64 uses a second tier kobj as
well, for its own private methods, this adds a second pmap install function
(pmap_mmu_init()) to perform pmap 'post-install pre-bootstrap'
initialization, before the IFUNCs get initialized.

Reviewed by: bdragon

show more ...

powerpc: Handle machine checks caused by D-ERAT multihit

Instead of crashing the user process when a D-ERAT multihit is detected, try
to flush the ERAT, and continue. This machine check indicates a

powerpc: Handle machine checks caused by D-ERAT multihit

Instead of crashing the user process when a D-ERAT multihit is detected, try
to flush the ERAT, and continue. This machine check indicates a likely PMAP
invalidation shortcoming that will need to be addressed, but it's
recoverable, so just recover. The recovery is pmap-specific to flush the
ERAT, so add a pmap function to do so, currently only implemented by the
POWER9 radix pmap.

show more ...

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures. The
PowerISA also specifies a process table -- a table of page table pointers--
which on the POWER9 is only available with the Radix MMU, so we can take
advantage of it with the Radix MMU driver.

Written by Matt Macy.

Differential Revision: https://reviews.freebsd.org/D19516

show more ...

powerpc/booke: Change Book-E 64-bit pmap to 4-level table

Summary:
The existing page table is fraught with errors, since it creates a hole
in the address space bits. Fix this by taking a cue from t

powerpc/booke: Change Book-E 64-bit pmap to 4-level table

Summary:
The existing page table is fraught with errors, since it creates a hole
in the address space bits. Fix this by taking a cue from the POWER9
radix pmap, and make the page table 4 levels, 52 bits.

Reviewed by: bdragon
Differential Revision: https://reviews.freebsd.org/D24220

show more ...

Introduce vm_page_astate.

This is a 32-bit structure embedded in each vm_page, consisting mostly
of page queue state. The use of a structure makes it easy to store a
snapshot of a page's queue stat

Introduce vm_page_astate.

This is a 32-bit structure embedded in each vm_page, consisting mostly
of page queue state. The use of a structure makes it easy to store a
snapshot of a page's queue state in a stack variable and use cmpset
loops to update that state without requiring the page lock.

This change merely adds the structure and updates references to atomic
state fields. No functional change intended.

Reviewed by: alc, jeff, kib
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D22650

show more ...

powerpc/pmap: NUMA-ize vm_page_array on powerpc

Summary:
This matches r351198 from amd64. This only applies to AIM64 and Book-E.
On AIM64 it short-circuits with one domain, to behave similar to
exi

powerpc/pmap: NUMA-ize vm_page_array on powerpc

Summary:
This matches r351198 from amd64. This only applies to AIM64 and Book-E.
On AIM64 it short-circuits with one domain, to behave similar to
existing. Otherwise it will allocate 16MB huge pages to hold the page
array, across all NUMA domains. On the first domain it will shift the
page array base up, to "upper-align" the page array in that domain, so
as to reduce the number of pages from the next domain appearing in this
domain. After the first domain, subsequent domains will be allocated in
full 16MB pages, until the final domain, which can be short. This means
some inner domains may have pages accounted in earlier domains.

On Book-E the page array is setup at MMU bootstrap time so that it's
always mapped in TLB1, on both 32-bit and 64-bit. This reduces the TLB0
overhead for touching the vm_page_array, which reduces up to one TLB
miss per array access.

Since page_range (vm_page_startup()) is no longer used on Book-E but is on
32-bit AIM, mark the variable as potentially unused, rather than using a
nasty #if defined() list.

Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D21449

show more ...

Fix assert in PowerPC pmaps after introduction of object busy.

The VM_PAGE_OBJECT_BUSY_ASSERT() in pmap_enter() implementation should
be only asserted when the code is executed as result of pmap_ent

Fix assert in PowerPC pmaps after introduction of object busy.

The VM_PAGE_OBJECT_BUSY_ASSERT() in pmap_enter() implementation should
be only asserted when the code is executed as result of pmap_enter(),
not when the same code is entered from e.g. pmap_enter_quick(). This
is relevant for all PowerPC pmap variants, because mmu_*_enter() is
used as the backend, and assert is located there.

Add a PowerPC private pmap_enter() PMAP_ENTER_QUICK_LOCKED flag to
indicate that the call is not from pmap_enter(). For non-quick-locked
calls, assert that the object is locked.

Reported and tested by: bdragon
Reviewed by: alc, bdragon, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D22041

show more ...

[PPC64] Initial kernel minidump implementation

Based on POWER9BSD implementation, with all POWER9 specific code removed and
addition of new methods in PPC64 MMU interface, to isolate platform specif

[PPC64] Initial kernel minidump implementation

Based on POWER9BSD implementation, with all POWER9 specific code removed and
addition of new methods in PPC64 MMU interface, to isolate platform specific
code. Currently, the new methods are implemented on pseries and PowerNV
(D21643).

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D21551

show more ...

Move phys_avail definition into MI code. It is consumed in the MI layer and
doing so adds more flexibility with less redundant code.

Reviewed by: jhb, markj, kib
Sponsored by: Netflix
Differential

Move phys_avail definition into MI code. It is consumed in the MI layer and
doing so adds more flexibility with less redundant code.

Reviewed by: jhb, markj, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21250

show more ...

powerpc: Unify pmap definitions between AIM and Book-E

This is part 2 of r347078, pulling the page directory out of the Book-E
pmap. This breaks KBI for anything that uses struct pmap (such as vm_m

powerpc: Unify pmap definitions between AIM and Book-E

This is part 2 of r347078, pulling the page directory out of the Book-E
pmap. This breaks KBI for anything that uses struct pmap (such as vm_map)
so any modules that access this must be rebuilt.

show more ...

powerpc64/mmu: Use a SLIST for the PVO delete list, instead of a RB_TREE

Summary:
Although it's convenient to reuse the pvo_plist for deletion, RB_TREE
insertion and removal is not free, and can res

powerpc64/mmu: Use a SLIST for the PVO delete list, instead of a RB_TREE

Summary:
Although it's convenient to reuse the pvo_plist for deletion, RB_TREE
insertion and removal is not free, and can result in a lot of extra work
to rebalance the tree. Instead, use a SLIST as a LIFO delete queue,
which gives us almost free insertion, deletion, and traversal.

Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D21061

show more ...

powerpc: Merge all pmap struct definitions

Summary:
A few ports fail to build due to missing pmap-related definitions, which are
specific per-pmap type. This tries to appease those ports, by mergin

powerpc: Merge all pmap struct definitions

Summary:
A few ports fail to build due to missing pmap-related definitions, which are
specific per-pmap type. This tries to appease those ports, by merging all
pmaps together.

A future change will move the inline page directory out of the Book-E pmap,
to eliminate the last #ifdefs in pmap.h and complete the merge.

Reviewed By: luporl
Differential Revision: https://reviews.freebsd.org/D20119

show more ...

Add kernel support for Intel userspace protection keys feature on
Skylake Xeons.

See SDM rev. 68 Vol 3 4.6.2 Protection Keys and the description of the
RDPKRU and WRPKRU instructions.

Reviewed by:

Add kernel support for Intel userspace protection keys feature on
Skylake Xeons.

See SDM rev. 68 Vol 3 4.6.2 Protection Keys and the description of the
RDPKRU and WRPKRU instructions.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D18893

show more ...

Revert r336509. Fails buildworld.

I had naively assumed that building kernel would be sufficient to test that
the header is sane. However, it turns out this now needs -fms-extensions to
build. Ra

Revert r336509. Fails buildworld.

I had naively assumed that building kernel would be sufficient to test that
the header is sane. However, it turns out this now needs -fms-extensions to
build. Rather than sprinkling -fms-extensions all over the place, revert
for now, and revisit with a better fix.

show more ...