sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

Adjust function definition in moea64_native.c to avoid clang 15 warning

With clang 15, the following -Werror warning is produced:

sys/powerpc/aim/moea64_native.c:306:22: error: a function decla

Adjust function definition in moea64_native.c to avoid clang 15 warning

With clang 15, the following -Werror warning is produced:

sys/powerpc/aim/moea64_native.c:306:22: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
moea64_install_native()
^
void

This is because moea64_install_native() is declared with a (void)
argument list, but defined with an empty argument list. Make the
definition match the declaration.

MFC after: 3 days

show more ...

ppc mmoea64: Workaround set but unused warnings for RB_NEXT.

Inline the unused expression passed as the second argument to RB_NEXT.

powerpc64: Split out DMAP and non-DMAP implementations of some methods

Summary:
Some methods are split between DMAP and non-DMAP, conditional on
hw_direct_map variable. Rather than checking this va

powerpc64: Split out DMAP and non-DMAP implementations of some methods

Summary:
Some methods are split between DMAP and non-DMAP, conditional on
hw_direct_map variable. Rather than checking this variable every time,
use it to install different functions via IFUNCs.

Reviewed By: luporl
Differential Revision: https://reviews.freebsd.org/D30071

show more ...

SPDX: Spell 4 clause BSD license correctly

[PowerPC] Fix powerpc64le boot after HPT superpages addition

The HPT is always stored in big-endian, as it is accessed directly by the
hardware as well as the kernel. As such, it is necessary to con

[PowerPC] Fix powerpc64le boot after HPT superpages addition

The HPT is always stored in big-endian, as it is accessed directly by the
hardware as well as the kernel. As such, it is necessary to convert values
to and from native endian when running on LE.

Some unconverted accesses snuck in accidentally with r367417.

Apply the appropriate conversions to fix boot hanging on powerpc64le.

Sponsored by: Tag1 Consulting, Inc.

show more ...

Fix powerpc and LINT builds

Fix build errors introduced by r367417 and r367390:

- Guard label reached only by powerpc64
- Guard vm_reserv_level_iffullpop call, that is not defined on powerpc
vari

Fix powerpc and LINT builds

Fix build errors introduced by r367417 and r367390:

- Guard label reached only by powerpc64
- Guard vm_reserv_level_iffullpop call, that is not defined on powerpc
variants that don't support superpages
- Add missing hwpmc file, for when hwpmc is built into kernel

show more ...

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes wer

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes were inspired by RISC-V implementation of superpages,
by @markj (r344106), but heavily adapted to fit PPC64 HPT architecture
and existing MMU OEA64 code.

While these changes are not better tested, superpages support is disabled by
default. To enable it, use vm.pmap.superpages_enabled=1.

In this initial implementation, when superpages are disabled, system
performance stays at the same level as without these changes. When
superpages are enabled, buildworld time increases a bit (~2%). However,
for workloads that put a heavy pressure on the TLB the performance boost
is much bigger (see HPC Challenge and pgbench on D25237).

Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D25237

show more ...

[PowerPC64LE] Implement endian-independent dword atomic PTE lock.

It's much easier to implement this in an endian-independent way when we
don't also have to worry about masking half of the dword off

[PowerPC64LE] Implement endian-independent dword atomic PTE lock.

It's much easier to implement this in an endian-independent way when we
don't also have to worry about masking half of the dword off.

Given that this code ran on a machine that ran a poudriere bulk with no
kernel oddities, I am relatively certain it is correctly implemented. ;)

This should be a minor performance boost on BE as well.

Sponsored by: Tag1 Consulting, Inc.

show more ...

[PowerPC64LE] Fix endian conversion bugs in moea64.

For a body of code that had its endian conversion bits written blind without
the ability to test, moea64 was VERY close to being correct.

There w

[PowerPC64LE] Fix endian conversion bugs in moea64.

For a body of code that had its endian conversion bits written blind without
the ability to test, moea64 was VERY close to being correct.

There were only four instances where the existing code was getting it wrong.

Sponsored by: Tag1 Consulting, Inc.

show more ...

[PowerPC64LE] Set up the powernv partition table correctly.

The partition table is always big endian.

Sponsored by: Tag1 Consulting, Inc.

powerpc: clean up empty lines in .c and .h files

powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobj

With IFUNC support in the kernel, we can finally get rid of our poor-man's
ifunc for pmap, utilizing kobj. Since moea64 uses a second ti

powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobj

With IFUNC support in the kernel, we can finally get rid of our poor-man's
ifunc for pmap, utilizing kobj. Since moea64 uses a second tier kobj as
well, for its own private methods, this adds a second pmap install function
(pmap_mmu_init()) to perform pmap 'post-install pre-bootstrap'
initialization, before the IFUNCs get initialized.

Reviewed by: bdragon

show more ...

powerpc64/pmap: micro-optimize some PVO-PTE logic

Summary:
moea64_pte_sync_native() and moea64_pte_unset_native() don't need the
full PTE created, they only need to check that the PVO has a matching

powerpc64/pmap: micro-optimize some PVO-PTE logic

Summary:
moea64_pte_sync_native() and moea64_pte_unset_native() don't need the
full PTE created, they only need to check that the PVO has a matching
PTE to the PTE in the page table. Don't waste time creating the full
PTE in this case.

Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D22341

show more ...

powerpc64: Don't guard ISA 3.0 partition table setup with hw_direct_map

PowerISA 3.0 eliminated the 64-bit bridge mode which allowed 32-bit kernels
to run on 64-bit AIM/Book-S hardware. Since there

powerpc64: Don't guard ISA 3.0 partition table setup with hw_direct_map

PowerISA 3.0 eliminated the 64-bit bridge mode which allowed 32-bit kernels
to run on 64-bit AIM/Book-S hardware. Since therefore only a 64-bit kernel
can run on this hardware, and 64-bit native always has the direct map, there
is no need to guard it.

show more ...

[PPC64] Add minidump support to PowerNV

Implementation of PowerNV specific minidump code.

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D21643

powerpc64/pmap: Simplify the code path for moea64_pte_replace_native()

Summary:
MOEA64_PTE_REPLACE() is called often with the pmap lock held, and
sometimes with the page pv lock held. The less work

powerpc64/pmap: Simplify the code path for moea64_pte_replace_native()

Summary:
MOEA64_PTE_REPLACE() is called often with the pmap lock held, and
sometimes with the page pv lock held. The less work done while holding
a lock, the better. Since we are intending to replace the same PTE
(same hash index), we don't need to recalculate anything, just flat
replace the PTE. This cuts more than 200 instructions off the
invalidating code path. In addition, we don't need to replace a PTE
that's not occupied by this PVO.

Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D21515

show more ...

powerpc/pmap64: Make moea64 statistics optional

Summary:
It turns out statistics accounting is very expensive in the pmap driver,
and doesn't seem necessary in the common case. Make this optional
b

powerpc/pmap64: Make moea64 statistics optional

Summary:
It turns out statistics accounting is very expensive in the pmap driver,
and doesn't seem necessary in the common case. Make this optional
behind a MOEA64_STATS #define, which one can set if they really need
statistics.

This saves ~7-8% on buildworld time on a POWER9.

Found by bdragon.

Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D20903

show more ...

powerpc/moea: Fix moea64 native VA invalidation

Summary:
moea64_insert_pteg_native()'s invalidation only works by happenstance.
The purpose of the shifts and XORs is to extract the VSID in order to

powerpc/moea: Fix moea64 native VA invalidation

Summary:
moea64_insert_pteg_native()'s invalidation only works by happenstance.
The purpose of the shifts and XORs is to extract the VSID in order to
reverse-engineer the lower bits of the VPN. Currently a segment size is 256MB
(2**28), and ADDR_API_SHFT64 is 16, so ADDR_PIDX_SHIFT is equivalent. However,
it's semantically incorrect, in that we don't want to shift by the page shift
size, we want to shift to get to the VSID.

Tested by: bdragon
Differential Revision: https://reviews.freebsd.org/D20467

show more ...

powerpc64/pmap: Reapply r334235 to OEA64 pmap, clearing HID0_RADIX

This was lost in the re-merger of ISA3 MMU into moea64_native.

powerpc64: Micro-optimize moea64 native pmap tlbie

* Cache moea64_need_lock in a local variable; gcc generates slightly better
code this way, it doesn't need to reload the value from memory each r

powerpc64: Micro-optimize moea64 native pmap tlbie

* Cache moea64_need_lock in a local variable; gcc generates slightly better
code this way, it doesn't need to reload the value from memory each read.
* VPN cropping is only needed on PowerPC ISA 2.02 and older cores, a subset
of those that need serialization, so move this under the need_lock check,
so those that don't need the lock don't even need to check this.

show more ...

powerpc: Re-merge isa3 HPT with moea64 native HPT

r345402 fixed the bug that led to the split of the ISA 3.0 HPT handling from
the existing manager. The cause of the bug was gcc moving the register

powerpc: Re-merge isa3 HPT with moea64 native HPT

r345402 fixed the bug that led to the split of the ISA 3.0 HPT handling from
the existing manager. The cause of the bug was gcc moving the register
holding VPN to a different register (not r0), which triggered bizarre
behaviors. With the fix, things work, so they can be re-merged. No
performance lost with the merge.

show more ...

powerpc64: Handle the modern (2.05+) implementaiton of tlbie

By happenstance gcc4 puts 'vpn' into r0 in all uses of TLBIE(), but modern
gcc does not. Also, the single-argument form of tlbie zeros a

powerpc64: Handle the modern (2.05+) implementaiton of tlbie

By happenstance gcc4 puts 'vpn' into r0 in all uses of TLBIE(), but modern
gcc does not. Also, the single-argument form of tlbie zeros all unused
arguments, making the modern tlbie instruction use r0 as the RS field
(LPID).

The vpn argument has the bottom 12 bits cleared (the input having been
left-shifted by 12 bits), which just so happens, on the POWER9 and previous
incarnations, to be the number of LPID bits supported. With those bits
being zero, the instruction:

tlbie r0, r0

will invalidate the VPN in r0, in LPAR 0 (ignoring the upper bits of r0 for
the RS field). One build with gcc8 yields:

tlbie r9, r0

with r0 having arbitrary contents, not equal to r9. This leads to strange
crashes, behaviors, and panics, due to the requested TLB entry not actually
being invalidated.

As the moea64_native must work on both old and new, we explicitly zero out
r0 so that it can work with only the single argument, built with base gcc
and modern gcc. isa3_hashtb takes a different approach, encoding the
two-argument form, soas not to explicitly clobber r0, and instead let the
compiler decide.

Reported by: Brandon Bergren
Tested by: Brandon Bergren
MFC after: 1 week

show more ...

Split the PowerISA 3.0 HPT implementation from historic

PowerISA 3.0 makes several changes to not only the format of the HPT but
also the behavior surrounding it. For instance, TLBIE no longer requ

Split the PowerISA 3.0 HPT implementation from historic

PowerISA 3.0 makes several changes to not only the format of the HPT but
also the behavior surrounding it. For instance, TLBIE no longer requires
serialization. Removing this lock cuts buildworld time in half on a
18-core/72-thread POWER9 system, demonstrating that this lock is highly
contended on such a system.

There was odd behavior observed trying to make this change in a
backwards-compatible manner in moea64_native.c, so the best option was to
fully split it, and largely revert the original changes adding POWER9
support to the original file.

Suggested by: nwhitehorn

show more ...