Create sys/reg.h for the common code previously in machine/reg.h

Move the common kernel function signatures from machine/reg.h to a new
sys/reg.h. This is in preperation for adding PT_GETREGSET to p

Create sys/reg.h for the common code previously in machine/reg.h

Move the common kernel function signatures from machine/reg.h to a new
sys/reg.h. This is in preperation for adding PT_GETREGSET to ptrace(2).

Reviewed by: imp, markj
Sponsored by: DARPA, AFRL (original work)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D19830

show more ...

powerpc/aim: Update timebase directly on resume instead of through platform

This only works on single-CPU G4 systems, and more work is needed for
dual-CPU systems. That said, platform sleep does no

powerpc/aim: Update timebase directly on resume instead of through platform

This only works on single-CPU G4 systems, and more work is needed for
dual-CPU systems. That said, platform sleep does not work, and this is
currently only used for PMU-based CPU speed change.

The elimination of the platform_smp_timebase_sync() call is so that the
timebase sync rendezvous can be enhanced to perform better
synchronization, which requires a full rendezvous. This would be
impossible to do on this single-threaded run.

show more ...

powerpc/powermac: Constrain 'cpu_sleep()' for AIM to mpc745x

Rename cpu_sleep() to mpc745x_sleep() to denote what it's actually
intended for. This function is very G4-specific, and will not work on

powerpc/powermac: Constrain 'cpu_sleep()' for AIM to mpc745x

Rename cpu_sleep() to mpc745x_sleep() to denote what it's actually
intended for. This function is very G4-specific, and will not work on
any other CPU. This will afterward eliminate a
platform_smp_timebase_sync() call by directly updating the timebase
instead.

show more ...

powerpc/aim: Add machine check handler for TLB multihit

Handle TLB multi-hit the same as ERAT multi-hit, by flushing the full
TLB.

[PowerPC64LE] LE bringup work: locore / machdep / platform

This is the initial LE changes required in the machdep code to get as far
as platform attachment on qemu pseries.

Sponsored by: Tag1 Consu

[PowerPC64LE] LE bringup work: locore / machdep / platform

This is the initial LE changes required in the machdep code to get as far
as platform attachment on qemu pseries.

Sponsored by: Tag1 Consulting, Inc.

show more ...

[PowerPC64] Hide dssall instruction from llvm assembler

When doing a build for a modern CPUTYPE, llvm will throw errors if obsolete
instructions are used, even if they will never run due to runtime

[PowerPC64] Hide dssall instruction from llvm assembler

When doing a build for a modern CPUTYPE, llvm will throw errors if obsolete
instructions are used, even if they will never run due to runtime checks.

Hiding the dssall instruction from the assembler fixes kernel build when
overriding CPUTYPE, without having any effect on the generated binary.

This has been in my local tree for over a year and is well tested across
a variety of machines.

Sponsored by: Tag1 Consulting, Inc.

show more ...

powerpc: clean up empty lines in .c and .h files

[PowerPC] Fix translation-related crashes during startup

After spending a lot of time trying to track down what was going on, I have
isolated the "black screen" failures when using boot1 to boot a G

[PowerPC] Fix translation-related crashes during startup

After spending a lot of time trying to track down what was going on, I have
isolated the "black screen" failures when using boot1 to boot a G4 machine.

It turns out we were replacing the traps before installing the temporary
BAT entry for the bottom of physical memory. That meant that until the MMU
was bootstrapped, the cached translations were the only thing keeping us
from losing.

Throwing boot1 into the mix was affecting execution flow enough to cause us
to hit an uncached page and crash.

Fix this by properly setting up the initial BAT entry at the same time we
are replacing the OpenFirmware traps, so we can continue executing in
segment 0 until the rest of the DMAP has been set up.

A second thing discovered while researching this is that we were entering a
BAT region for segment 16. It turns out this range was a) considered part
of KVA, and b) has firmware mappings with varying attributes.

If we ever accessed an unmapped page in segment 16, it would cause a BAT
entry to be installed for the whole segment, which would bypass the
existing mappings until it was flushed out again.

Instead, translate the OFW memory attributes into VM memory attributes and
install the ranges into the kernel address space properly.

Reviewed by: adalava
MFC after: 3 weeks
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D25547

show more ...

powerpc: Handle machine checks caused by D-ERAT multihit

Instead of crashing the user process when a D-ERAT multihit is detected, try
to flush the ERAT, and continue. This machine check indicates a

powerpc: Handle machine checks caused by D-ERAT multihit

Instead of crashing the user process when a D-ERAT multihit is detected, try
to flush the ERAT, and continue. This machine check indicates a likely PMAP
invalidation shortcoming that will need to be addressed, but it's
recoverable, so just recover. The recovery is pmap-specific to flush the
ERAT, so add a pmap function to do so, currently only implemented by the
POWER9 radix pmap.

show more ...

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures. The
PowerISA also specifies a process table -- a table of page table pointers--
which on the POWER9 is only available with the Radix MMU, so we can take
advantage of it with the Radix MMU driver.

Written by Matt Macy.

Differential Revision: https://reviews.freebsd.org/D19516

show more ...

powerpc: Add a CPU-custom machine check handler

Summary:
Some machine checks are process-recoverable, others are not. Let a
CPU-specific handler decide what to do.

This works around a machine chec

powerpc: Add a CPU-custom machine check handler

Summary:
Some machine checks are process-recoverable, others are not. Let a
CPU-specific handler decide what to do.

This works around a machine check error hit while building www/firefox
and mail/thunderbird, which would otherwise cause the build to fail.

More work is needed to handle all possible machine check conditions, but
this is sufficient to unblock some ports building.

Differential Revision: https://reviews.freebsd.org/D23731

show more ...

[PowerPC] kernel ifunc support for powerpc*, fix ppc64 relocation oddities.

This is a general cleanup of the relocatable kernel support on powerpc,
needed to enable kernel ifuncs.

* Fix some reloc

[PowerPC] kernel ifunc support for powerpc*, fix ppc64 relocation oddities.

This is a general cleanup of the relocatable kernel support on powerpc,
needed to enable kernel ifuncs.

* Fix some relocatable issues in the kernel linker, and change to using
a RELOCATABLE_KERNEL #define instead of #ifdef __powerpc__ for parts that
other platforms can use in the future if they wish to have ET_DYN kernels.

* Get rid of the DB_STOFFS hack now that the kernel is relocated to the DMAP
properly across the board on powerpc64.

* Add powerpc64 and powerpc32 ifunc functionality.

* Allow AIM64 virtual mode OF kernels to run from the DMAP like other AIM64
by implementing a virtual mode restart. This fixes the runtime address on
PowerMac G5.

* Fix symbol relocation problems on post-relocation kernels by relocating
the symbol table.

* Add an undocumented method for supplying kernel symbols on powernv and
other powerpc machines using linux-style kernel/initrd loading -- If
you pass the kernel in as the initrd as well, the copy resident in initrd
will be used as a source for symbols when initializing the debugger.
This method is subject to removal once we have a better way of doing this.

Approved by: jhibbits
Relnotes: yes
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D23156

show more ...

[PowerPC] Fix VSX context handling

In r356767, memcpy/memmove/bcopy optimizations were added to libc to
improve performance.

This exposed an existing kernel issue in VSX handling. The PSL_VSX flag

[PowerPC] Fix VSX context handling

In r356767, memcpy/memmove/bcopy optimizations were added to libc to
improve performance.

This exposed an existing kernel issue in VSX handling. The PSL_VSX flag was
not being excluded from the psl_userstatic set, which meant that any thread
that used these and then called swapcontext(3) would get an EINVAL error.

Fixing this exposed a second issue - in r344123, the FPU was being forced
off in set_mcontext(). However, this was neglecting to ensure VSX was turned
off at the same time.

While here, add some code comments to explain what's going on.

Reviewed by: jhibbits, luporl (earlier rev), pkubaj (earlier rev)
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D23497

show more ...

D23057: [PowerPC] Fix offset calculations in bridge mode

In rS354701, I replaced text relocations with offsets from &generictrap.

Unfortunately, the magic variable I was using doesn't actually mean

D23057: [PowerPC] Fix offset calculations in bridge mode

In rS354701, I replaced text relocations with offsets from &generictrap.

Unfortunately, the magic variable I was using doesn't actually mean the
address of &generictrap, in bridge mode it actually means &generictrap64.

So, for bridge mode to work, it is necessary to differentiate between
"where do we need to branch to to handle a trap" and "where is &generictrap
for purposes of doing relative math".

Introduce a new TRAP_ENTRY and use it instead of TRAP_GENTRAP for doing
actual calls to the generic trap handler.

Reported by: Mark Millard <marklmi@yahoo.com>
Reviewed by: jhibbits
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D23057

show more ...

powerpc: Add POWER8NVL definition

The POWER8NVL (POWER8 NVLink) architecturally behaves identically to the
POWER8, with a different PVR identifier. Mark it as such, so it shows up
appropriately to

powerpc: Add POWER8NVL definition

The POWER8NVL (POWER8 NVLink) architecturally behaves identically to the
POWER8, with a different PVR identifier. Mark it as such, so it shows up
appropriately to the user.

Reported by: Alexey Kardashevskiy
MFC after: 2 weeks

show more ...

powerpc: Re-merge isa3 HPT with moea64 native HPT

r345402 fixed the bug that led to the split of the ISA 3.0 HPT handling from
the existing manager. The cause of the bug was gcc moving the register

powerpc: Re-merge isa3 HPT with moea64 native HPT

r345402 fixed the bug that led to the split of the ISA 3.0 HPT handling from
the existing manager. The cause of the bug was gcc moving the register
holding VPN to a different register (not r0), which triggered bizarre
behaviors. With the fix, things work, so they can be re-merged. No
performance lost with the merge.

show more ...

ppc64: handle exception 0x1500 (soft patch)

This change adds a hypervisor trap handler for exception 0x1500 (soft patch),
normalizing all VSX registers and returning.
This avoids a kernel panic due

ppc64: handle exception 0x1500 (soft patch)

This change adds a hypervisor trap handler for exception 0x1500 (soft patch),
normalizing all VSX registers and returning.
This avoids a kernel panic due to unknown exception.

Change made with the collaboration of leonardo.bianconi_eldorado.org.br,
that found out that this is a hypervisor exception and not a supervisor one,
and fixed this in the code.

Reviewed by: jhibbits, sbruno
Differential Revision: https://reviews.freebsd.org/D17806

show more ...

Split the PowerISA 3.0 HPT implementation from historic

PowerISA 3.0 makes several changes to not only the format of the HPT but
also the behavior surrounding it. For instance, TLBIE no longer requ

Split the PowerISA 3.0 HPT implementation from historic

PowerISA 3.0 makes several changes to not only the format of the HPT but
also the behavior surrounding it. For instance, TLBIE no longer requires
serialization. Removing this lock cuts buildworld time in half on a
18-core/72-thread POWER9 system, demonstrating that this lock is highly
contended on such a system.

There was odd behavior observed trying to make this change in a
backwards-compatible manner in moea64_native.c, so the best option was to
fully split it, and largely revert the original changes adding POWER9
support to the original file.

Suggested by: nwhitehorn

show more ...

Add hypervisor trap handling, using HSRR0/HSRR1

Summary:
Some hypervisor exceptions on POWER architecture only save state to HSRR0/HSRR1.
Until we have bhyve on POWER, use a lightweight exception fr

Add hypervisor trap handling, using HSRR0/HSRR1

Summary:
Some hypervisor exceptions on POWER architecture only save state to HSRR0/HSRR1.
Until we have bhyve on POWER, use a lightweight exception frontend which copies
HSRR0/HSRR1 into SRR0/SRR1, and run the normal trap handler.

The first user of this is the Hypervisor Virtualization Interrupt, which targets
the XIVE interrupt controller on POWER9.

Reviewed By: nwhitehorn
Differential Revision: https://reviews.freebsd.org/D15487

show more ...

powerpc64: Set n_slbs = 32 for POWER9

Summary:
POWER9 also contains 32 slbs entries as explained by the POWER9 User Manual:

"For HPT translation, the POWER9 core contains a unified (combined for b

powerpc64: Set n_slbs = 32 for POWER9

Summary:
POWER9 also contains 32 slbs entries as explained by the POWER9 User Manual:

"For HPT translation, the POWER9 core contains a unified (combined for both
instruction and data), 32-entry, fully-associative SLB per thread"

Submitted by: Breno Leitao
Differential Revision: https://reviews.freebsd.org/D15128

show more ...

Move most of the contents of opt_compat.h to opt_global.h.

opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
close

Move most of the contents of opt_compat.h to opt_global.h.

opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c. A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by: kib, cem, jhb, jtl
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14941

show more ...

Restore missing temporary variable, deleted by accident in r330845. This
unbreaks the ppc32 AIM build.

Reported by: jhibbits

Execute PowerPC64/AIM kernel from direct map region when possible.

When the kernel can be in real mode in early boot, we can execute from
high addresses aliased to the kernel's physical memory. If t

Execute PowerPC64/AIM kernel from direct map region when possible.

When the kernel can be in real mode in early boot, we can execute from
high addresses aliased to the kernel's physical memory. If that high
address has the first two bits set to 1 (0xc...), those addresses will
automatically become part of the direct map. This reduces page table
pressure from the kernel and it sets up the kernel to be used with
radix translation, for which it has to be up here.

This is accomplished by exploiting the fact that all PowerPC kernels are
built as position-independent executables and relocate themselves
on start. Before this patch, the kernel runs at 1:1 VA:PA, but that
VA/PA is random and set by the bootloader. Very early, it processes
its ELF relocations to operate wherever it happens to find itself.
This patch uses that mechanism to re-enter and re-relocate the kernel
a second time witha new base address set up in the early parts of
powerpc_init().

Reviewed by: jhibbits
Differential Revision: D14647

show more ...

Move the powerpc64 direct map base address from zero to high memory. This
accomplishes a few things:
- Makes NULL an invalid address in the kernel, which is useful for catching
bugs.
- Lays groundw

Move the powerpc64 direct map base address from zero to high memory. This
accomplishes a few things:
- Makes NULL an invalid address in the kernel, which is useful for catching
bugs.
- Lays groundwork for radix-tree translation on POWER9, which requires the
direct map be at high memory.
- Similarly lays groundwork for a direct map on 64-bit Book-E.

The new base address is chosen as the base of the fourth radix quadrant
(the minimum kernel address in this translation mode) and because all
supported CPUs ignore at least the first two bits of addresses in real
mode, allowing direct-map addresses to be used in real-mode handlers.
This is required by Linux and is part of the architecture standard
starting in POWER ISA 3, so can be relied upon.

Reviewed by: jhibbits, Breno Leitao
Differential Revision: D14499

show more ...

Merge AIM and Book-E PCPU fields

This is part of a long-term goal of merging Book-E and AIM into a single GENERIC
kernel. As more work is done, the struct may be optimized further.

Reviewed by: nw

Merge AIM and Book-E PCPU fields

This is part of a long-term goal of merging Book-E and AIM into a single GENERIC
kernel. As more work is done, the struct may be optimized further.

Reviewed by: nwhitehorn

show more ...