sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

powernv: Include NUMA locality information in the CPU topology

ULE uses this topology to try and preserve locality when migrating
threads between CPUs and when performing work stealing. Ensure that

powernv: Include NUMA locality information in the CPU topology

ULE uses this topology to try and preserve locality when migrating
threads between CPUs and when performing work stealing. Ensure that on
NUMA systems it will at least take the NUMA topology into account.

Reviewed by: bdragon, jhibbits (previous version)
Tested by: bdragon
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D28580

show more ...

[PowerPC] Fix NUMA checking for powernv

At this point in startup, vm_ndomains has not been initialized. Switch
to checking kenv instead.

Fixes incorrect NUMA information being set on multi-domain s

[PowerPC] Fix NUMA checking for powernv

At this point in startup, vm_ndomains has not been initialized. Switch
to checking kenv instead.

Fixes incorrect NUMA information being set on multi-domain systems like
Talos II.

Submitted by: jhibbits
MFC after: 2 weeks

show more ...

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes wer

Implement superpages for PowerPC64 (HPT)

This change adds support for transparent superpages for PowerPC64
systems using Hashed Page Tables (HPT). All pmap operations are
supported.

The changes were inspired by RISC-V implementation of superpages,
by @markj (r344106), but heavily adapted to fit PPC64 HPT architecture
and existing MMU OEA64 code.

While these changes are not better tested, superpages support is disabled by
default. To enable it, use vm.pmap.superpages_enabled=1.

In this initial implementation, when superpages are disabled, system
performance stays at the same level as without these changes. When
superpages are enabled, buildworld time increases a bit (~2%). However,
for workloads that put a heavy pressure on the TLB the performance boost
is much bigger (see HPC Challenge and pgbench on D25237).

Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D25237

show more ...

[PowerPC64LE] powernv ILE setup code.

When running without a hypervisor, we need to set the ILE bit in the LPCR
ourselves.

For the boot processor, handle it in powernv_attach() like we do for other

[PowerPC64LE] powernv ILE setup code.

When running without a hypervisor, we need to set the ILE bit in the LPCR
ourselves.

For the boot processor, handle it in powernv_attach() like we do for other
LPCR bits.

No change for the APs, as they will use the lpcr global to set up their own
LPCR when they do their own cpudep_ap_early_bootstrap() and pick up this
automatically.

Sponsored by: Tag1 Consulting, Inc.

show more ...

powerpc: clean up empty lines in .c and .h files

powerpc/powernv: Don't configure disabled CPUs

If the POWER firmware detects a bad CPU core, it will "GUARD" it out,
marking it disabled. Any attempt to spin up a bad CPU will trigger a panic
later

powerpc/powernv: Don't configure disabled CPUs

If the POWER firmware detects a bad CPU core, it will "GUARD" it out,
marking it disabled. Any attempt to spin up a bad CPU will trigger a panic
later on when waiting for threads on said core to wake up. Support limping
along on fewer cores instead.

show more ...

powerpc/powernv: powernv_node_numa_domain() fix non-NUMA case

If NUMA is not enabled in the kernel config, or is disabled at boot, this
function should just return domain 0 regardless of what's in t

powerpc/powernv: powernv_node_numa_domain() fix non-NUMA case

If NUMA is not enabled in the kernel config, or is disabled at boot, this
function should just return domain 0 regardless of what's in the device
tree.

Fixes a panic in iflib with NUMA disabled.

Reported by: luporl

show more ...

[PPC64] Fix NUMA on POWER8

On some POWER8 machines, 'ibm,associativity' property may have 6
cells, which would overflow the 5 cells buffer being used.
There was also an issue with the "check if node

[PPC64] Fix NUMA on POWER8

On some POWER8 machines, 'ibm,associativity' property may have 6
cells, which would overflow the 5 cells buffer being used.
There was also an issue with the "check if node is root" part,
that have been fixed too.

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D23414

show more ...

powerpc: Fix the NUMA domain list on powernv

Summary:
Consolidate the NUMA associativity handling into a platform function.
Non-NUMA platforms will just fall back to the default (0). Currently
only

powerpc: Fix the NUMA domain list on powernv

Summary:
Consolidate the NUMA associativity handling into a platform function.
Non-NUMA platforms will just fall back to the default (0). Currently
only implemented for powernv, which uses a lookup table to map the
device tree associativity into a system NUMA domain.

Fixes hangs on powernv after r356534, and corrects a fairly longstanding
bug in powernv's NUMA handling, which ended up using domains 1 and 2 for
devices and memory on power9, while CPUs were bound to domains 0 and 1.

Reviewed by: bdragon, luporl
Differential Revision: https://reviews.freebsd.org/D23220

show more ...

powerpc/powernv: Set the PTCR for the Nest MMU

The Nest MMU manages address translation for accelerators on the POWER9. To
do so, it needs a page table, so export the system page table to the Nest

powerpc/powernv: Set the PTCR for the Nest MMU

The Nest MMU manages address translation for accelerators on the POWER9. To
do so, it needs a page table, so export the system page table to the Nest
MMU. This will quietly fail on pre-POWER9 systems that do not have a NMMU.

The NMMU is currently unused, so this change is currently effectively a NOP,
but the NMMU and VAS will eventually be used.

show more ...

Add NUMA support to powerpc

Summary:
Initial NUMA support:
- associate CPU with domain
- associate memory ranges with domain
- identify domain for devices
- limit device interrupt bi

Add NUMA support to powerpc

Summary:
Initial NUMA support:
- associate CPU with domain
- associate memory ranges with domain
- identify domain for devices
- limit device interrupt binding to appropriate domain

- Additionally fixes a bug in the setting of Maxmem which led to
only memory attached to the first socket being enabled for DMA

A pmap variant can opt in to numa support by by calling `numa_mem_regions`
at the end of pmap_bootstrap - registering the corresponding ranges with the
VM.

This yields a ~20% improvement in build times of llvm on dual socket POWER9
over non-NUMA.

Original patch by mmacy.

Differential Revision: https://reviews.freebsd.org/D17933

show more ...

powerpc/powernv: Add a driver for the POWER9 XIVE interrupt controller

The XIVE (External Interrupt Virtualization Engine) is a new interrupt
controller present in IBM's POWER9 processor. It's a ve

powerpc/powernv: Add a driver for the POWER9 XIVE interrupt controller

The XIVE (External Interrupt Virtualization Engine) is a new interrupt
controller present in IBM's POWER9 processor. It's a very powerful,
very complex device using queues and shared memory to improve interrupt
dispatch performance in a virtualized environment.

This yields a ~10% performance improvment over the XICS emulation mode,
measured in both buildworld, and 'dd' from nvme to /dev/null.

Currently, this only supports native access.

MFC after: 1 month

show more ...

powerpc: Fix regression introduced in r342771

In r342771, I introduced a regression in Power by abusing the platform
smp_topo() method as a shortcut for providing the MI information needed for
the s

powerpc: Fix regression introduced in r342771

In r342771, I introduced a regression in Power by abusing the platform
smp_topo() method as a shortcut for providing the MI information needed for
the stated sysctls. The smp_topo() method was already called later by
sched_ule (under the name cpu_topo()), and initializes a static array of
scheduler topology information. I had skimmed the smp_topo_foo() functions
and assumed they were idempotent; empirically, they are not (or at least,
detect re-initialization and panic).

Do the cleaner thing I should have done in the first place and add a
platform method specifically for core- and thread-count probing.

Reported by: luporl via jhibbits
Reviewed by: luporl
X-MFC-With: r342771
Differential Revision: https://reviews.freebsd.org/D18777

show more ...

Expose threads-per-core and physical core count information

With new sysctls (to the best of our ability do detect them). Restructured
smp.4 slightly for clarity (keep relevant stuff closer to the

Expose threads-per-core and physical core count information

With new sysctls (to the best of our ability do detect them). Restructured
smp.4 slightly for clarity (keep relevant stuff closer to the top) while
documenting.

Reviewed by: markj, jhibbits (ppc parts)
MFC after: 3 days
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D18322

show more ...

Add support for the XIVE XICS emulation mode for POWER9 systems

Summary:
POWER9 systems use a new interrupt controller, XIVE, managed through OPAL
firmware calls. The OPAL firmware includes support

Add support for the XIVE XICS emulation mode for POWER9 systems

Summary:
POWER9 systems use a new interrupt controller, XIVE, managed through OPAL
firmware calls. The OPAL firmware includes support for emulating the previous
generation XICS presentation layer in addition to a new "XIVE Exploitation"
mode. As a stopgap until we have XIVE exploitation mode, enable XICS emulation
mode so that we at least have an interrupt controller.

Since the CPPR is local to the current CPU, it cannot be updated for APs when
initializing on the BSP. This adds a new function, directly called by the
powernv platform code, to initialize the CPPR on AP bringup.

Reviewed by: nwhitehorn
Differential Revision: https://reviews.freebsd.org/D15492

show more ...

Fix use of unitialized variables.

PowerNV: Put processor to power-save state in idle thread

When processor enters power-save state it releases resources shared with other
cpu threads which makes other cores working much faster.

Thi

PowerNV: Put processor to power-save state in idle thread

When processor enters power-save state it releases resources shared with other
cpu threads which makes other cores working much faster.

This patch also implements saving and restoring registers that might get
corrupted in power-save state.

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Reviewed by: jhibbits, nwhitehorn, wma
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D14330

show more ...

PowerNV: move LPCR and LPID altering to cpudep_ap_early_bootstrap

It turns out that under some circumstances we can get DSI or DSE before we set
LPCR and LPID so we should set it as early as possibl

PowerNV: move LPCR and LPID altering to cpudep_ap_early_bootstrap

It turns out that under some circumstances we can get DSI or DSE before we set
LPCR and LPID so we should set it as early as possible.

Authored by: Patryk Duda <pdk@semihalf.com>
Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies

show more ...

PPC64: use hwref instead of cpuid

On CHRP and PowerNV, use the interrupt server number in the cpuref and pcpu
hwref field instead of the device-tree phandle and make the CPU IDs reported
to the sche

PPC64: use hwref instead of cpuid

On CHRP and PowerNV, use the interrupt server number in the cpuref and pcpu
hwref field instead of the device-tree phandle and make the CPU IDs reported
to the scheduler dense and with the BSP at 0.

Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D14011

show more ...

PPC64: cleanup APs startup routines

Cleaning up AP startup routines. This is a mix of changes
required to make PowerNV running and to modify the code
to be more robust. Previously, some races were s

PPC64: cleanup APs startup routines

Cleaning up AP startup routines. This is a mix of changes
required to make PowerNV running and to modify the code
to be more robust. Previously, some races were seen if more
than 90CPUs were online.

Authored by: Patryk Duda <pdk@semihalf.com>
Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D14026

show more ...

PowerNV: update OPAL driver

Update OPAL driver with:
- better console support
- proper AP configuration
- enhanced IRQ/OFW mapping
- RTC support

Created by: Nathan Whitehorn <nwhitehorn@

PowerNV: update OPAL driver

Update OPAL driver with:
- better console support
- proper AP configuration
- enhanced IRQ/OFW mapping
- RTC support

Created by: Nathan Whitehorn <nwhitehorn@freebsd.org>
Submitted by: Wojciech Macek <wma@semihalf.com>
Sponsored by: FreeBSD Foundation

show more ...

PowerNV: set LPCR[LPES] correctly

Make sure to set LPCR[LPES] so that external interrupts set SRR0 and SRR1
instead of HSRR0 and HSRR1. Without this, external interrupt handlers would
get the wrong

PowerNV: set LPCR[LPES] correctly

Make sure to set LPCR[LPES] so that external interrupts set SRR0 and SRR1
instead of HSRR0 and HSRR1. Without this, external interrupt handlers would
get the wrong MSR value when executing, causing eventual madness.

Created by: Nathan Whitehorn <nwhitehorn@freebsd.org>
Submitted by: Wojciech Macek <wma@freebsd.org>
Sponsored by: FreeBSD Foundation

show more ...

PowerNV: correctly start secondary CPUs

Fix AP startup, which was broken.

Created by: Nathan Whitehorn <nwhitehorn@freebsd.org>
Submitted by: Wojciech Macek <wma@freebsd.org>
Sp

PowerNV: correctly start secondary CPUs

Fix AP startup, which was broken.

Created by: Nathan Whitehorn <nwhitehorn@freebsd.org>
Submitted by: Wojciech Macek <wma@freebsd.org>
Sponsored by: FreeBSD Foundation

show more ...