arm/pic: Sprinkle dtrace probes on interrupt handlers like x86.

arm/pic: Use a local variable to reduce #ifdef scope.

Avoids straddling a conditional this way.

arm/pic: Mark definitions static to match declarations.

arm/pic: Assert ci_cpl doesn't lower across interrupt handler.

Catch up with the per lwp astpending (rather than per cpu) flag in the
preemption code. NFC as preemption (still) isn't enabled.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
I

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

show more ...

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually require

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

show more ...

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state f

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

show more ...

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Fin

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

show more ...

Revert recent pic optimizations until I have more time to work on this.

Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag,

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

show more ...

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (in

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

show more ...

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU