xref: /dports/emulators/qemu60/qemu-6.0.0/roms/skiboot/doc/release-notes/skiboot-6.3.rst

.. _skiboot-6.3:

skiboot-6.3
===========

skiboot v6.3 was released on Friday May 3rd 2019. It is the first
release of skiboot 6.3, which becomes the new stable release
of skiboot following the 6.2 release, first released December 14th 2018.

Skiboot 6.3 will mark the basis for op-build v2.3.

skiboot v6.3 contains all bug fixes as of :ref:`skiboot-6.0.20`,
and :ref:`skiboot-6.2.3` (the currently maintained
stable releases).

For how the skiboot stable releases work, see :ref:`stable-rules` for details.

Over skiboot 6.2, we have the following changes:

.. _skiboot-6.3-new-features:

New Features
------------

- hw/imc: Enable opal calls to init/start/stop IMC Trace mode

  New OPAL APIs for In-Memory Collection Counter infrastructure(IMC),
  including a new device type called OPAL_IMC_COUNTERS_TRACE.
- xive: Add calls to save/restore the queues and VPs HW state

  To be able to support migration of guests using the XIVE native
  exploitation mode, (where the queue is effectively owned by the
  guest), KVM needs to be able to save and restore the HW-modified
  fields of the queue, such as the current queue producer pointer and
  generation bit, and to retrieve the modified thread context registers
  of the VP from the NVT structure : the VP interrupt pending bits.

  However, there is no need to set back the NVT structure on P9. P10
  should be the same.
- witherspoon: Add nvlink2 interconnect information

  GPUs on Redbud and Sequoia platforms are interconnected in groups of
  2 or 3 GPUs. The problem with that is if the user decides to pass a single
  GPU from a group to the userspace, we need to ensure that links between
  GPUs do not get enabled.

  A V100 GPU provides a way to disable selected links. In order to only
  disable links to peer GPUs, we need a topology map.

  This adds an "ibm,nvlink-peers" property to a GPU DT node with phandles
  of peer GPUs and NVLink2 bridges. The index in the property is a GPU link
  number.
- platforms/romulus: Also support talos

  The two are similar enough and I'd like to have a slot table for our
  Talos.
- OpenCAPI support! (see :ref:`skiboot-6.3-OpenCAPI` section)
- opal/hmi: set a flag to inform OS that TOD/TB has failed.

  Set a flag to indicate OS about TOD/TB failure as part of new
  opal_handle_hmi2 handler. This flag then can be used by OS to make sure
  functions depending on TB value (e.g. udelay()) are aware of TB not
  ticking.
- astbmc: Enable IPMI HIOMAP for AMI platforms

  Required for Habanero, Palmetto and Romulus.
- power-mgmt : occ : Add 'freq-domain-mask' DT property

  Add a new device-tree property freq-domain-indicator to define group of
  CPUs which would share same frequency. This property has been added under
  power-mgmt node. It is a bitmask.

  Bitwise AND is taken between this bitmask value and PIR of cpu. All the
  CPUs lying in the same frequency domain will have same result for AND.

  For example, For POWER9, 0xFFF0 indicates quad wide frequency domain.
  Taking AND with the PIR of CPUs will yield us frequency domain which is
  quad wise distribution as last 4 bits have been masked which represent the
  cores.

  Similarly, 0xFFF8 will represent core wide frequency domain for P8.

  Also, Add a new device-tree property domain-runs-at which will denote the
  strategy OCC is using to change the frequency of a frequency-domain. There
  can be two strategy - FREQ_MOST_RECENTLY_SET and FREQ_MAX_IN_DOMAIN.

  FREQ_MOST_RECENTLY_SET : the OCC sets the frequency of the quad to the most
  recent frequency value requested by the CPUs in the quad.

  FREQ_MAX_IN_DOMAIN : the OCC sets the frequency of the CPUs in
  the Quad to the maximum of the latest frequency requested by each of
  the component cores.
- powercap: occ: Fix the powercapping range allowed for user

  OCC provides two limits for minimum powercap. One being hard powercap
  minimum which is guaranteed by OCC and the other one is a soft
  powercap minimum which is lesser than hard-min and may or may not be
  asserted due to various power-thermal reasons. So to allow the users
  to access the entire powercap range, this patch exports soft powercap
  minimum as the "powercap-min" DT property. And it also adds a new
  DT property called "powercap-hard-min" to export the hard-min powercap
  limit.
- Add NVDIMM support

  NVDIMMs are memory modules that use a battery backup system to allow the
  contents RAM to be saved to non-volatile storage if system power goes
  away unexpectedly. This allows them to be used a high-performance
  storage device, suitable for serving as a cache for SSDs and the like.

  Configuration of NVDIMMs is handled by hostboot and communicated to OPAL
  via the HDAT. We need to parse out the NVDIMM memory ranges and create
  memory regions with the "pmem-region" compatible label to make them
  available to the host.
- core/exceptions: implement support for MCE interrupts in powersave

  The ISA specifies that MCE interrupts in power saving modes will enter
  at 0x200 with powersave bits in SRR1 set. This is not currently
  supported properly, the MCE will just happen like a normal interrupt,
  but GPRs could be lost, which would lead to crashes (e.g., r1, r2, r13
  etc).

  So check the power save bits similarly to the sreset vector, and
  handle this properly.
- core/exceptions: allow recoverable sreset exceptions

  This requires implementing the MSR[RI] bit. Then just allow all
  non-fatal sreset exceptions to recover.
- core/exceptions: implement an exception handler for non-powersave sresets

  Detect non-powersave sresets and send them to the normal exception
  handler which prints registers and stack.
- Add PVR_TYPE_P9P

  Enable a new PVR to get us running on another p9 variant.

Since v6.3-rc2:

- Expose PNOR Flash partitions to host MTD driver via devicetree

  This makes it possible for the host to directly address each
  partition without requiring each application to directly parse
  the FFS headers.  This has been in use for some time already to
  allow BOOTKERNFW partition updates from the host.

  All partitions except BOOTKERNFW are marked readonly.

  The BOOTKERNFW partition is currently exclusively used by the TalosII platform

- Write boot progress to LPC port 80h

  This is an adaptation of what we currently do for op_display() on FSP
  machines, inventing an encoding for what we can write into the single
  byte at LPC port 80h.

  Port 80h is often used on x86 systems to indicate boot progress/status
  and dates back a decent amount of time. Since a byte isn't exactly very
  expressive for everything that can go on (and wrong) during boot, it's
  all about compromise.

  Some systems (such as Zaius/Barreleye G2) have a physical dual 7 segment
  display that display these codes. So far, this has only been driven by
  hostboot (see hostboot commit 90ec2e65314c).

- Write boot progress to LPC ports 81 and 82

  There's a thought to write more extensive boot progress codes to LPC
  ports 81 and 82 to supplement/replace any reliance on port 80.

  We want to still emit port 80 for platforms like Zaius and Barreleye
  that have the physical display. Ports 81 and 82 can be monitored by a
  BMC though.

- Add Talos II platform

  Talos II has some hardware differences from Romulus, therefore
  we cannot guarantee Talos II == Romulus in skiboot.  Copy and
  slightly modify the Romulus files for Talos II.

Since v6.3-rc1:

- cpufeatures: Add tm-suspend-hypervisor-assist and tm-suspend-xer-so-bug node

  tm-suspend-hypervisor-assist for P9 >=DD2.2
  And a tm-suspend-xer-so-bug node for P9 DD2.2 only.

  I also treat P9P as P9 DD2.3 and add a unit test for the cpufeatures
  infrastructure.

  Fixes: https://github.com/open-power/skiboot/issues/233


Deprecated/Removed Features
---------------------------

- opal: Deprecate reading the PHB status

  The OPAL_PCI_EEH_FREEZE_STATUS call takes a bunch of parameters, one of
  them is @phb_status. It is defined as __be64* and always NULL in
  the current Linux upstream but if anyone ever decides to read that status,
  then the PHB3's handler will assume it is struct OpalIoPhb3ErrorData*
  (which is a lot bigger than 8 bytes) and zero it causing the stack
  corruption; p7ioc-phb has the same issue.

  This removes @phb_status from all eeh_freeze_status() hooks and moves
  the error message from PHB4 to the affected OPAL handlers.

  As far as we can tell, nobody has ever used this and thus it's safe to remove.
- Remove POWER9N DD1 support

  This is not a shipping product and is no longer supported by Linux
  or other firmware components.

Since v6.3-rc3:

- Disable fast-reset for POWER8

  There is a bug with fast-reset when CPU cores are busy, which can be
  reproduced by running `stress` and then trying `reboot -ff` (this is
  what the op-test test cases FastRebootHostStress and
  FastRebootHostStressTorture do). What happens is the cores lock up,
  which isn't the best thing in the world when you want them to start
  executing instructions again.

  A workaround is to use instruction ramming, which while greatly
  increasing the reliability of fast-reset on p8, doesn't make it perfect.

  Instruction ramming is what pdbg was modified to do in order to have the
  sreset functionality work reliably on p8.
  pdbg patches: https://patchwork.ozlabs.org/project/pdbg/list/?series=96593&state=*

  Fixes: https://github.com/open-power/skiboot/issues/185

General
-------

- core/i2c: Various bits of refactoring
- refactor backtrace generation infrastructure
- astbmc: Handle failure to initialise raw flash

  Initialising raw flash lead to a dead assignment to rc. Check the return
  code and take the failure path as necessary. Both before and after the
  fix we see output along the lines of the following when flash_init()
  fails: ::

    [   53.283182881,7] IRQ: Registering 0800..0ff7 ops @0x300d4b98 (data 0x3052b9d8)
    [   53.283184335,7] IRQ: Registering 0ff8..0fff ops @0x300d4bc8 (data 0x3052b9d8)
    [   53.283185513,7] PHB#0000: Initializing PHB...
    [   53.288260827,4] FLASH: Can't load resource id:0. No system flash found
    [   53.288354442,4] FLASH: Can't load resource id:1. No system flash found
    [   53.342933439,3] CAPP: Error loading ucode lid. index=200ea
    [   53.462749486,2] NVRAM: Failed to load
    [   53.462819095,2] NVRAM: Failed to load
    [   53.462894236,2] NVRAM: Failed to load
    [   53.462967071,2] NVRAM: Failed to load
    [   53.463033077,2] NVRAM: Failed to load
    [   53.463144847,2] NVRAM: Failed to load

  Eventually followed by: ::

    [   57.216942479,5] INIT: platform wait for kernel load failed
    [   57.217051132,5] INIT: Assuming kernel at 0x20000000
    [   57.217127508,3] INIT: ELF header not found. Assuming raw binary.
    [   57.217249886,2] NVRAM: Failed to load
    [   57.221294487,0] FATAL: Kernel is zeros, can't execute!
    [   57.221397429,0] Assert fail: core/init.c:615:0
    [   57.221471414,0] Aborting!
    CPU 0028 Backtrace:
     S: 0000000031d43c60 R: 000000003001b274   ._abort+0x4c
     S: 0000000031d43ce0 R: 000000003001b2f0   .assert_fail+0x34
     S: 0000000031d43d60 R: 0000000030014814   .load_and_boot_kernel+0xae4
     S: 0000000031d43e30 R: 0000000030015164   .main_cpu_entry+0x680
     S: 0000000031d43f00 R: 0000000030002718   boot_entry+0x1c0
     --- OPAL boot ---

  Analysis of the execution paths suggests we'll always "safely" end this
  way due the setup sequence for the blocklevel callbacks in flash_init()
  and error handling in blocklevel_get_info(), and there's no current risk
  of executing from unexpected memory locations. As such the issue is
  reduced to down to a fix for poor error hygene in the original change
  and a resolution for a Coverity warning (famous last words etc).
- core/flash: Retry requests as necessary in flash_load_resource()

  We would like to successfully boot if we have a dependency on the BMC
  for flash even if the BMC is not current ready to service flash
  requests. On the assumption that it will become ready, retry for several
  minutes to cover a BMC reboot cycle and *eventually* rather than
  *immediately* crash out with: ::

        [  269.549748] reboot: Restarting system
        [  390.297462587,5] OPAL: Reboot request...
        [  390.297737995,5] RESET: Initiating fast reboot 1...
        [  391.074707590,5] Clearing unused memory:
        [  391.075198880,5] PCI: Clearing all devices...
        [  391.075201618,7] Clearing region 201ffe000000-201fff800000
        [  391.086235699,5] PCI: Resetting PHBs and training links...
        [  391.254089525,3] FFS: Error 17 reading flash header
        [  391.254159668,3] FLASH: Can't open ffs handle: 17
        [  392.307245135,5] PCI: Probing slots...
        [  392.363723191,5] PCI Summary:
        ...
        [  393.423255262,5] OCC: All Chip Rdy after 0 ms
        [  393.453092828,5] INIT: Starting kernel at 0x20000000, fdt at
        0x30800a88 390645 bytes
        [  393.453202605,0] FATAL: Kernel is zeros, can't execute!
        [  393.453247064,0] Assert fail: core/init.c:593:0
        [  393.453289682,0] Aborting!
        CPU 0040 Backtrace:
         S: 0000000031e03ca0 R: 000000003001af60   ._abort+0x4c
         S: 0000000031e03d20 R: 000000003001afdc   .assert_fail+0x34
         S: 0000000031e03da0 R: 00000000300146d8   .load_and_boot_kernel+0xb30
         S: 0000000031e03e70 R: 0000000030026cf0   .fast_reboot_entry+0x39c
         S: 0000000031e03f00 R: 0000000030002a4c   fast_reset_entry+0x2c
         --- OPAL boot ---

  The OPAL flash API hooks directly into the blocklevel layer, so there's
  no delay for e.g. the host kernel, just for asynchronously loaded
  resources during boot.
- fast-reboot: occ: Call occ_pstates_init() on fast-reset on all machines

  Commit 815417dcda2e ("init, occ: Initialise OCC earlier on BMC systems")
  conditionally invoked occ_pstates_init() only on FSP based systems in
  load_and_boot_kernel(). Due to this pstate table is re-parsed on FSP
  system and skipped on BMC system during fast-reboot. So this patch fixes
  this by invoking occ_pstates_init() on all boxes during fast-reboot.
- opal/hmi: Don't retry TOD recovery if it is already in failed state.

  On TOD failure, all cores/thread receives HMI and very first thread that
  gets interrupt fixes the TOD where as others just resets the respective
  HMER error bit and return. But when TOD is unrecoverable, all the threads
  try to do TOD recovery one by one causing threads to spend more time inside
  opal. Set a global flag when TOD is unrecoverable so that rest of the
  threads go back to linux immediately avoiding lock ups in system
  reboot/panic path.
- hw/bt: Do not disable ipmi message retry during OPAL boot

  Currently OPAL doesn't know whether BMC is functioning or not. If BMC is
  down (like BMC reboot), then we keep on retry sending message to BMC. So
  in some corner cases we may hit hard lockup issue in kernel.

  Ideally we should avoid using synchronous path as much as possible. But
  for now commit 01f977c3 added option to disable message retry in synchronous.
  But this fix is not required during boot. Hence lets disable IPMI message
  retry during OPAL boot.
- hdata/memory: Fix warning message

  Even though we added memory to device tree, we are getting below warning. ::

    [   57.136949696,3] Unable to use memory range 0 from MSAREA 0
    [   57.137049753,3] Unable to use memory range 0 from MSAREA 1
    [   57.137152335,3] Unable to use memory range 0 from MSAREA 2
    [   57.137251218,3] Unable to use memory range 0 from MSAREA 3
- hw/bt: Add backend interface to disable ipmi message retry option

  During boot OPAL makes IPMI_GET_BT_CAPS call to BMC to get BT interface
  capabilities which includes IPMI message max resend count, message
  timeout, etc,. Most of the time OPAL gets response from BMC within
  specified timeout. In some corner cases (like mboxd daemon reset in BMC,
  BMC reboot, etc) OPAL may not get response within timeout period. In
  such scenarios, OPAL resends message until max resend count reaches.

  OPAL uses synchronous IPMI message (ipmi_queue_msg_sync()) for few
  operations like flash read, write, etc. Thread will wait in OPAL until
  it gets response from BMC. In some corner cases like BMC reboot, thread
  may wait in OPAL for long time (more than 20 seconds) and results in
  kernel hardlockup.

  This patch introduces new interface to disable message resend option. We
  will disable message resend option for synchrous message. This will
  greatly reduces kernel hardlock up issues.

  This is short term fix. Long term solution is to convert all synchronous
  messages to asynhrounous one.
- ipmi/power: Fix system reboot issue

  Kernel makes reboot/shudown OPAL call for reboot/shutdown. Once kernel
  gets response from OPAL it runs opal_poll_events() until firmware
  handles the request.

  On BMC based system, OPAL makes IPMI call (IPMI_CHASSIS_CONTROL) to
  initiate system reboot/shutdown. At present OPAL queues IPMI messages
  and return SUCESS to Host. If BMC is not ready to accept command (like
  BMC reboot), then these message will fail. We have to manually
  reboot/shutdown the system using BMC interface.

  This patch adds logic to validate message return value. If message failed,
  then it will resend the message. At some stage BMC will be ready to accept
  message and handles IPMI message.
- firmware-versions: Add test case for parsing VERSION

  Also make it possible to use with afl-lop/afl-fuzz just to help make
  *sure* we're all good.

  Additionally, if we hit a entry in VERSION that is larger than our
  buffer size, we skip over it gracefully rather than overwriting the
  stack. This is only a problem if VERSION isn't trusted, which as of
  4b8cc05a94513816d43fb8bd6178896b430af08f it is verified as part of
  Secure Boot.
- core/fast-reboot: improve NMI handling during fast reset

  Improve sreset and MCE handling in fast reboot. Switch the HILE bit
  off before copying OPAL's exception vectors, so NMIs can be handled
  properly. Also disable MSR[ME] while the vectors are being overwritten
- core/cpu: HID update race

  If the per-core HID register is updated concurrently by multiple
  threads, updates can get lost. This has been observed during fast
  reboot where the HILE bit does not get cleared on all cores, which
  can cause machine check exception interrupts to crash.

  Fix this by only updating HID on thread0.
- SLW: Print verbose info on errors only

  Change print level from debug to warning for reporting
  bad EC_PPM_SPECIAL_WKUP_* scom values. To reduce cluttering
  in the log print only on error.

Since v6.3-rc2:

- hw/xscom: add missing P9P chip name
- asm/head: balance branches to avoid link stack predictor mispredicts

  The Linux wrapper for OPAL call and return is arranged like this: ::

      __opal_call:
          mflr   r0
          std    r0,PPC_STK_LROFF(r1)
          LOAD_REG_ADDR(r11, opal_return)
          mtlr   r11
          hrfid  -> OPAL

      opal_return:
          ld     r0,PPC_STK_LROFF(r1)
          mtlr   r0
          blr

  When skiboot returns to Linux, it branches to LR (i.e., opal_return)
  with a blr. This unbalances the link stack predictor and will cause
  mispredicts back up the return stack.
- external/mambo: also invoke readline for the non-autorun case
- asm/head.S: set POWER9 radix HID bit at entry

  When running in virtual memory mode, the radix MMU hid bit should not
  be changed, so set this in the initial boot SPR setup.

  As a side effect, fast reboot also has HID0:RADIX bit set by the
  shared spr init, so no need for an explicit call.
- build: link with --orphan-handling=warn

  The linker can warn when the linker script does not explicitly place
  all sections. These orphan sections are placed according to
  heuristics, which may not always be desirable. Enable this warning.
- build: -fno-asynchronous-unwind-tables

  skiboot does not use unwind tables, this option saves about 100kB,
  mostly from .text.
- opal/hmi: Initialize the hmi event with old value of TFMR.

  Do this before we fix TFAC errors. Otherwise the event at host console
  shows no thread error reported in TFMR register.

  Without this patch the console event show TFMR with no thread error:
  (DEC parity error TFMR[59] injection) ::

    [   53.737572] Severe Hypervisor Maintenance interrupt [Recovered]
    [   53.737596]  Error detail: Timer facility experienced an error
    [   53.737611]  HMER: 0840000000000000
    [   53.737621]  TFMR: 3212000870e04000

  After this patch it shows old TFMR value on host console: ::

    [ 2302.267271] Severe Hypervisor Maintenance interrupt [Recovered]
    [ 2302.267305]  Error detail: Timer facility experienced an error
    [ 2302.267320]  HMER: 0840000000000000
    [ 2302.267330]  TFMR: 3212000870e14010


IBM FSP based platforms
-----------------------

- platforms/firenze: Rework I2C controller fixups
- platforms/zz: Re-enable LXVPD slot information parsing

  From memory this was disabled in the distant past since we were waiting
  for an updates to the LXPVD format. It looks like that never happened
  so re-enable it for the ZZ platform so that we can get PCI slot location
  codes on ZZ.

HIOMAP
------
- astbmc: Try IPMI HIOMAP for P8

  The HIOMAP protocol was developed after the release of P8 in preparation
  for P9. As a consequence P9 always uses it, but it has rarely been
  enabled for P8. P8DTU has recently added IPMI HIOMAP support to its BMC
  firmware, so enable its use in skiboot with P8 machines. Doing so
  requires some rework to ensure fallback works correctly as in the past
  the fallback was to mbox, which will only work for P9.
- libflash/ipmi-hiomap: Enforce message size for empty response

  The protocol defines the response to the associated messages as empty
  except for the command ID and sequence fields. If the BMC is returning
  extra data consider the message malformed.
- libflash/ipmi-hiomap: Remove unused close handling

  Issuing a HIOMAP_C_CLOSE is not required by the protocol specification,
  rather a close can be implicit in a subsequent
  CREATE_{READ,WRITE}_WINDOW request. The implicit close provides an
  opportunity to reduce LPC traffic and the implementation takes up that
  optimisation, so remove the case from the IPMI callback handler.
- libflash/ipmi-hiomap: Overhaul event handling

  Reworking the event handling was inspired by a bug report by Vasant
  where the host would get wedged on multiple flash access attempts in the
  face of a persistent error state on the BMC-side. The cause of this bug
  was the early-exit based on ctx->update, which erronously assumed that
  all events had been completely handled in prior calls to
  ipmi_hiomap_handle_events(). This is not true if e.g.
  HIOMAP_E_DAEMON_READY is clear in the prior calls.

  Regardless, there were other correctness and efficiency problems with
  the handling strategy:

  * Ack-able event state was not restored in the face of errors in the
    process of re-establishing protocol state
  * It forced needless window restoration with respect to the context in
    which ipmi_hiomap_handle_events() was called.
  * Tests for HIOMAP_E_DAEMON_READY and HIOMAP_E_FLASH_LOST were redundant
    with the overhauled error handling introduced in the previous patch

  Fix all of the above issues and add comments to explain the event
  handling flow.
- libflash/ipmi-hiomap: Overhaul error handling

  The aim is to improve the robustness with respect to absence of the
  BMC-side daemon. The current error handling roughly mirrors what was
  done for the mailbox implementation, but there's room for improvement.

  Errors are split into two classes, those that affect the transport state
  and those that affect the window validity. From here, we push the
  transport state error checks right to the bottom of the stack, to ensure
  the link is known to be in a good state before any message is sent.
  Window validity tests remain as they were in the hiomap_window_move()
  and ipmi_hiomap_read() functions. Validity tests are not necessary in
  the write and erase paths as we will receive an error response from the
  BMC when performing a dirty or flush on an invalid window.

  Recovery also remains as it was, done on entry to the blocklevel
  callbacks. If an error state is encountered in the middle of an
  operation no attempt is made to recover it on the spot, instead the
  error is returned up the stack and the caller can choose how it wishes
  to respond.
- libflash/ipmi-hiomap: Fix leak of msg in callback

Since v6.3-rc1:

- libflash/ipmi-hiomap: Fix blocks count issue

  We convert data size to block count and pass block count to BMC.
  If data size is not block aligned then we endup sending block count
  less than actual data. BMC will write partial data to flash memory.

  Sample log ::

    [  594.388458416,7] HIOMAP: Marked flash dirty at 0x42010 for 8
    [  594.398756487,7] HIOMAP: Flushed writes
    [  594.409596439,7] HIOMAP: Marked flash dirty at 0x42018 for 3970
    [  594.419897507,7] HIOMAP: Flushed writes

  In this case HIOMAP sent data with block count=0 and hence BMC didn't
  flush data to flash.



POWER8
------
- hw/phb3/naples: Disable D-states

  Putting "Mellanox Technologies MT27700 Family [ConnectX-4] [15b3:1013]"
  (more precisely, the second of 2 its PCI functions, no matter in what
  order) into the D3 state causes EEH with the "PCT timeout" error.
  This has been noticed on garrison machines only and firestones do not
  seem to have this issue.

  This disables D-states changing for devices on root buses on Naples by
  installing a config space access filter (copied from PHB4).
- cpufeatures: Always advertise POWER8NVL as DD2

  Despite the major version of PVR being 1 (0x004c0100) for POWER8NVL,
  these chips are functionally equalent to P8/P8E DD2 levels.

  This advertises POWER8NVL as DD2. As the result, skiboot adds
  ibm,powerpc-cpu-features/processor-control-facility for such CPUs and
  the linux kernel can use hypervisor doorbell messages to wake secondary
  threads; otherwise "KVM: CPU %d seems to be stuck" would appear because
  of missing LPCR_PECEDH.

p8dtu Platform
^^^^^^^^^^^^^^
- p8dtu: Configure BMC graphics

  We can no-longer read the values from the BMC in the way we have in the
  past. Values were provided by Eric Chen of SMC.
- p8dtu: Enable HIOMAP support

Vesnin Platform
^^^^^^^^^^^^^^^
- platforms/vesnin: Disable PCIe port bifurcation

  PCIe ports connected to CPU1 and CPU3 now work as x16 instead of x8x8.

- Fix hang in pnv_platform_error_reboot path due to TOD failure.

  On TOD failure, with TB stuck, when linux heads down to
  pnv_platform_error_reboot() path due to unrecoverable hmi event, the panic
  cpu gets stuck in OPAL inside ipmi_queue_msg_sync(). At this time, rest
  all other cpus are in smp_handle_nmi_ipi() waiting for panic cpu to proceed.
  But with panic cpu stuck inside OPAL, linux never recovers/reboot. ::

    p0 c1 t0
    NIA : 0x000000003001dd3c <.time_wait+0x64>
    CFAR : 0x000000003001dce4 <.time_wait+0xc>
    MSR : 0x9000000002803002
    LR : 0x000000003002ecf8 <.ipmi_queue_msg_sync+0xec>

    STACK: SP NIA
    0x0000000031c236e0 0x0000000031c23760 (big-endian)
    0x0000000031c23760 0x000000003002ecf8 <.ipmi_queue_msg_sync+0xec>
    0x0000000031c237f0 0x00000000300aa5f8 <.hiomap_queue_msg_sync+0x7c>
    0x0000000031c23880 0x00000000300aaadc <.hiomap_window_move+0x150>
    0x0000000031c23950 0x00000000300ab1d8 <.ipmi_hiomap_write+0xcc>
    0x0000000031c23a90 0x00000000300a7b18 <.blocklevel_raw_write+0xbc>
    0x0000000031c23b30 0x00000000300a7c34 <.blocklevel_write+0xfc>
    0x0000000031c23bf0 0x0000000030030be0 <.flash_nvram_write+0xd4>
    0x0000000031c23c90 0x000000003002c128 <.opal_write_nvram+0xd0>
    0x0000000031c23d20 0x00000000300051e4 <opal_entry+0x134>
    0xc000001fea6e7870 0xc0000000000a9060 <opal_nvram_write+0x80>
    0xc000001fea6e78c0 0xc000000000030b84 <nvram_write_os_partition+0x94>
    0xc000001fea6e7960 0xc0000000000310b0 <nvram_pstore_write+0xb0>
    0xc000001fea6e7990 0xc0000000004792d4 <pstore_dump+0x1d4>
    0xc000001fea6e7ad0 0xc00000000018a570 <kmsg_dump+0x140>
    0xc000001fea6e7b40 0xc000000000028e5c <panic_flush_kmsg_end+0x2c>
    0xc000001fea6e7b60 0xc0000000000a7168 <pnv_platform_error_reboot+0x68>
    0xc000001fea6e7bd0 0xc0000000000ac9b8 <hmi_event_handler+0x1d8>
    0xc000001fea6e7c80 0xc00000000012d6c8 <process_one_work+0x1b8>
    0xc000001fea6e7d20 0xc00000000012da28 <worker_thread+0x88>
    0xc000001fea6e7db0 0xc0000000001366f4 <kthread+0x164>
    0xc000001fea6e7e20 0xc00000000000b65c <ret_from_kernel_thread+0x5c>

  This is because, there is a while loop towards the end of
  ipmi_queue_msg_sync() which keeps looping until "sync_msg" does not match
  with "msg". It loops over time_wait_ms() until exit condition is met. In
  normal scenario time_wait_ms() calls run pollers so that ipmi backend gets
  a chance to check ipmi response and set sync_msg to NULL. ::

            while (sync_msg == msg)
                    time_wait_ms(10);

  But in the event when TB is in failed state time_wait_ms()->time_wait_poll()
  returns immediately without calling pollers and hence we end up looping
  forever. This patch fixes this hang by calling opal_run_pollers() in TB
  failed state as well.


.. _skiboot-6.3-power9:

POWER9
------

- Retry link training at PCIe GEN1 if presence detected but training repeatedly failed

  Certain older PCIe 1.0 devices will not train unless the training process starts at GEN1 speeds.
  As a last resort when a device will not train, fall back to GEN1 speed for the last training attempt.

  This is verified to fix devices based on the Conexant CX23888 on the Talos II platform.
- hw/phb4: Drop FRESET_DEASSERT_DELAY state

  The delay between the ASSERT_DELAY and DEASSERT_DELAY states is set to
  one timebase tick. This state seems to have been a hold over from PHB3
  where it was used to add a 1s delay between de-asserting PERST and
  polling the link for the CAPI FPGA. There's no requirement for that here
  since the link polling on PHB4 is a bit smarter so we should be fine.
- hw/phb4: Factor out PERST control

  Some time ago Mikey added some code work around a bug we found where a
  certain RAID card wouldn't come back again after a fast-reboot. The
  workaround is setting the Link Disable bit before asserting PERST and
  clear it after de-asserting PERST.

  Currently we do this in the FRESET path, but not in the CRESET path.
  This patch moves the PERST control into its own function to reduce
  duplication and to the workaround is applied in all circumstances.
- hw/phb4: Remove FRESET presence check

  When we do an freset the first step is to check if a card is present in
  the slot. However, this only occurs when we enter phb4_freset() with the
  slot state set to SLOT_NORMAL. This occurs in:

  a) The creset path, and
  b) When the OS manually requests an FRESET via an OPAL call.

  (a) is problematic because in the boot path the generic code will put the
  slot into FRESET_START manually before calling into phb4_freset(). This
  can result in a situation where a device is detected on boot, but not
  after a CRESET.

  I've noticed this occurring on systems where the PHB's slot presence
  detect signal is not wired to an adapter. In this situation we can rely
  on the in-band presence mechanism, but the presence check will make
  us exit before that has a chance to work.

  Additionally, if we enter from the CRESET path this early exit leaves
  the slot's PERST signal being left asserted. This isn't currently an issue,
  but if we want to support hotplug of devices into the root port it will
  be.
- hw/phb4: Skip FRESET PERST when coming from CRESET

  PERST is asserted at the beginning of the CRESET process to prevent
  the downstream device from interacting with the host while the PHB logic
  is being reset and re-initialised. There is at least a 100ms wait during
  the CRESET processing so it's not necessary to wait this time again
  in the FRESET handler.

  This patch extends the delay after re-setting the PHB logic to extend
  to the 250ms PERST wait period that we typically use and sets the
  skip_perst flag so that we don't wait this time again in the FRESET
  handler.
- hw/phb4: Look for the hub-id from in the PBCQ node

  The hub-id is stored in the PBCQ node rather than the stack node so we
  never add it to the PHB node. This breaks the lxvpd slot lookup code
  since the hub-id is encoded in the VPD record that we need to find the
  slot information.
- hdata/iohub: Look for IOVPD on P9

  P8 and P9 use the same IO VPD setup, so we need to load the IOHUB VPD on
  P9 systems too.

Since v6.3-rc2:

- hw/phb4: Squash the IO bridge window

  The PCI-PCI bridge spec says that bridges that implement an IO window
  should hardcode the IO base and limit registers to zero.
  Unfortunately, these registers only define the upper bits of the IO
  window and the low bits are assumed to be 0 for the base and 1 for the
  limit address. As a result, setting both to zero can be mis-interpreted
  as a 4K IO window.

  This patch fixes the problem the same way PHB3 does. It sets the IO base
  and limit values to 0xf000 and 0x1000 respectively which most software
  interprets as a disabled window.

  lspci before patch: ::

    0000:00:00.0 PCI bridge: IBM Device 04c1 (prog-if 00 [Normal decode])
            I/O behind bridge: 00000000-00000fff

  lspci after patch: ::

    0000:00:00.0 PCI bridge: IBM Device 04c1 (prog-if 00 [Normal decode])
            I/O behind bridge: None

- hw/xscom: Enable sw xstop by default on p9

  This was disabled at some point during bringup to make life easier for
  the lab folks trying to debug NVLink issues. This hack really should
  have never made it out into the wild though, so we now have the
  following situation occuring in the field:

  1) A bad happens
  2) The host kernel recieves an unrecoverable HMI and calls into OPAL to
     request a platform reboot.
  3) OPAL rejects the reboot attempt and returns to the kernel with
     OPAL_PARAMETER.
  4) Kernel panics and attempts to kexec into a kdump kernel.

  A side effect of the HMI seems to be CPUs becoming stuck which results
  in the initialisation of the kdump kernel taking a extremely long time
  (6+ hours). It's also been observed that after performing a dump the
  kdump kernel then crashes itself because OPAL has ended up in a bad
  state as a side effect of the HMI.

  All up, it's not very good so re-enable the software checkstop by
  default. If people still want to turn it off they can using the nvram
  override.


CAPI2
^^^^^
- capp/phb4: Prevent HMI from getting triggered when disabling CAPP

  While disabling CAPP an HMI gets triggered as soon as ETU is put in
  reset mode. This is caused as before we can disabled CAPP, it detects
  PHB link going down and triggers an HMI requesting Opal to perform
  CAPP recovery. This has an un-intended side effect of spamming the
  Opal logs with malfunction alert messages and may also confuse the
  user.

  To prevent this we mask the CAPP FIR error 'PHB Link Down' Bit(31)
  when we are disabling CAPP just before we put ETU in reset in
  phb4_creset(). Also now since bringing down the PHB link now wont
  trigger an HMI and CAPP recovery, hence we manually set the
  PHB4_CAPP_RECOVERY flag on the phb to force recovery during creset.

- phb4/capp: Implement sequence to disable CAPP and enable fast-reset

  We implement h/w sequence to disable CAPP in disable_capi_mode() and
  with it also enable fast-reset for CAPI mode in phb4_set_capi_mode().

  Sequence to disable CAPP is executed in three phases. The first two
  phase is implemented in disable_capi_mode() where we reset the CAPP
  registers followed by PEC registers to their init values. The final
  third final phase is to reset the PHB CAPI Compare/Mask Register and
  is done in phb4_init_ioda3(). The reason to move the PHB reset to
  phb4_init_ioda3() is because by the time Opal PCI reset state machine
  reaches this function the PHB is already un-fenced and its
  configuration registers accessible via mmio.
- capp/phb4: Force CAPP to PCIe mode during kernel shutdown

  This patch introduces a new opal syncer for PHB4 named
  phb4_host_sync_reset(). We register this opal syncer when CAPP is
  activated successfully in phb4_set_capi_mode() so that it will be
  called at kernel shutdown during fast-reset.

  During kernel shutdown the function will then repeatedly call
  phb->ops->set_capi_mode() to switch switch CAPP to PCIe mode. In case
  set_capi_mode() indicates its OPAL_BUSY, which indicates that CAPP is
  still transitioning to new state; it calls slot->ops.run_sm() to
  ensure that Opal slot reset state machine makes forward progress.


Witherspoon Platform
^^^^^^^^^^^^^^^^^^^^
- platforms/witherspoon: Make PCIe shared slot error message more informative

  If we're missing chips for some reason, we print a warning when configuring
  the PCIe shared slot.

  The warning doesn't really make it clear what "shared slot" is, and if it's
  printed, it'll come right after a bunch of messages about NPU setup, so
  let's clarify the message to explicitly mention PCI.
- witherspoon: Add nvlink2 interconnect information

  See :ref:`skiboot-6.3-new-features` for details.

Zaius Platform
^^^^^^^^^^^^^^

- zaius: Add BMC description

  Frederic reported that Zaius was failing with a NULL dereference when
  trying to initialise IPMI HIOMAP. It turns out that the BMC wasn't
  described at all, so add a description.

p9dsu platform
^^^^^^^^^^^^^^
- p9dsu: Fix p9dsu default variant

  Add the default when no riser_id is returned from the ipmi query.

  Allow a little more time for BMC reply and cleanup some label strings.


PCIe
----

See :ref:`skiboot-6.3-power9` for POWER9 specific PCIe changes.

- core/pcie-slot: Don't bail early in the power on case

  Exiting early in the power off case makes sense since we can't disable
  slot power (or assert PERST) for suprise hotplug slots. However, we
  should not exit early in the power-on case since it's possible slot
  power may have been disabled (or just not enabled at boot time).
- firenze-pci: Always init slot info from LXVPD

  We can slot information from the LXVPD without having power control
  information about that slot. This patch changes the init path so that
  we always override the add_properties() call rather than only when we
  have power control information about the slot.
- fsp/lxvpd: Print more LXVPD slot information

  Useful to know since it changes the behaviour of the slot core.
- core/pcie-slot: Set power state from the PWRCTL flag

  For some reason we look at the power control indicator and use that to
  determine if the slot is "off" rather than the power control flag that
  is used to power down the slot.

  While we're here change the default behaviour so that the slot is
  assumed to be powered on if there's no slot capability, or if there's
  no power control available.
- core/pci: Increase the max slot string size

  The maximum string length for the slot label / device location code in
  the PCI summary is currently 32 characters. This results in some IBM
  location codes being truncated due to their length, e.g. ::

    PHB#0001:02:11.0 [SWDN]  SLOT=C11  x8
    PHB#0001:13:00.0 [EP  ] *snip* LOC_CODE=U78D3.ND1.WZS004A-P1-C
    PHB#0001:13:00.1 [EP  ] *snip* LOC_CODE=U78D3.ND1.WZS004A-P1-C
    PHB#0001:13:00.2 [EP  ] *snip* LOC_CODE=U78D3.ND1.WZS004A-P1-C
    PHB#0001:13:00.3 [EP  ] *snip* LOC_CODE=U78D3.ND1.WZS004A-P1-C

  Which obscure the actual location of the card, and it looks bad. This
  patch increases the maximum length of the label string to 80 characters
  since that's the maximum length for a location code.


Since v6.3-rc3:

- pci: Try harder to add meaningful ibm,loc-code

  We keep the existing logic of looking to the parent for the slot-label or
  slot-location-code, but we add logic to (if all that fails) we look
  directly for the slot-location-code (as this should give us the correct
  loc code for things directly under the PHB), and otherwise we just look
  for a loc-code.

  The applicable bit of PAPR here is:

      R1–12.1–1. Each instance of a hardware entity (FRU) has a platform
      unique location code and any node in the OF
      device tree that describes a part of a hardware entity must include the
      “ibm,loc-code” property with a
      value that represents the location code for that hardware entity.

  which we weren't really fully obeying at any recent (ever?) point in
  time. Now we should do okay, at least for PCI.

Since v6.3-rc2:
- core/pci: Use PHB io-base-location by default for PHB slots

  On witherspoon only the GPU slots and the three pluggable PCI slots
  (SLOT0, 1, 2) have platform defined slot names. For builtin devices such
  as the SATA controller or the PLX switch that fans out to the GPU slots
  we have no location codes which some people consider an issue.

  This patch address the problem by making the ibm,slot-location-code for
  the root port device default to the ibm,io-base-location-code which is
  typically the location code for the system itself.

  e.g. ::

    pciex@600c3c0100000/ibm,loc-code
                     "UOPWR.0000000-Node0-Proc0"

    pciex@600c3c0100000/pci@0/ibm,loc-code
                     "UOPWR.0000000-Node0-Proc0"

    pciex@600c3c0100000/pci@0/usb-xhci@0/ibm,loc-code
                     "UOPWR.0000000-Node0"

  The PHB node, and the root complex nodes have a loc code of the
  processor they are attached to, while the usb-xhci device under the
  root port has a location code of the system itself.

- hw/phb4: Read ibm,loc-code from PBCQ node

  On P9 the PBCQs are subdivided by stacks which implement the PCI Express
  logic. When phb4 was forked from phb3 most of the properties that were
  in the pbcq node moved into the stack node, but ibm,loc-code was not one
  of them. This patch fixes the phb4 init sequence to read the base
  location code from the PBCQ node (parent of the stack node) rather than
  the stack node itself.


.. _skiboot-6.3-OpenCAPI:

OpenCAPI
--------
- npu2/hw-procedures: Fix parallel zcal for opencapi

  For opencapi, we currently do impedance calibration when initializing
  the PHY for the device, which could run in parallel if we have
  multiple opencapi devices. But if 2 devices are on the same
  obus, the 2 calibration sequences could overlap, which likely yields
  bad results and is useless anyway since it only needs to be done once
  per obus.

  This patch splits the opencapi PHY reset in 2 parts:

  - a 'init' part called serially at boot. That's when zcal is done. If
    we have 2 devices on the same socket, the zcal won't be redone,
    since we're called serially and we'll see it has already be done for
    the obus
  - a 'reset' part called during fundamental reset as a prereq for link
    training. It does the PHY setup for a set of lanes and the dccal.

  The PHY team confirmed there's no dependency between zcal and the
  other reset steps and it can be moved earlier.
- npu2-hw-procedures: Fix zcal in mixed opencapi and nvlink mode

  The zcal procedure needs to be run once per obus. We keep track of
  which obus is already calibrated in an array indexed by the obus
  number. However, the obus number is inferred from the brick index,
  which works well for nvlink but not for opencapi.

  Create an obus_index() function, which, from a device, returns the
  correct obus index, irrespective of the device type.
- npu2-opencapi: Fix adapter reset when using 2 adapters

  If two opencapi adapters are on the same obus, we may try to train the
  two links in parallel at boot time, when all the PCI links are being
  trained. Both links use the same i2c controller to handle the reset
  signal, so some care is needed to make sure resetting one doesn't
  interfere with the reset of the other. We need to keep track of the
  current state of the i2c controller (and use locking).

  This went mostly unnoticed as you need to have 2 opencapi cards on the
  same socket and links tended to train anyway because of the retries.
- npu2-opencapi: Extend delay after releasing reset on adapter

  Give more time to the FPGA to process the reset signal. The previous
  delay, 5ms, is too short for newer adapters with bigger FPGAs. Extend
  it to 250ms.
  Ultimately, that delay will likely end up being added to the opencapi
  specification, but we are not there yet.
- npu2-opencapi: ODL should be in reset when enabled

  We haven't hit any problem so far, but from the ODL designer, the ODL
  should be in reset when it is enabled.

  The ODL remains in reset until we start a fundamental reset to
  initiate link training. We still assert and deassert the ODL reset
  signal as part of the normal procedure just before training the
  link. Asserting is therefore useless at boot, since the ODL is already
  in reset, but we keep it as it's only a scom write and it's needed
  when we reset/retrain from the OS.
- npu2-opencapi: Keep ODL and adapter in reset at the same time

  Split the function to assert and deassert the reset signal on the ODL,
  so that we can keep the ODL in reset while we reset the adapter,
  therefore having a window where both sides are in reset.

  It is actually not required with our current DLx at boot time, but I
  need to split the ODL reset function for the following patch and it
  will become useful/required later when we introduce resetting an
  opencapi link from the OS.
- npu2-opencapi: Setup perf counters to detect CRC errors

  It's possible to set up performance counters for the PLL to detect
  various conditions for the links in nvlink or opencapi mode. Since
  those counters are currently unused, let's configure them when an obus
  is in opencapi mode to detect CRC errors on the link. Each link has
  two counters:
  - CRC error detected by the host
  - CRC error detected by the DLx (NAK received by the host)

  We also dump the counters shortly after the link trains, but they can
  be read multiple times through cronus, pdbg or linux. The counters are
  configured to be reset after each read.

Since v6.3-rc1:

- opal/hmi: Never trust a cow!

  With opencapi, it's fairly common to trigger HMIs during AFU
  development on the FPGA, by not replying in time to an NPU command,
  for example. So shift the blame reported by that cow to avoid crowding
  my mailbox.
- hw/npu2: Dump (more) npu2 registers on link error and HMIs

  We were already logging some NPU registers during an HMI. This patch
  cleans up a bit how it is done and separates what is global from what
  is specific to nvlink or opencapi.

  Since we can now receive an error interrupt when an opencapi link goes
  down unexpectedly, we also dump the NPU state but we limit it to the
  registers of the brick which hit the error.

  The list of registers to dump was worked out with the hw team to
  allow for proper debugging. For each register, we print the name as
  found in the NPU workbook, the scom address and the register value.
- hw/npu2: Report errors to the OS if an OpenCAPI brick is fenced

  Now that the NPU may report interrupts due to the link going down
  unexpectedly, report those errors to the OS when queried by the
  'next_error' PHB callback.

  The hardware doesn't support recovery of the link when it goes down
  unexpectedly. So we report the PHB as dead, so that the OS can log the
  proper message, notify the drivers and take the devices down.
- hw/npu2: Fix OpenCAPI PE assignment

  When we support mixing NVLink and OpenCAPI devices on the same NPU, we're
  going to have to share the same range of 16 PE numbers between NVLink and
  OpenCAPI PHBs.

  For OpenCAPI devices, PE assignment is only significant for determining
  which System Interrupt Log register is used for a particular brick - unlike
  NVLink, it doesn't play any role in determining how links are fenced.

  Split the PE range into a lower half which is used for NVLink, and an upper
  half that is used for OpenCAPI, with a fixed PE number assigned per brick.

  As the PE assignment for OpenCAPI devices is fixed, set the PE once
  during device init and then ignore calls to the set_pe() operation.

- opal-api: Reserve 2 OPAL API calls for future OpenCAPI LPC use

  OpenCAPI Lowest Point of Coherency (LPC) memory is going to require
  some extra OPAL calls to set up NPU BARs. These calls will most likely be
  called OPAL_NPU_LPC_ALLOC and OPAL_NPU_LPC_RELEASE, we're not quite ready
  to upstream that code yet though.



NVLINK2
-------
- npu2: Allow ATSD for LPAR other than 0

  Each XTS MMIO ATSD# register is accompanied by another register -
  XTS MMIO ATSD0 LPARID# - which controls LPID filtering for ATSD
  transactions.

  When a host system passes a GPU through to a guest, we need to enable
  some ATSD for an LPAR. At the moment the host assigns one ATSD to
  a NVLink bridge and this maps it to an LPAR when GPU is assigned to
  the LPAR. The link number is used for an ATSD index.

  ATSD6&7 stay mapped to the host (LPAR=0) all the time which seems to be
  acceptable price for the simplicity.
- npu2: Add XTS_BDF_MAP wildcard refcount

  Currently PID wildcard is programmed into the NPU once and never cleared
  up. This works for the bare metal as MSR does not change while the host
  OS is running.

  However with the device virtualization, we need to keep track of wildcard
  entries use and clear them up before switching a GPU from a host to
  a guest or vice versa.

  This adds refcount to a NPU2, one counter per wildcard entry. The index
  is a short lparid (4 bits long) which is allocated in opal_npu_map_lpar()
  and should be smaller than NPU2_XTS_BDF_MAP_SIZE (defined as 16).

Since v6.3-rc2:
- npu2: Disable Probe-to-Invalid-Return-Modified-or-Owned snarfing by default

  V100 GPUs are known to violate NVLink2 protocol in some cases (one is when
  memory was accessed by the CPU and they by GPU using so called block
  linear mapping) and issue double probes to NPU which can cope with this
  problem only if CONFIG_ENABLE_SNARF_CPM ("disable/enable Probe.I.MO
  snarfing a cp_m") is not set in the CQ_SM Misc Config register #0.
  If the bit is set (which is the case today), NPU issues the machine
  check stop.

  The snarfing feature is designed to detect 2 probes in flight and combine
  them into one.

  This adds a new "opal-npu2-snarf-cpm" nvram variable which controls
  CONFIG_ENABLE_SNARF_CPM for all NVLinks to prevent the machine check
  stop from happening.

  This disables snarfing by default as otherwise a broken GPU driver can
  crash the entire box even when a GPU is passed through to a guest.
  This provides a dial to allow regression tests (might be useful for
  a bare metal). To enable snarfing, the user needs to run: ::

    sudo nvram -p ibm,skiboot --update-config opal-npu2-snarf-cpm=enable

  and reboot the host system.

- hw/npu2: Show name of opencapi error interrupts


Debugging and simulation
------------------------

- external/mambo: Error out if kernel is too large

  If you're trying to boot a gigantic kernel in mambo (which you can
  reproduce by building a kernel with CONFIG_MODULES=n) you'll get
  misleading errors like: ::

    WARNING: 0: (0): [0:0]: Invalid/unsupported instr 0x00000000[INVALID]
    WARNING: 0: (0):  PC(EA): 0x0000000030000010 PC(RA):0x0000000030000010 MSR: 0x9000000000000000 LR: 0x0000000000000000
    WARNING: 0: (0):  numInstructions = 0
    WARNING: 1: (1): [0:0]: Invalid/unsupported instr 0x00000000[INVALID]
    WARNING: 1: (1):  PC(EA): 0x0000000000000E40 PC(RA):0x0000000000000E40 MSR: 0x9000000000000000 LR: 0x0000000000000000
    WARNING: 1: (1):  numInstructions = 1
    WARNING: 1: (1): Interrupt to 0x0000000000000E40 from 0x0000000000000E40
    INFO: 1: (2): ** Execution stopped: Continuous Interrupt, Instruction caused exception,  **

  So add an error to skiboot.tcl to warn the user before this happens.
  Making PAYLOAD_ADDR further back is one way to do this but if there's a
  less gross way to generally work around this very niche problem, I can
  suggest that instead.
- external/mambo: Populate kernel-base-address in the DT

  skiboot.tcl defines PAYLOAD_ADDR as 0x20000000, which is the default in
  skiboot.  This is also the default in skiboot unless kernel-base-address
  is set in the device tree.

  If you change PAYLOAD_ADDR to something else for mambo, skiboot won't
  see it because it doesn't set that DT property, so fix it so that it does.
- external/mambo: allow CPU targeting for most debug utils

  Debug util functions target CPU 0:0:0 by default Some can be
  overidden explicitly per invocation, and others can't at all.
  Even for those that can be overidden, it is a pain to type
  them out when you're debugging a particular thread.

  Provide a new 'target' function that allows the default CPU
  target to be changed. Wire that up that default to all other utils.
  Provide a new 'S' step command which only steps the target CPU.
- qemu: bt device isn't always hanging off /

  Just use the normal for_each_compatible instead.

  Otherwise in the qemu model as executed by op-test,
  we wouldn't go down the astbmc_init() path, thus not having flash.
- devicetree: Add p9-simics.dts

  Add a p9-based devicetree that's suitable for use with Simics.
- devicetree: Move power9-phb4.dts

  Clean up the formatting of power9-phb4.dts and move it to
  external/devicetree/p9.dts. This sets us up to include it as the basis
  for other trees.
- devicetree: Add nx node to power9-phb4.dts

  A (non-qemu) p9 without an nx node will assert in p9_darn_init(): ::

      dt_for_each_compatible(dt_root, nx, "ibm,power9-nx")
              break;
      if (!nx) {
              if (!dt_node_is_compatible(dt_root, "qemu,powernv"))
                    assert(nx);
              return;
      }

  Since NX is this essential, add it to the device tree.
- devicetree: Fix typo in power9-phb4.dts

  Change "impi" to "ipmi".
- devicetree: Fix syntax error in power9-phb4.dts

  Remove the extra space causing this: ::

      Error: power9-phb4.dts:156.15-16 syntax error
      FATAL ERROR: Unable to parse input tree
- core/init: enable machine check on secondaries

  Secondary CPUs currently run with MSR[ME]=0 during boot, whih means
  if they take a machine check, the system will checkstop.

  Enable ME where possible and allow them to print registers.

Utilities
---------
- pflash: Don't try update RO ToC

  In the future it's likely the ToC will be marked as read-only. Don't
  error out by assuming its writable.
- pflash: Support encoding/decoding ECC'd partitions

  With the new --ecc option, pflash can add/remove ECC when
  reading/writing flash partitions protected by ECC.

  This is *not* flawless with current PNORs out in the wild though, as
  they do not typically fill the whole partition with valid ECC data, so
  you have to know how big the valid ECC'd data is and specify the size
  manually. Note that for some partitions this is pratically impossible
  without knowing the details of the content of the partition.

  A future patch is likely to introduce an option to "stop reading data
  when ECC starts failing and assume everything is okay rather than error
  out" to support reading the "valid" data from existing PNOR images.

Since v6.3-rc2:

- opal-prd: Fix memory leak in is-fsp-system check
- opal-prd: Check malloc return value