1.. _skiboot-5.9: 2 3skiboot-5.9 4=========== 5 6skiboot v5.9 was released on Tuesday October 31st 2017. It is the first 7release of skiboot 5.9 and becomes the new stable release 8of skiboot following the 5.8 release, first released August 31st 2017. 9In this cyle we have had five release candidate releases, mostly centered 10around bug fixing for POWER9 platforms. 11 12This release should be considered suitable for early-access POWER9 systems. 13 14skiboot v5.9 contains all bug fixes as of :ref:`skiboot-5.4.8` 15and :ref:`skiboot-5.1.21` (the currently maintained stable releases). 16There may be some 5.9.x stable releases, depending on what issues are found. 17 18For how the skiboot stable releases work, see :ref:`stable-rules` for details. 19 20Over :ref:`skiboot-5.8`, we have the following changes: 21 22New Features 23------------ 24 25POWER8 26^^^^^^ 27- fast-reset by default (if possible) 28 29 Currently, this is limited to POWER8 systems. 30 31 A normal reboot will, rather than doing a full IPL, go through a 32 fast reboot procedure. This reduces the "reboot to petitboot" time 33 from minutes to a handful of seconds. 34 35POWER9 36^^^^^^ 37 38Since :ref:`skiboot-5.9-rc3`: 39 40- occ-sensors : Add OCC inband sensor region to exports 41 (useful for debugging) 42 43Two SRESET fixes (see below for feature description): 44 45- core: direct-controls: Fix clearing of special wakeup 46 47 'special_wakeup_count' is incremented on successfully asserting 48 special wakeup. So we will never clear the special wakeup if we 49 check 'special_wakeup_count' to be zero. Fix this issue by checking 50 the 'special_wakeup_count' to 1 in dctl_clear_special_wakeup(). 51- core/direct-controls: increase special wakeup timeout on POWER9 52 53 Some instances have been observed where the special wakeup assert 54 times out. The current timeout is too short for deeper sleep states. 55 Hostboot uses 100ms, so match that. 56 57 58Since :ref:`skiboot-5.9-rc2`: 59- cpu: Add OPAL_REINIT_CPUS_TM_SUSPEND_DISABLED 60 61 Add a new CPU reinit flag, "TM Suspend Disabled", which requests that 62 CPUs be configured so that TM (Transactional Memory) suspend mode is 63 disabled. 64 65 Currently this always fails, because skiboot has no way to query the 66 state. A future hostboot change will add a mechanism for skiboot to 67 determine the status and return an appropriate error code. 68 69Since :ref:`skiboot-5.8`: 70 71- POWER9 power management during boot 72 73 Less power should be consumed during boot. 74- OPAL_SIGNAL_SYSTEM_RESET for POWER9 75 76 This implements OPAL_SIGNAL_SYSTEM_RESET, using scom registers to 77 quiesce the target thread and raise a system reset exception on it. 78 It has been tested on DD2 with stop0 ESL=0 and ESL=1 shallow power 79 saving modes. 80 81 DD1 is not implemented because it is sufficiently different as to 82 make support difficult. 83- Enable deep idle states for POWER9 84 85 - SLW: Add support for p9_stop_api 86 87 p9_stop_api's are used to set SPR state on a core wakeup form a deeper 88 low power state. p9_stop_api uses low level platform formware and 89 self-restore microcode to restore the sprs to requested values. 90 91 Code is taken from : 92 https://github.com/open-power/hostboot/tree/master/src/import/chips/p9/procedures/utils/stopreg 93 - SLW: Removing timebase related flags for stop4 94 95 When a core enters stop4, it does not loose decrementer and time base. 96 Hence removing flags OPAL_PM_DEC_STOP and OPAL_PM_TIMEBASE_STOP. 97 - SLW: Allow deep states if homer address is known 98 99 Use a common variable has_wakeup_engine instead of has_slw to tell if 100 the: 101 - SLW image is populated in case of power8 102 - CME image is populated in case of power9 103 104 Currently we expect CME to be loaded if homer address is known ( except 105 for simulators) 106 - SLW: Configure self-restore for HRMOR 107 108 Make a stop api call using libpore to restore HRMOR register. HRMOR needs 109 to be cleared so that when thread exits stop, they arrives at linux 110 system_reset vector (0x100). 111 - SLW: Add opal_slw_set_reg support for power9 112 113 This OPAL call is made from Linux to OPAL to configure values in 114 various SPRs after wakeup from a deep idle state. 115- PHB4: CAPP recovery 116 117 CAPP recovery is initiated when a CAPP Machine Check is detected. 118 The capp recovery procedure is initiated via a Hypervisor Maintenance 119 interrupt (HMI). 120 121 CAPP Machine Check may arise from either an error that results in a PHB 122 freeze or from an internal CAPP error with CAPP checkstop FIR action. 123 An error that causes a PHB freeze will result in the link down signal 124 being asserted. The system continues running and the CAPP and PSL will 125 be re-initialized. 126 127 This implements CAPP recovery for POWER9 systems 128- Add ``wafer-location`` property for POWER9 129 130 Extract wafer-location from ECID and add property under xscom node. 131 - bits 64:71 are the chip x location (7:0) 132 - bits 72:79 are the chip y location (7:0) 133 134 Sample output: :: 135 136 [root@wsp xscom@623fc00000000]# lsprop ecid 137 ecid 019a00d4 03100718 852c0000 00fd7911 138 [root@wsp xscom@623fc00000000]# lsprop wafer-location 139 wafer-location 00000085 0000002c 140- Add ``wafer-id`` property for POWER9 141 142 Wafer id is derived from ECID data. 143 - bits 4:63 are the wafer id ( ten 6 bit fields each containing a code) 144 145 Sample output: :: 146 147 [root@wsp xscom@623fc00000000]# lsprop ecid 148 ecid 019a00d4 03100718 852c0000 00fd7911 149 [root@wsp xscom@623fc00000000]# lsprop wafer-id 150 wafer-id "6Q0DG340SO" 151- Add ``ecid`` property under ``xscom`` node for POWER9. 152 Sample output: :: 153 154 [root@wsp xscom@623fc00000000]# lsprop ecid 155 ecid 019a00d4 03100718 852c0000 00fd7911 156- Add ibm,firmware-versions device tree node 157 158 In P8, hostboot provides mini device tree. It contains ``/ibm,firmware-versions`` 159 node which has various firmware component version details. 160 161 In P9, OPAL is building device tree. This patch adds support to parse VERSION 162 section of PNOR and create ``/ibm,firmware-versions`` device tree node. 163 164 Sample output: :: 165 166 /sys/firmware/devicetree/base/ibm,firmware-versions # lsprop . 167 occ "6a00709" 168 skiboot "v5.7-rc1-p344fb62" 169 buildroot "2017.02.2-7-g23118ce" 170 capp-ucode "9c73e9f" 171 petitboot "v1.4.3-p98b6d83" 172 sbe "02021c6" 173 open-power "witherspoon-v1.17-128-gf1b53c7-dirty" 174 .... 175 .... 176 177POWER9 178------ 179Since :ref:`skiboot-5.9-rc5`: 180 181- Suppress XSCOM chiplet-offline errors on P9 182 183 Workaround on P9: PRD does operations it *knows* will fail with this 184 error to work around a hardware issue where accesses via the PIB 185 (FSI or OCC) work as expected, accesses via the ADU (what xscom goes 186 through) do not. The chip logic will always return all FFs if there 187 is any error on the scom. 188- asm/head: initialize preferred DSCR value 189 190 POWER7/8 use DSCR=0. POWER9 preferred value has "stride-N" enabled. 191 192Since :ref:`skiboot-5.9-rc4`: 193- opal/hmi: Workaround Power9 hw logic bug for couple of TFMR TB errors. 194- opal/hmi: Fix TB reside and HDEC parity error recovery for power9 195 196Since :ref:`skiboot-5.9-rc2`: 197- hw/imc: Fix IMC Catalog load for DD2.X processors 198 199Since :ref:`skiboot-5.9-rc1`: 200- xive: Fix VP free block group mode false-positive parameter check 201 202 The check to ensure the buddy allocation idx is aligned to its 203 allocation order was not taking into account the allocation split. 204 This would result in opal_xive_free_vp_block failures despite 205 giving the same value as returned by opal_xive_alloc_vp_block. 206 207 E.g., starting then stopping 4 KVM guests gives the following pattern 208 in the host: :: 209 210 opal_xive_alloc_vp_block(5)=0x45000020 211 opal_xive_alloc_vp_block(5)=0x45000040 212 opal_xive_alloc_vp_block(5)=0x45000060 213 opal_xive_alloc_vp_block(5)=0x45000080 214 opal_xive_free_vp_block(0x45000020)=-1 215 opal_xive_free_vp_block(0x45000040)=0 216 opal_xive_free_vp_block(0x45000060)=-1 217 opal_xive_free_vp_block(0x45000080)=0 218 219- hw/imc: pause microcode at boot 220 221 IMC nest counters has both in-band (ucode access) and out of 222 band access to it. Since not all nest counter configurations 223 are supported by ucode, out of band tools are used to characterize 224 other configuration. 225 226 So it is prefer to pause the nest microcode at boot to aid the 227 nest out of band tools. If the ucode not paused and OS does not 228 have IMC driver support, then out to band tools will race with 229 ucode and end up getting undesirable values. Patch to check and 230 pause the ucode at boot. 231 232 OPAL provides APIs to control IMC counters. OPAL_IMC_COUNTERS_INIT 233 is used to initialize these counters at boot. OPAL_IMC_COUNTERS_START 234 and OPAL_IMC_COUNTERS_STOP API calls should be used to start and pause 235 these IMC engines. `doc/opal-api/opal-imc-counters.rst` details the 236 OPAL APIs and their usage. 237- hdata/i2c: update the list of known i2c devs 238 239 This updates the list of known i2c devices - as of HDAT spec v10.5e - so 240 that they can be properly identified during the hdat parsing. 241- hdata/i2c: log unknown i2c devices 242 243 An i2c device is unknown if either the i2c device list is outdated or 244 the device is marked as unknown (0xFF) in the hdat. 245 246Since :ref:`skiboot-5.8`: 247 248- Disable Transactional Memory on Power9 DD 2.1 249 250 Update pa_features_p9[] to disable TM (Transactional Memory). On DD 2.1 251 TM is not usable by Linux without other workarounds, so skiboot must 252 disable it. 253- xscom: Do not print error message for 'chiplet offline' return values 254 255 xscom_read/write operations returns CHIPLET_OFFLINE when chiplet is offline. 256 Some multicast xscom_read/write requests from HBRT results in xscom operation 257 on offline chiplet(s) and printing below warnings in OPAL console: :: 258 259 [ 135.036327572,3] XSCOM: Read failed, ret = -14 260 [ 135.092689829,3] XSCOM: Read failed, ret = -14 261 262 Some SCOM users can deal correctly with this error code (notably opal-prd), 263 so the error message is (in practice) erroneous. 264- IMC: Fix the core_imc_event_mask 265 266 CORE_IMC_EVENT_MASK is a scom that contains bits to control event sampling for 267 different machine state for core imc. The current event-mask setting sample 268 events only on host kernel (hypervisor) and host userspace. 269 270 Patch to enable the sampling of events in other machine states (like guest 271 kernel and guest userspace). 272- IMC: Update the nest_pmus array with occ/gpe microcode uav updates 273 274 OOC/gpe nest microcode maintains the list of individual nest units 275 supported. Sync the recent updates to the UAV with nest_pmus array. 276 277 For reference occ/gpr microcode link for the UAV: 278 https://github.com/open-power/occ/blob/master/src/occ_gpe1/gpe1_24x7.h 279- Parse IOSLOT information from HDAT 280 281 Add structure definitions that describe the physical PCIe topology of 282 a system and parse them into the device-tree based PCIe slot 283 description. 284- idle: user context state loss flags fix for stop states 285 286 The "lite" stop variants with PSSCR[ESL]=PSSCR[EC]=1 do not lose user 287 context, while the non-lite variants do (ESL: enable state loss). 288 289 Some of the POWER9 idle states had these wrong. 290 291CAPI 292^^^^ 293- POWER9 DD2 update 294 295 The CAPI initialization sequence has been updated in DD2. 296 This patch adapts to the changes, retaining compatibility with DD1. 297 The patch includes some changes to DD1 fix-ups as well. 298- Load CAPP microcode for POWER9 DD2.0 and DD2.1 299- capi: Mask Psl Credit timeout error for POWER9 300 301 Mask the PSL credit timeout error in CAPP FIR Mask register 302 bit(46). As per the h/w team this error is now deprecated and shouldn't 303 cause any fir-action for P9. 304 305NVLINK2 306^^^^^^^ 307 308A notabale change is that we now generate the device tree description of 309NVLINK based on the HDAT we get from hostboot. Since Hostboot will generate 310HDAT based on VPD, you now *MUST* have correct VPD programmed or we will 311*default* to a Sequoia layout, which will lead to random problems if you 312are not booting a Sequoia Witherspoon planar. In the case of booting with 313old VPD and/or Hostboot, we print a **giant scary warning** in order to scare you. 314 315Since :ref:`skiboot-5.9-rc2`: 316- Revert "npu2: Add vendor cap for IRQ testing" 317 318 This reverts commit 9817c9e29b6fe00daa3a0e4420e69a97c90eb373 which seems to 319 break setting the PCI dev flag and the link number in the PCIe vendor 320 specific config space. This leads to the device driver attempting to 321 re-init the DL when it shouldn't which can cause HMI's. 322 323Since :ref:`skiboot-5.8`: 324 325- npu2: Read slot label from the HDAT link node 326 327 Binding GPU to emulated NPU PCI devices is done using the slot labels 328 since the NPU devices do not have a patching slot node we need to 329 copy the label in here. 330 331- npu2: Copy link speed from the npu HDAT node 332 333 This needs to be in the PCI device node so the speed of the NVLink 334 can be passed to the GPU driver. 335- npu2: hw-procedures: Add settings to PHY_RESET 336 337 Set a few new values in the PHY_RESET procedure, as specified by our 338 updated programming guide documentation. 339- Parse NVLink information from HDAT 340 341 Add the per-chip structures that descibe how the A-Bus/NVLink/OpenCAPI 342 phy is configured. This generates the npu@xyz nodes for each chip on 343 systems that support it. 344- npu2: Add vendor cap for IRQ testing 345 346 Provide a way to test recoverable data link interrupts via a new 347 vendor capability byte. 348- npu2: Enable recoverable data link (no-stall) interrupts 349 350 Allow the NPU2 to trigger "recoverable data link" interrupts. 351 352- npu2: Implement basic FLR (Function Level Reset) 353- npu2: hw-procedures: Update PHY DC calibration procedure 354- npu2: hw-procedures: Change rx_pr_phase_step value 355 356XIVE 357^^^^ 358- xive: Fix opal_xive_dump_tm() to access W2 properly. 359 The HW only supported limited access sizes. 360- xive: Make opal_xive_allocate_irq() properly try all chips 361 362 When requested via OPAL_XIVE_ANY_CHIP, we need to try all 363 chips. We first try the current one (on which the caller 364 sits) and if that fails, we iterate all chips until the 365 allocation succeeds. 366- xive: Fix initialization & cleanup of HW thread contexts 367 368 Instead of trying to "pull" everything and clear VT (which didn't 369 work and caused some FIRs to be set), instead just clear and then 370 set the PTER thread enable bit. This has the side effect of 371 completely resetting the corresponding thread context. 372 373 This fixes the spurrious XIVE FIRs reported by PRD and fircheck 374- xive: Add debug option for detecting misrouted IPI in emulation 375 376 This is high overhead so we don't enable it by default even 377 in debug builds, it's also a bit messy, but it allowed me to 378 detect and debug a locking issue earlier so it can be useful. 379- xive: Increase the interrupt "gap" on debug builds 380 381 We normally allocate IPIs from 0x10. Make that 0x1000 on debug 382 builds to limit the chances of overlapping with Linux interrupt 383 numbers which makes debugging code that confuses them easier. 384 385 Also add a warning in emulation if we get an interrupt in the 386 queue whose number is below the gap. 387- xive: Fix locking around cache scrub & watch 388 389 Thankfully the missing locking only affects debug code and 390 init code that doesn't run concurrently. Also adds a DEBUG 391 option that checks the lock is properly held. 392- xive: Workaround HW issue with scrub facility 393 394 Without this, we sometimes don't observe from a CPU the 395 values written to the ENDs or NVTs via the cache watch. 396- xive: Add exerciser for cache watch/scrub facility in DEBUG builds 397- xive: Make assertion in xive_eq_for_target() more informative 398- xive: Add debug code to check initial cache updates 399- xive: Ensure pressure relief interrupts are disabled 400 401 We don't use them and we hijack the VP field with their 402 configuration to store the EQ reference, so make sure the 403 kernel or guest can't turn them back on by doing MMIO 404 writes to ACK# 405- xive: Don't try setting the reserved ACK# field in VPs 406 407 That doesn't work, the HW doesn't implement it in the cache 408 watch facility anyway. 409- xive: Remove useless memory barriers in VP/EQ inits 410 411 We no longer update "live" memory structures, we use a temporary 412 copy on the stack and update the actual memory structure using 413 the cache watch, so those barriers are pointless. 414 415PHB4 416^^^^ 417Since :ref:`skiboot-5.9-rc4`: 418 419- phb4: Escalate freeze to fence to avoid checkstop 420 421 Freeze events such as MMIO loads can cause the PHB to lose it's 422 limited powerbus credits. If all credits are used and a further MMIO 423 will cause a checkstop. 424 425 To work around this, we escalate the troublesome freeze events to a 426 fence. The fence will cause a full PHB reset which resets the powerbus 427 credits and avoids the checkstop. 428- phb4: Update some init registers 429 430 New inits based on next PHB4 workbook. Increases some timeouts to 431 avoid some spurious error conditions. 432- phb4: Enable PHB MMIO in phb4_root_port_init() 433 434 Linux EEH flow is somewhat broken. It saves the PCIe config space of 435 the PHB on boot, which it then uses to restore on EEH recovery. It 436 does this to restore MMIO bars and some other pieces. 437 438 Unfortunately this save is done before any drivers are bound to 439 devices under the PHB. A number of other things are configured in the 440 PHB after drivers start, hence some configuration space settings 441 aren't saved correctly. These include bus master and MMIO bits in the 442 command register. 443 444 Linux tried to hack around this in this linux commit 445 ``bf898ec5cb`` powerpc/eeh: Enable PCI_COMMAND_MASTER for PCI bridges 446 This sets the bus master bit but ignores the MMIO bit. 447 448 Hence we lose MMIO after a full PHB reset. This causes the next MMIO 449 access to the device to fail and for us to perform a PE freeze 450 recovery, which still doesn't set the MMIO bit and hence we still 451 fail. 452 453 This works around this by forcing MMIO on during 454 phb4_root_port_init(). 455 456 With this we can recovery from a PHB fence event on POWER9. 457- phb4: Reduce link degraded message log level to debug 458 459 If we hit this message we'll retry and fix the problem. If we run out 460 of retries and can't fix the problem, we'll still print a log message 461 at error level indicating a problem. 462- phb4: Fix GEN3 for DD2.00 463 464 In this fix: ``62ac7631ae phb4: Fix PCIe GEN4 on DD2.1 and above`` 465 We fixed DD2.1 GEN4 but broke DD2.00 as GEN3. 466 467 This fixes DD2.00 back to GEN3. This time for sure! 468 469Since :ref:`skiboot-5.9-rc3`: 470- phb4: Fix PCIe GEN4 on DD2.1 and above 471 472 In this change: 473 eef0e197ab PHB4: Default to PCIe GEN3 on POWER9 DD2.00 474 475 We clamped DD2.00 parts to GEN3 but unfortunately this change also 476 applies to DD2.1 and above. 477 478 This fixes this to only apply to DD2.00. 479 480Since :ref:`skiboot-5.8`: 481 482- phb4: Mask RXE_ARB: DEC Stage Valid Error 483 484 Change the inits to mask out the RXE ARB: DEC Stage Valid Error (bit 485 370. This has been a fatal error but should be informational only. 486 487 This update will be in the next version of the phb4 workbook. 488- phb4: Add additional adapter to retrain whitelist 489 490 The single port version of the ConnectX-5 has a different device ID 0x1017. 491 Updated descriptions to match pciutils database. 492- PHB4: Default to PCIe GEN3 on POWER9 DD2.00 493 494 You can use the NVRAM override for DD2.00 screened parts. 495- phb4: Retrain link if degraded 496 497 On P9 Scale Out (Nimbus) DD2.0 and Scale in (Cumulus) DD1.0 (and 498 below) the PCIe PHY can lockup causing training issues. This can cause 499 a degradation in speed or width in ~5% of training cases (depending on 500 the card). This is fixed in later chip revisions. This issue can also 501 cause PCIe links to not train at all, but this case is already 502 handled. 503 504 This patch checks if the PCIe link has trained optimally and if not, 505 does a full PHB reset (to fix the PHY lockup) and retrain. 506 507 One complication is some devices are known to train degraded unless 508 device specific configuration is performed. Because of this, we only 509 retrain when the device is in a whitelist. All devices in the current 510 whitelist have been testing on a P9DSU/Boston, ZZ and Witherspoon. 511 512 We always gather information on the link and print it in the logs even 513 if the card is not in the whitelist. 514 515 For testing purposes, there's an nvram to retry all PCIe cards and all 516 P9 chips when a degraded link is detected. The new option is 517 'pci-retry-all=true' which can be set using: 518 `nvram -p ibm,skiboot --update-config pci-retry-all=true`. 519 This option may increase the boot time if used on a badly behaving 520 card. 521 522 523IBM FSP platforms 524----------------- 525 526Since :ref:`skiboot-5.9-rc5`: 527- FSP/CONSOLE: Disable notification on unresponsive consoles 528 529 Commit fd6b71fc fixed the situation where ipmi console was open (hvc0) but got 530 data on different console (hvc1). 531 532 During FSP Reset/Reload OPAL closes all consoles. After Reset/Reload 533 complete FSP requests to open hvc1 and sends data on this. If hvc1 registration failed or not opened in host kernel then it will not read data and results in RCU stalls. 534 535 Note that this is workaround for older kernel where we don't have separate irq 536 for each console. Latest kernel works fine without this patch. 537 538Since :ref:`skiboot-5.9-rc1`: 539 540- FSP/CONSOLE: Limit number of error logging 541 542 Commit c8a7535f (FSP/CONSOLE: Workaround for unresponsive ipmi daemon) added 543 error logging when buffer is full. In some corner cases kernel may call this 544 function multiple time and we may endup logging error again and again. 545 546 This patch fixes it by generating error log only once. 547 548- FSP/CONSOLE: Fix fsp_console_write_buffer_space() call 549 550 Kernel calls fsp_console_write_buffer_space() to check console buffer space 551 availability. If there is enough buffer space to write data, then kernel will 552 call fsp_console_write() to write actual data. 553 554 In some extreme corner cases (like one explained in commit c8a7535f) 555 console becomes full and this function returns 0 to kernel (or space available 556 in console buffer < next incoming data size). Kernel will continue retrying 557 until it gets enough space. So we will start seeing RCU stalls. 558 559 This patch keeps track of previous available space. If previous space is same 560 as current means not enough space in console buffer to write incoming data. 561 It may be due to very high console write operation and slow response from FSP 562 -OR- FSP has stopped processing data (ex: because of ipmi daemon died). At this 563 point we will start timer with timeout of SER_BUFFER_OUT_TIMEOUT (10 secs). 564 If situation is not improved within 10 seconds means something went bad. Lets 565 return OPAL_RESOURCE so that kernel can drop console write and continue. 566- FSP/CONSOLE: Close SOL session during R/R 567 568 Presently we are not closing SOL and FW console sessions during R/R. Host will 569 continue to write to SOL buffer during FSP R/R. If there is heavy console write 570 operation happening during FSP R/R (like running `top` command inside console), 571 then at some point console buffer becomes full. fsp_console_write_buffer_space() 572 returns 0 (or less than required space to write data) to host. While one thread 573 is busy writing to console, if some other threads tries to write data to console 574 we may see RCU stalls (like below) in kernel. :: 575 576 [ 2082.828363] INFO: rcu_sched detected stalls on CPUs/tasks: { 32} (detected by 16, t=6002 jiffies, g=23154, c=23153, q=254769) 577 [ 2082.828365] Task dump for CPU 32: 578 [ 2082.828368] kworker/32:3 R running task 0 4637 2 0x00000884 579 [ 2082.828375] Workqueue: events dump_work_fn 580 [ 2082.828376] Call Trace: 581 [ 2082.828382] [c000000f1633fa00] [c00000000013b6b0] console_unlock+0x570/0x600 (unreliable) 582 [ 2082.828384] [c000000f1633fae0] [c00000000013ba34] vprintk_emit+0x2f4/0x5c0 583 [ 2082.828389] [c000000f1633fb60] [c00000000099e644] printk+0x84/0x98 584 [ 2082.828391] [c000000f1633fb90] [c0000000000851a8] dump_work_fn+0x238/0x250 585 [ 2082.828394] [c000000f1633fc60] [c0000000000ecb98] process_one_work+0x198/0x4b0 586 [ 2082.828396] [c000000f1633fcf0] [c0000000000ed3dc] worker_thread+0x18c/0x5a0 587 [ 2082.828399] [c000000f1633fd80] [c0000000000f4650] kthread+0x110/0x130 588 [ 2082.828403] [c000000f1633fe30] [c000000000009674] ret_from_kernel_thread+0x5c/0x68 589 590 Hence lets close SOL (and FW console) during FSP R/R. 591- FSP/CONSOLE: Do not associate unavailable console 592 593 Presently OPAL sends associate/unassociate MBOX command for all 594 FSP serial console (like below OPAL message). We have to check 595 console is available or not before sending this message. :: 596 597 [ 5013.227994012,7] FSP: Reassociating HVSI console 1 598 [ 5013.227997540,7] FSP: Reassociating HVSI console 2 599- FSP: Disable PSI link whenever FSP tells OPAL about impending R/R 600 601 Commit 42d5d047 fixed scenario where DPO has been initiated, but FSP went 602 into reset before the CEC power down came in. But this is generic issue 603 that can happen in normal shutdown path as well. 604 605 Hence disable PSI link as soon as we detect FSP impending R/R. 606 607- fsp: return OPAL_BUSY_EVENT on failure sending FSP_CMD_POWERDOWN_NORM 608 Also, return OPAL_BUSY_EVENT on failure sending FSP_CMD_REBOOT / DEEP_REBOOT. 609 610 We had a race condition between FSP Reset/Reload and powering down 611 the system from the host: 612 613 Roughly: 614 615 == ======================== ========================================================== 616 # FSP Host 617 == ======================== ========================================================== 618 1 Power on 619 2 Power on 620 3 (inject EPOW) 621 4 (trigger FSP R/R) 622 5 Processes EPOW event, starts shutting down 623 6 calls OPAL_CEC_POWER_DOWN 624 7 (is still in R/R) 625 8 gets OPAL_INTERNAL_ERROR, spins in opal_poll_events 626 9 (FSP comes back) 627 10 spinning in opal_poll_events 628 11 (thinks host is running) 629 == ======================== ========================================================== 630 631 The call to OPAL_CEC_POWER_DOWN is only made once as the reset/reload 632 error path for fsp_sync_msg() is to return -1, which means we give 633 the OS OPAL_INTERNAL_ERROR, which is fine, except that our own API 634 docs give us the opportunity to return OPAL_BUSY when trying again 635 later may be successful, and we're ambiguous as to if you should retry 636 on OPAL_INTERNAL_ERROR. 637 638 For reference, the linux code looks like this: :: 639 640 static void __noreturn pnv_power_off(void) 641 { 642 long rc = OPAL_BUSY; 643 644 pnv_prepare_going_down(); 645 646 while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) { 647 rc = opal_cec_power_down(0); 648 if (rc == OPAL_BUSY_EVENT) 649 opal_poll_events(NULL); 650 else 651 mdelay(10); 652 } 653 for (;;) 654 opal_poll_events(NULL); 655 } 656 657 Which means that *practically* our only option is to return OPAL_BUSY 658 or OPAL_BUSY_EVENT. 659 660 We choose OPAL_BUSY_EVENT for FSP systems as we do want to ensure we're 661 running pollers to communicate with the FSP and do the final bits of 662 Reset/Reload handling before we power off the system. 663 664 665Since :ref:`skiboot-5.8`: 666 667- FSP/NVRAM: Handle "get vNVRAM statistics" command 668 669 FSP sends MBOX command (cmd : 0xEB, subcmd : 0x05, mod : 0x00) to get vNVRAM 670 statistics. OPAL doesn't maintain any such statistics. Hence return 671 FSP_STATUS_INVALID_SUBCMD. 672 673 Fixes these messages appearing in the OPAL log: :: 674 675 [16944.384670488,3] FSP: Unhandled message eb0500 676 [16944.474110465,3] FSP: Unhandled message eb0500 677 [16945.111280784,3] FSP: Unhandled message eb0500 678 [16945.293393485,3] FSP: Unhandled message eb0500 679- fsp: Move common prints to trace 680 681 These two prints just end up filling the skiboot logs on any machine 682 that's been booted for more than a few hours. 683 684 They have never been useful, so make them trace level. They were: :: 685 SURV: Received heartbeat acknowledge from FSP 686 SURV: Sending the heartbeat command to FSP 687 688BMC based systems 689----------------- 690- hw/lpc-uart: read from RBR to clear character timeout interrupts 691 692 When using the aspeed SUART, we see a condition where the UART sends 693 continuous character timeout interrupts. This change adds a (heavily 694 commented) dummy read from the RBR to clear the interrupt condition on 695 init. 696 697 This was observed on p9dsu systems, but likely applies to other systems 698 using the SUART. 699- astbmc: Add methods for handing Device Tree based slots 700 e.g. ones from HDAT on POWER9. 701 702General 703------- 704 705Since :ref:`skiboot-5.9-rc5`: 706 707- p8-i2c: Further timeout reworks 708 709 This patch reworks the way timeouts are set so that rather than imposing 710 a hard deadline based on the transaction length it uses a 711 kick-the-can-down-the-road approach where the timeout will be reset each 712 time data is written to or received from the master. This fits better 713 with the actual failure modes that timeouts are designed to handle, such 714 as unusually slow or broken devices. 715 716 Additionally this patch moves all the special case detection out of the 717 timeout handler. This is help to improve the robustness of the driver and 718 prepare for a more substantial rework of the driver as a whole later on. 719- npu: Fix broken fast reset 720 721 0679f61244b "fast-reset: by default (if possible)" broke NPU - now 722 the NV links does not get enabled after reboot. 723 724 This disables fast reboot for NPU machines till a better solution is found. 725 726Since :ref:`skiboot-5.9-rc2`: 727 728- Improvements to vpd device tree entries 729 730 Previously we would miss some properties 731 732Since :ref:`skiboot-5.9-rc1`: 733 734- hw/p8-i2c: Fix deadlock in p9_i2c_bus_owner_change 735 736 When debugging a system where Linux was taking soft lockup errors with 737 two CPUs stuck in OPAL: 738 739 ======================= ============== 740 CPU0 CPU1 741 ======================= ============== 742 lock 743 p8_i2c_recover 744 opal_handle_interrupt 745 sync_timer 746 cancel_timer 747 p9_i2c_bus_owner_change 748 occ_p9_interrupt 749 xive_source_interrupt 750 opal_handle_interrupt 751 ======================= ============== 752 753 p8_i2c_recover() is a timer, and is stuck trying to take master->lock. 754 p9_i2c_bus_owner_change() has taken master->lock, but then is stuck waiting 755 for all timers to complete. We deadlock. 756 757 Fix this by using cancel_timer_async(). 758- opal/cpu: Mark the core as bad while disabling threads of the core. 759 760 If any of the core fails to sync its TB during chipTOD initialization, 761 all the threads of that core are disabled. But this does not make 762 linux kernel to ignore the core/cpus. It crashes while bringing them up 763 with below backtrace: :: 764 765 [ 38.883898] kexec_core: Starting new kernel 766 cpu 0x0: Vector: 300 (Data Access) at [c0000003f277b730] 767 pc: c0000000001b9890: internal_create_group+0x30/0x304 768 lr: c0000000001b9880: internal_create_group+0x20/0x304 769 sp: c0000003f277b9b0 770 msr: 900000000280b033 771 dar: 40 772 dsisr: 40000000 773 current = 0xc0000003f9f41000 774 paca = 0xc00000000fe00000 softe: 0 irq_happened: 0x01 775 pid = 2572, comm = kexec 776 Linux version 4.13.2-openpower1 (jenkins@p89) (gcc version 6.4.0 (Buildroot 2017.08-00006-g319c6e1)) #1 SMP Wed Sep 20 05:42:11 UTC 2017 777 enter ? for help 778 [c0000003f277b9b0] c0000000008a8780 (unreliable) 779 [c0000003f277ba50] c00000000041c3ac topology_add_dev+0x2c/0x40 780 [c0000003f277ba70] c00000000006b078 cpuhp_invoke_callback+0x88/0x170 781 [c0000003f277bac0] c00000000006b22c cpuhp_up_callbacks+0x54/0xb8 782 [c0000003f277bb10] c00000000006bc68 cpu_up+0x11c/0x168 783 [c0000003f277bbc0] c00000000002f0e0 default_machine_kexec+0x1fc/0x274 784 [c0000003f277bc50] c00000000002e2d8 machine_kexec+0x50/0x58 785 [c0000003f277bc70] c0000000000de4e8 kernel_kexec+0x98/0xb4 786 [c0000003f277bce0] c00000000008b0f0 SyS_reboot+0x1c8/0x1f4 787 [c0000003f277be30] c00000000000b118 system_call+0x58/0x6c 788 789Since :ref:`skiboot-5.8`: 790 791- ipmi: Convert common debug prints to trace 792 793 OPAL logs messages for every IPMI request from host. Sometime OPAL console 794 is filled with only these messages. This path is pretty stable now and 795 we have enough logs to cover bad path. Hence lets convert these debug 796 message to trace/info message. Examples are: :: 797 798 [ 1356.423958816,7] opal_ipmi_recv(cmd: 0xf0 netfn: 0x3b resp_size: 0x02) 799 [ 1356.430774496,7] opal_ipmi_send(cmd: 0xf0 netfn: 0x3a len: 0x3b) 800 [ 1356.430797392,7] BT: seq 0x20 netfn 0x3a cmd 0xf0: Message sent to host 801 [ 1356.431668496,7] BT: seq 0x20 netfn 0x3a cmd 0xf0: IPMI MSG done 802- libflash/file: Handle short read()s and write()s correctly 803 804 Currently we don't move the buffer along for a short read() or write() 805 and nor do we request only the remaining amount. 806 807- hw/p8-i2c: Rework timeout handling 808 809 Currently we treat a timeout as a hard failure and will automatically 810 fail any transations that hit their timeout. This results in 811 unnecessarily failing I2C requests if interrupts are dropped, etc. 812 Although these are bad things that we should log we can handle them 813 better by checking the actual hardware status and completing the 814 transation if there are no real errors. This patch reworks the timeout 815 handling to check the status and continue the transaction if it can. 816 if it can while logging an error if it detects a timeout due to a 817 dropped interrupt. 818- core/flash: Only expect ELF header for BOOTKERNEL partition flash resource 819 820 When loading a flash resource which isn't signed (secure and trusted 821 boot) and which doesn't have a subpartition, we assume it's the 822 BOOTKERNEL since previously this was the only such resource. Thus we 823 also assumed it had an ELF header which we parsed to get the size of the 824 partition rather than trusting the actual_size field in the FFS header. 825 A previous commit (9727fe3 DT: Add ibm,firmware-versions node) added the 826 version resource which isn't signed and also doesn't have a subpartition, 827 thus we expect it to have an ELF header. It doesn't so we print the 828 error message "FLASH: Invalid ELF header part VERSION". 829 830 It is a fluke that this works currently since we load the secure boot 831 header unconditionally and this happen to be the same size as the 832 version partition. We also don't update the return code on error so 833 happen to return OPAL_SUCCESS. 834 835 To make this explicitly correct; only check for an ELF header if we are 836 loading the BOOTKERNEL resource, otherwise use the partition size from 837 the FFS header. Also set the return code on error so we don't 838 erroneously return OPAL_SUCCESS. Add a check that the resource will fit 839 in the supplied buffer to prevent buffer overrun. 840- flash: Support adding the no-erase property to flash 841 842 The mbox protocol explicitly states that an erase is not required 843 before a write. This means that issuing an erase from userspace, 844 through the mtd device, and back returns a successful operation 845 that does nothing. Unfortunately, this makes userspace tools unhappy. 846 Linux MTD devices support the MTD_NO_ERASE flag which conveys that 847 writes do not require erases on the underlying flash devices. We 848 should set this property on all of our 849 devices which do not require erases to be performed. 850 851 NOTE: This still requires a linux kernel component to set the 852 MTD_NO_ERASE flag from the device tree property. 853 854Utilities 855--------- 856 857Since :ref:`skiboot-5.9-rc1`: 858- opal-prd: Fix memory leak 859 860Since :ref:`skiboot-5.8`: 861 862- external/gard: Clear entire guard partition instead of entry by entry 863 864 When using the current implementation of the gard tool to ecc clear the 865 entire GUARD partition it is done one gard record at a time. While this 866 may be ok when accessing the actual flash this is very slow when done 867 from the host over the mbox protocol (on the order of 4 minutes) because 868 the bmc side is required to do many read, erase, writes under the hood. 869 870 Fix this by rewriting the gard tool reset_partition() function. Now we 871 allocate all the erased guard entries and (if required) apply ecc to the 872 entire buffer. Then we can do one big erase and write of the entire 873 partition. This reduces the time to clear the guard partition to on the 874 order of 4 seconds. 875- opal-prd: Fix opal-prd command line options 876 877 HBRT OCC reset interface depends on service processor type. 878 879 - FSP: reset_pm_complex() 880 - BMC: process_occ_reset() 881 882 We have both `occ` and `pm-complex` command line interfaces. 883 This patch adds support to dispaly appropriate message depending 884 on system type. 885 886 === ==================== ============================ 887 SP Command Action 888 === ==================== ============================ 889 FSP opal-prd occ display error message 890 FSP opal-prd pm-complex Call pm_complex_reset() 891 BMC opal-prd occ Call process_occ_reset() 892 BMC opal-prd pm-complex display error message 893 === ==================== ============================ 894 895- opal-prd: detect service processor type and 896 then make appropriate occ reset call. 897- pflash: Fix erase command for unaligned start address 898 899 The erase_range() function handles erasing the flash for a given start 900 address and length, and can handle an unaligned start address and 901 length. However in the unaligned start address case we are incorrectly 902 calculating the remaining size which can lead to incomplete erases. 903 904 If we're going to update the remaining size based on what the start 905 address was then we probably want to do that before we overide the 906 origin start address. So rearrange the code so that this is indeed the 907 case. 908- external/gard: Print an error if run on an FSP system 909 910Simulators 911---------- 912 913- mambo: Add mambo socket program 914 915 This adds a program that can be run inside a mambo simulator in linux 916 userspace which enables TCP sockets to be proxied in and out of the 917 simulator to the host. 918 919 Unlike mambo bogusnet, it's requires no linux or skiboot specific 920 drivers/infrastructure to run. 921 922 Run inside the simulator: 923 924 - to forward host ssh connections to sim ssh server: 925 ``./mambo-socket-proxy -h 10022 -s 22``, then connect to port 10022 926 on your host with ``ssh -p 10022 localhost`` 927 - to allow http proxy access from inside the sim to local http proxy: 928 ``./mambo-socket-proxy -b proxy.mynetwork -h 3128 -s 3128`` 929 930 Multiple connections are supported. 931- idle: disable stop*_lite POWER9 idle states for Mambo platform 932 933 Mambo prior to Mambo.7.8.21 had a bug where the stop idle instruction 934 with PSSCR[ESL]=PSSCR[EC]=0 would resume with MSR set as though it had 935 taken a system reset interrupt. 936 937 Linux currently executes this instruction with MSR already set that 938 way, so the problem went unnoticed. A proposed patch to Linux changes 939 that, and causes the idle code to crash. Work around this by disabling 940 lite stop states for the mambo platform for now. 941 942Contributors 943------------ 944 945- 209 csets from 32 developers 946- 2 employers found 947- A total of 9619 lines added, 1612 removed (delta 8007) 948 949Extending the analysis done for some previous releases, we can see our trends 950in code review across versions: 951 952======= ====== ======== ========= ========= =========== 953Release csets Ack % Reviews % Tested % Reported % 954======= ====== ======== ========= ========= =========== 9555.0 329 15 (5%) 20 (6%) 1 (0%) 0 (0%) 9565.1 372 13 (3%) 38 (10%) 1 (0%) 4 (1%) 9575.2-rc1 334 20 (6%) 34 (10%) 6 (2%) 11 (3%) 9585.3-rc1 302 36 (12%) 53 (18%) 4 (1%) 5 (2%) 9595.4 361 16 (4%) 28 (8%) 1 (0%) 9 (2%) 9605.5 408 11 (3%) 48 (12%) 14 (3%) 10 (2%) 9615.6 87 12 (14%) 6 (7%) 5 (6%) 2 (2%) 9625.7 232 30 (13%) 32 (14%) 5 (2%) 2 (1%) 9635.8 157 13 (8%) 36 (23%) 2 (1%) 6 (4%) 9645.9 209 15 (7%) 78 (37%) 3 (1%) 10 (5%) 965======= ====== ======== ========= ========= =========== 966 967The review count here is largely bogus, there was a series of 25 whitespace 968patches that got "Reviewed-by" and if we exclude them, we're back to 14%, 969which is more like what I'd expect. 970 971 972Developers with the most changesets 973^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 974 975========================== === ======= 976Developer # % 977========================== === ======= 978Stewart Smith 28 (13.4%) 979Vasant Hegde 25 (12.0%) 980Joel Stanley 25 (12.0%) 981Michael Neuling 24 (11.5%) 982Oliver O'Halloran 20 (9.6%) 983Benjamin Herrenschmidt 16 (7.7%) 984Nicholas Piggin 12 (5.7%) 985Akshay Adiga 8 (3.8%) 986Madhavan Srinivasan 7 (3.3%) 987Reza Arbab 6 (2.9%) 988Mahesh Salgaonkar 3 (1.4%) 989Claudio Carvalho 3 (1.4%) 990Suraj Jitindar Singh 3 (1.4%) 991Sam Bobroff 3 (1.4%) 992Shilpasri G Bhat 2 (1.0%) 993Michael Ellerman 2 (1.0%) 994Andrew Donnellan 2 (1.0%) 995Vaibhav Jain 2 (1.0%) 996Jeremy Kerr 2 (1.0%) 997Cyril Bur 2 (1.0%) 998Christophe Lombard 2 (1.0%) 999Daniel Black 2 (1.0%) 1000Alexey Kardashevskiy 1 (0.5%) 1001Alistair Popple 1 (0.5%) 1002Anton Blanchard 1 (0.5%) 1003Guilherme G. Piccoli 1 (0.5%) 1004John W Walthour 1 (0.5%) 1005Anju T Sudhakar 1 (0.5%) 1006Balbir Singh 1 (0.5%) 1007Russell Currey 1 (0.5%) 1008William A. Kennington III 1 (0.5%) 1009Sukadev Bhattiprolu 1 (0.5%) 1010========================== === ======= 1011 1012Developers with the most changed lines 1013^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1014 1015========================== ==== ======= 1016Developer # % 1017========================== ==== ======= 1018Akshay Adiga 2731 (27.9%) 1019Oliver O'Halloran 1512 (15.5%) 1020Stewart Smith 1355 (13.9%) 1021Nicholas Piggin 929 (9.5%) 1022Vasant Hegde 827 (8.5%) 1023Michael Neuling 719 (7.4%) 1024Benjamin Herrenschmidt 522 (5.3%) 1025Madhavan Srinivasan 180 (1.8%) 1026Sam Bobroff 172 (1.8%) 1027Christophe Lombard 170 (1.7%) 1028Mahesh Salgaonkar 166 (1.7%) 1029Andrew Donnellan 125 (1.3%) 1030Joel Stanley 70 (0.7%) 1031Reza Arbab 64 (0.7%) 1032Claudio Carvalho 51 (0.5%) 1033Suraj Jitindar Singh 42 (0.4%) 1034Alistair Popple 28 (0.3%) 1035Jeremy Kerr 25 (0.3%) 1036Michael Ellerman 21 (0.2%) 1037Cyril Bur 18 (0.2%) 1038Shilpasri G Bhat 17 (0.2%) 1039Vaibhav Jain 8 (0.1%) 1040Daniel Black 6 (0.1%) 1041William A. Kennington III 4 (0.0%) 1042Sukadev Bhattiprolu 4 (0.0%) 1043Alexey Kardashevskiy 3 (0.0%) 1044John W Walthour 3 (0.0%) 1045Balbir Singh 3 (0.0%) 1046Guilherme G. Piccoli 2 (0.0%) 1047Anton Blanchard 1 (0.0%) 1048Anju T Sudhakar 1 (0.0%) 1049Russell Currey 1 (0.0%) 1050========================== ==== ======= 1051 1052Developers with the most lines removed 1053^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1054 1055========================== ==== ======= 1056Developer # % 1057========================== ==== ======= 1058Alistair Popple 28 (1.7%) 1059========================== ==== ======= 1060 1061Developers with the most signoffs 1062^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1063 1064========================== === ======= 1065Developer # % 1066========================== === ======= 1067Stewart Smith 180 (97.8%) 1068Shilpasri G Bhat 2 (1.1%) 1069Mukesh Ojha 1 (0.5%) 1070Michael Neuling 1 (0.5%) 1071Total 184 (100%) 1072========================== === ======= 1073 1074Developers with the most reviews 1075^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1076 1077=========================== == ======= 1078Developer # % 1079=========================== == ======= 1080Michael Neuling 25 (32.5%) 1081Russell Currey 25 (32.5%) 1082Vaidyanathan Srinivasan 9 (11.7%) 1083Oliver O'Halloran 4 (5.2%) 1084Andrew Donnellan 3 (3.9%) 1085Frederic Barrat 2 (2.6%) 1086Suraj Jitindar Singh 2 (2.6%) 1087Vasant Hegde 2 (2.6%) 1088Andrew Jeffery 1 (1.3%) 1089Samuel Mendoza-Jonas 1 (1.3%) 1090Alexey Kardashevskiy 1 (1.3%) 1091Cyril Bur 1 (1.3%) 1092Akshay Adiga 1 (1.3%) 1093Total 77 (100%) 1094=========================== == ======= 1095 1096 1097Developers with the most test credits 1098^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1099 1100=========================== == ======= 1101Developer # % 1102=========================== == ======= 1103Pridhiviraj Paidipeddi 3 (100.0%) 1104=========================== == ======= 1105 1106Developers who gave the most tested-by credits 1107^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1108 1109=========================== == ======= 1110Developer # % 1111=========================== == ======= 1112Vasant Hegde 2 (66.7%) 1113Michael Neuling 1 (33.3%) 1114=========================== == ======= 1115 1116 1117Developers with the most report credits 1118^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1119 1120=========================== == ======= 1121Developer # % 1122=========================== == ======= 1123Pridhiviraj Paidipeddi 6 (60.0%) 1124Andrew Donnellan 1 (10.0%) 1125Stewart Smith 1 (10.0%) 1126Shriya 1 (10.0%) 1127Robert Lippert 1 (10.0%) 1128Total 10 (100%) 1129=========================== == ======= 1130 1131Developers who gave the most report credits 1132^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1133 1134=========================== == ======= 1135Developer # % 1136=========================== == ======= 1137Stewart Smith 3 (30.0%) 1138Suraj Jitindar Singh 3 (30.0%) 1139Vasant Hegde 2 (20.0%) 1140Michael Neuling 1 (10.0%) 1141Madhavan Srinivasan 1 (10.0%) 1142Total 10 (100%) 1143=========================== == ======= 1144 1145Changesets and Employers 1146^^^^^^^^^^^^^^^^^^^^^^^^ 1147 1148Top changeset contributors by employer: 1149 1150=========================== === ======= 1151Employer # % 1152=========================== === ======= 1153IBM 208 (99.5%) 1154Google 1 (0.5%) 1155=========================== === ======= 1156 1157Top lines changed by employer: 1158 1159=========================== ==== ======= 1160Employer # % 1161=========================== ==== ======= 1162IBM 9776 (100.0%) 1163Google 4 (0.0%) 1164=========================== ==== ======= 1165 1166Employers with the most signoffs (total 184): 1167 1168=========================== === ======= 1169Employer # % 1170=========================== === ======= 1171IBM 184 (100.0%) 1172=========================== === ======= 1173 1174Employers with the most hackers (total 32): 1175 1176=========================== === ======= 1177Employer # % 1178=========================== === ======= 1179IBM 31 (96.9%) 1180Google 1 (3.1%) 1181=========================== === ======= 1182