11b13d190SFrançois Tigeot /* 21b13d190SFrançois Tigeot * Copyright © 2014 Intel Corporation 31b13d190SFrançois Tigeot * 41b13d190SFrançois Tigeot * Permission is hereby granted, free of charge, to any person obtaining a 51b13d190SFrançois Tigeot * copy of this software and associated documentation files (the "Software"), 61b13d190SFrançois Tigeot * to deal in the Software without restriction, including without limitation 71b13d190SFrançois Tigeot * the rights to use, copy, modify, merge, publish, distribute, sublicense, 81b13d190SFrançois Tigeot * and/or sell copies of the Software, and to permit persons to whom the 91b13d190SFrançois Tigeot * Software is furnished to do so, subject to the following conditions: 101b13d190SFrançois Tigeot * 111b13d190SFrançois Tigeot * The above copyright notice and this permission notice (including the next 121b13d190SFrançois Tigeot * paragraph) shall be included in all copies or substantial portions of the 131b13d190SFrançois Tigeot * Software. 141b13d190SFrançois Tigeot * 151b13d190SFrançois Tigeot * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 161b13d190SFrançois Tigeot * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 171b13d190SFrançois Tigeot * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 181b13d190SFrançois Tigeot * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 191b13d190SFrançois Tigeot * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 201b13d190SFrançois Tigeot * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 211b13d190SFrançois Tigeot * IN THE SOFTWARE. 221b13d190SFrançois Tigeot * 231b13d190SFrançois Tigeot * Authors: 241b13d190SFrançois Tigeot * Ben Widawsky <ben@bwidawsk.net> 251b13d190SFrançois Tigeot * Michel Thierry <michel.thierry@intel.com> 261b13d190SFrançois Tigeot * Thomas Daniel <thomas.daniel@intel.com> 271b13d190SFrançois Tigeot * Oscar Mateo <oscar.mateo@intel.com> 281b13d190SFrançois Tigeot * 291b13d190SFrançois Tigeot */ 301b13d190SFrançois Tigeot 311b13d190SFrançois Tigeot /** 321b13d190SFrançois Tigeot * DOC: Logical Rings, Logical Ring Contexts and Execlists 331b13d190SFrançois Tigeot * 341b13d190SFrançois Tigeot * Motivation: 351b13d190SFrançois Tigeot * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts". 361b13d190SFrançois Tigeot * These expanded contexts enable a number of new abilities, especially 371b13d190SFrançois Tigeot * "Execlists" (also implemented in this file). 381b13d190SFrançois Tigeot * 391b13d190SFrançois Tigeot * One of the main differences with the legacy HW contexts is that logical 401b13d190SFrançois Tigeot * ring contexts incorporate many more things to the context's state, like 411b13d190SFrançois Tigeot * PDPs or ringbuffer control registers: 421b13d190SFrançois Tigeot * 431b13d190SFrançois Tigeot * The reason why PDPs are included in the context is straightforward: as 441b13d190SFrançois Tigeot * PPGTTs (per-process GTTs) are actually per-context, having the PDPs 451b13d190SFrançois Tigeot * contained there mean you don't need to do a ppgtt->switch_mm yourself, 461b13d190SFrançois Tigeot * instead, the GPU will do it for you on the context switch. 471b13d190SFrançois Tigeot * 481b13d190SFrançois Tigeot * But, what about the ringbuffer control registers (head, tail, etc..)? 491b13d190SFrançois Tigeot * shouldn't we just need a set of those per engine command streamer? This is 501b13d190SFrançois Tigeot * where the name "Logical Rings" starts to make sense: by virtualizing the 511b13d190SFrançois Tigeot * rings, the engine cs shifts to a new "ring buffer" with every context 521b13d190SFrançois Tigeot * switch. When you want to submit a workload to the GPU you: A) choose your 531b13d190SFrançois Tigeot * context, B) find its appropriate virtualized ring, C) write commands to it 541b13d190SFrançois Tigeot * and then, finally, D) tell the GPU to switch to that context. 551b13d190SFrançois Tigeot * 561b13d190SFrançois Tigeot * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch 571b13d190SFrançois Tigeot * to a contexts is via a context execution list, ergo "Execlists". 581b13d190SFrançois Tigeot * 591b13d190SFrançois Tigeot * LRC implementation: 601b13d190SFrançois Tigeot * Regarding the creation of contexts, we have: 611b13d190SFrançois Tigeot * 621b13d190SFrançois Tigeot * - One global default context. 631b13d190SFrançois Tigeot * - One local default context for each opened fd. 641b13d190SFrançois Tigeot * - One local extra context for each context create ioctl call. 651b13d190SFrançois Tigeot * 661b13d190SFrançois Tigeot * Now that ringbuffers belong per-context (and not per-engine, like before) 671b13d190SFrançois Tigeot * and that contexts are uniquely tied to a given engine (and not reusable, 681b13d190SFrançois Tigeot * like before) we need: 691b13d190SFrançois Tigeot * 701b13d190SFrançois Tigeot * - One ringbuffer per-engine inside each context. 711b13d190SFrançois Tigeot * - One backing object per-engine inside each context. 721b13d190SFrançois Tigeot * 731b13d190SFrançois Tigeot * The global default context starts its life with these new objects fully 741b13d190SFrançois Tigeot * allocated and populated. The local default context for each opened fd is 751b13d190SFrançois Tigeot * more complex, because we don't know at creation time which engine is going 761b13d190SFrançois Tigeot * to use them. To handle this, we have implemented a deferred creation of LR 771b13d190SFrançois Tigeot * contexts: 781b13d190SFrançois Tigeot * 791b13d190SFrançois Tigeot * The local context starts its life as a hollow or blank holder, that only 801b13d190SFrançois Tigeot * gets populated for a given engine once we receive an execbuffer. If later 811b13d190SFrançois Tigeot * on we receive another execbuffer ioctl for the same context but a different 821b13d190SFrançois Tigeot * engine, we allocate/populate a new ringbuffer and context backing object and 831b13d190SFrançois Tigeot * so on. 841b13d190SFrançois Tigeot * 851b13d190SFrançois Tigeot * Finally, regarding local contexts created using the ioctl call: as they are 861b13d190SFrançois Tigeot * only allowed with the render ring, we can allocate & populate them right 871b13d190SFrançois Tigeot * away (no need to defer anything, at least for now). 881b13d190SFrançois Tigeot * 891b13d190SFrançois Tigeot * Execlists implementation: 901b13d190SFrançois Tigeot * Execlists are the new method by which, on gen8+ hardware, workloads are 911b13d190SFrançois Tigeot * submitted for execution (as opposed to the legacy, ringbuffer-based, method). 921b13d190SFrançois Tigeot * This method works as follows: 931b13d190SFrançois Tigeot * 941b13d190SFrançois Tigeot * When a request is committed, its commands (the BB start and any leading or 951b13d190SFrançois Tigeot * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer 961b13d190SFrançois Tigeot * for the appropriate context. The tail pointer in the hardware context is not 971b13d190SFrançois Tigeot * updated at this time, but instead, kept by the driver in the ringbuffer 981b13d190SFrançois Tigeot * structure. A structure representing this request is added to a request queue 991b13d190SFrançois Tigeot * for the appropriate engine: this structure contains a copy of the context's 1001b13d190SFrançois Tigeot * tail after the request was written to the ring buffer and a pointer to the 1011b13d190SFrançois Tigeot * context itself. 1021b13d190SFrançois Tigeot * 1031b13d190SFrançois Tigeot * If the engine's request queue was empty before the request was added, the 1041b13d190SFrançois Tigeot * queue is processed immediately. Otherwise the queue will be processed during 1051b13d190SFrançois Tigeot * a context switch interrupt. In any case, elements on the queue will get sent 1061b13d190SFrançois Tigeot * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a 1071b13d190SFrançois Tigeot * globally unique 20-bits submission ID. 1081b13d190SFrançois Tigeot * 1091b13d190SFrançois Tigeot * When execution of a request completes, the GPU updates the context status 1101b13d190SFrançois Tigeot * buffer with a context complete event and generates a context switch interrupt. 1111b13d190SFrançois Tigeot * During the interrupt handling, the driver examines the events in the buffer: 1121b13d190SFrançois Tigeot * for each context complete event, if the announced ID matches that on the head 1131b13d190SFrançois Tigeot * of the request queue, then that request is retired and removed from the queue. 1141b13d190SFrançois Tigeot * 1151b13d190SFrançois Tigeot * After processing, if any requests were retired and the queue is not empty 1161b13d190SFrançois Tigeot * then a new execution list can be submitted. The two requests at the front of 1171b13d190SFrançois Tigeot * the queue are next to be submitted but since a context may not occur twice in 1181b13d190SFrançois Tigeot * an execution list, if subsequent requests have the same ID as the first then 1191b13d190SFrançois Tigeot * the two requests must be combined. This is done simply by discarding requests 1201b13d190SFrançois Tigeot * at the head of the queue until either only one requests is left (in which case 1211b13d190SFrançois Tigeot * we use a NULL second context) or the first two requests have unique IDs. 1221b13d190SFrançois Tigeot * 1231b13d190SFrançois Tigeot * By always executing the first two requests in the queue the driver ensures 1241b13d190SFrançois Tigeot * that the GPU is kept as busy as possible. In the case where a single context 1251b13d190SFrançois Tigeot * completes but a second context is still executing, the request for this second 1261b13d190SFrançois Tigeot * context will be at the head of the queue when we remove the first one. This 1271b13d190SFrançois Tigeot * request will then be resubmitted along with a new request for a different context, 1281b13d190SFrançois Tigeot * which will cause the hardware to continue executing the second request and queue 1291b13d190SFrançois Tigeot * the new request (the GPU detects the condition of a context getting preempted 1301b13d190SFrançois Tigeot * with the same context and optimizes the context switch flow by not doing 1311b13d190SFrançois Tigeot * preemption, but just sampling the new tail pointer). 1321b13d190SFrançois Tigeot * 1331b13d190SFrançois Tigeot */ 1348621f407SFrançois Tigeot #include <linux/interrupt.h> 1351b13d190SFrançois Tigeot 1361b13d190SFrançois Tigeot #include <drm/drmP.h> 1371b13d190SFrançois Tigeot #include <drm/i915_drm.h> 1381b13d190SFrançois Tigeot #include "i915_drv.h" 139a05eeebfSFrançois Tigeot #include "intel_mocs.h" 1401b13d190SFrançois Tigeot 1412c9916cdSFrançois Tigeot #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE) 1421b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE) 1431b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_OTHER_SIZE (2 * PAGE_SIZE) 1441b13d190SFrançois Tigeot 1451b13d190SFrançois Tigeot #define RING_EXECLIST_QFULL (1 << 0x2) 1461b13d190SFrançois Tigeot #define RING_EXECLIST1_VALID (1 << 0x3) 1471b13d190SFrançois Tigeot #define RING_EXECLIST0_VALID (1 << 0x4) 1481b13d190SFrançois Tigeot #define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE) 1491b13d190SFrançois Tigeot #define RING_EXECLIST1_ACTIVE (1 << 0x11) 1501b13d190SFrançois Tigeot #define RING_EXECLIST0_ACTIVE (1 << 0x12) 1511b13d190SFrançois Tigeot 1521b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0) 1531b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_PREEMPTED (1 << 1) 1541b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2) 1551b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3) 1561b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_COMPLETE (1 << 4) 1571b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15) 1581b13d190SFrançois Tigeot 1591b13d190SFrançois Tigeot #define CTX_LRI_HEADER_0 0x01 1601b13d190SFrançois Tigeot #define CTX_CONTEXT_CONTROL 0x02 1611b13d190SFrançois Tigeot #define CTX_RING_HEAD 0x04 1621b13d190SFrançois Tigeot #define CTX_RING_TAIL 0x06 1631b13d190SFrançois Tigeot #define CTX_RING_BUFFER_START 0x08 1641b13d190SFrançois Tigeot #define CTX_RING_BUFFER_CONTROL 0x0a 1651b13d190SFrançois Tigeot #define CTX_BB_HEAD_U 0x0c 1661b13d190SFrançois Tigeot #define CTX_BB_HEAD_L 0x0e 1671b13d190SFrançois Tigeot #define CTX_BB_STATE 0x10 1681b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_U 0x12 1691b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_L 0x14 1701b13d190SFrançois Tigeot #define CTX_SECOND_BB_STATE 0x16 1711b13d190SFrançois Tigeot #define CTX_BB_PER_CTX_PTR 0x18 1721b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX 0x1a 1731b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX_OFFSET 0x1c 1741b13d190SFrançois Tigeot #define CTX_LRI_HEADER_1 0x21 1751b13d190SFrançois Tigeot #define CTX_CTX_TIMESTAMP 0x22 1761b13d190SFrançois Tigeot #define CTX_PDP3_UDW 0x24 1771b13d190SFrançois Tigeot #define CTX_PDP3_LDW 0x26 1781b13d190SFrançois Tigeot #define CTX_PDP2_UDW 0x28 1791b13d190SFrançois Tigeot #define CTX_PDP2_LDW 0x2a 1801b13d190SFrançois Tigeot #define CTX_PDP1_UDW 0x2c 1811b13d190SFrançois Tigeot #define CTX_PDP1_LDW 0x2e 1821b13d190SFrançois Tigeot #define CTX_PDP0_UDW 0x30 1831b13d190SFrançois Tigeot #define CTX_PDP0_LDW 0x32 1841b13d190SFrançois Tigeot #define CTX_LRI_HEADER_2 0x41 1851b13d190SFrançois Tigeot #define CTX_R_PWR_CLK_STATE 0x42 1861b13d190SFrançois Tigeot #define CTX_GPGPU_CSR_BASE_ADDRESS 0x44 1871b13d190SFrançois Tigeot 1881b13d190SFrançois Tigeot #define GEN8_CTX_VALID (1<<0) 1891b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_PD_RESTORE (1<<1) 1901b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_RESTORE (1<<2) 1911b13d190SFrançois Tigeot #define GEN8_CTX_L3LLC_COHERENT (1<<5) 1921b13d190SFrançois Tigeot #define GEN8_CTX_PRIVILEGE (1<<8) 19319c468b4SFrançois Tigeot 194aee94f86SFrançois Tigeot #define ASSIGN_CTX_REG(reg_state, pos, reg, val) do { \ 195aee94f86SFrançois Tigeot (reg_state)[(pos)+0] = i915_mmio_reg_offset(reg); \ 196aee94f86SFrançois Tigeot (reg_state)[(pos)+1] = (val); \ 197aee94f86SFrançois Tigeot } while (0) 198aee94f86SFrançois Tigeot 199aee94f86SFrançois Tigeot #define ASSIGN_CTX_PDP(ppgtt, reg_state, n) do { \ 200a05eeebfSFrançois Tigeot const u64 _addr = i915_page_dir_dma_addr((ppgtt), (n)); \ 20119c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _UDW+1] = upper_32_bits(_addr); \ 20219c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _LDW+1] = lower_32_bits(_addr); \ 203aee94f86SFrançois Tigeot } while (0) 20419c468b4SFrançois Tigeot 205aee94f86SFrançois Tigeot #define ASSIGN_CTX_PML4(ppgtt, reg_state) do { \ 206352ff8bdSFrançois Tigeot reg_state[CTX_PDP0_UDW + 1] = upper_32_bits(px_dma(&ppgtt->pml4)); \ 207352ff8bdSFrançois Tigeot reg_state[CTX_PDP0_LDW + 1] = lower_32_bits(px_dma(&ppgtt->pml4)); \ 208aee94f86SFrançois Tigeot } while (0) 209352ff8bdSFrançois Tigeot 2101b13d190SFrançois Tigeot enum { 2111b13d190SFrançois Tigeot FAULT_AND_HANG = 0, 2121b13d190SFrançois Tigeot FAULT_AND_HALT, /* Debug only */ 2131b13d190SFrançois Tigeot FAULT_AND_STREAM, 2141b13d190SFrançois Tigeot FAULT_AND_CONTINUE /* Unsupported */ 2151b13d190SFrançois Tigeot }; 2161b13d190SFrançois Tigeot #define GEN8_CTX_ID_SHIFT 32 217*1487f786SFrançois Tigeot #define GEN8_CTX_ID_WIDTH 21 218c0e85e96SFrançois Tigeot #define GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT 0x17 219c0e85e96SFrançois Tigeot #define GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT 0x26 2201b13d190SFrançois Tigeot 221*1487f786SFrançois Tigeot /* Typical size of the average request (2 pipecontrols and a MI_BB) */ 222*1487f786SFrançois Tigeot #define EXECLISTS_REQUEST_SIZE 64 /* bytes */ 223*1487f786SFrançois Tigeot 224*1487f786SFrançois Tigeot static int execlists_context_deferred_alloc(struct i915_gem_context *ctx, 225*1487f786SFrançois Tigeot struct intel_engine_cs *engine); 226*1487f786SFrançois Tigeot static int intel_lr_context_pin(struct i915_gem_context *ctx, 227c0e85e96SFrançois Tigeot struct intel_engine_cs *engine); 2282c9916cdSFrançois Tigeot 2291b13d190SFrançois Tigeot /** 2301b13d190SFrançois Tigeot * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists 231*1487f786SFrançois Tigeot * @dev_priv: i915 device private 2321b13d190SFrançois Tigeot * @enable_execlists: value of i915.enable_execlists module parameter. 2331b13d190SFrançois Tigeot * 2341b13d190SFrançois Tigeot * Only certain platforms support Execlists (the prerequisites being 2352c9916cdSFrançois Tigeot * support for Logical Ring Contexts and Aliasing PPGTT or better). 2361b13d190SFrançois Tigeot * 2371b13d190SFrançois Tigeot * Return: 1 if Execlists is supported and has to be enabled. 2381b13d190SFrançois Tigeot */ 239*1487f786SFrançois Tigeot int intel_sanitize_enable_execlists(struct drm_i915_private *dev_priv, int enable_execlists) 2401b13d190SFrançois Tigeot { 241352ff8bdSFrançois Tigeot /* On platforms with execlist available, vGPU will only 242352ff8bdSFrançois Tigeot * support execlist mode, no ring buffer mode. 243352ff8bdSFrançois Tigeot */ 244*1487f786SFrançois Tigeot if (HAS_LOGICAL_RING_CONTEXTS(dev_priv) && intel_vgpu_active(dev_priv)) 245352ff8bdSFrançois Tigeot return 1; 246352ff8bdSFrançois Tigeot 247*1487f786SFrançois Tigeot if (INTEL_GEN(dev_priv) >= 9) 2482c9916cdSFrançois Tigeot return 1; 2492c9916cdSFrançois Tigeot 2501b13d190SFrançois Tigeot if (enable_execlists == 0) 2511b13d190SFrançois Tigeot return 0; 2521b13d190SFrançois Tigeot 253*1487f786SFrançois Tigeot if (HAS_LOGICAL_RING_CONTEXTS(dev_priv) && 254*1487f786SFrançois Tigeot USES_PPGTT(dev_priv) && 2551b13d190SFrançois Tigeot i915.use_mmio_flip >= 0) 2561b13d190SFrançois Tigeot return 1; 2571b13d190SFrançois Tigeot 2581b13d190SFrançois Tigeot return 0; 2591b13d190SFrançois Tigeot } 2601b13d190SFrançois Tigeot 261c0e85e96SFrançois Tigeot static void 2628621f407SFrançois Tigeot logical_ring_init_platform_invariants(struct intel_engine_cs *engine) 263c0e85e96SFrançois Tigeot { 264*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 265c0e85e96SFrançois Tigeot 266*1487f786SFrançois Tigeot if (IS_GEN8(dev_priv) || IS_GEN9(dev_priv)) 2678621f407SFrançois Tigeot engine->idle_lite_restore_wa = ~0; 2688621f407SFrançois Tigeot 269*1487f786SFrançois Tigeot engine->disable_lite_restore_wa = (IS_SKL_REVID(dev_priv, 0, SKL_REVID_B0) || 270*1487f786SFrançois Tigeot IS_BXT_REVID(dev_priv, 0, BXT_REVID_A1)) && 2718621f407SFrançois Tigeot (engine->id == VCS || engine->id == VCS2); 272c0e85e96SFrançois Tigeot 2738621f407SFrançois Tigeot engine->ctx_desc_template = GEN8_CTX_VALID; 274*1487f786SFrançois Tigeot if (IS_GEN8(dev_priv)) 2758621f407SFrançois Tigeot engine->ctx_desc_template |= GEN8_CTX_L3LLC_COHERENT; 2768621f407SFrançois Tigeot engine->ctx_desc_template |= GEN8_CTX_PRIVILEGE; 277c0e85e96SFrançois Tigeot 278c0e85e96SFrançois Tigeot /* TODO: WaDisableLiteRestore when we start using semaphore 279c0e85e96SFrançois Tigeot * signalling between Command Streamers */ 280c0e85e96SFrançois Tigeot /* ring->ctx_desc_template |= GEN8_CTX_FORCE_RESTORE; */ 281c0e85e96SFrançois Tigeot 282c0e85e96SFrançois Tigeot /* WaEnableForceRestoreInCtxtDescForVCS:skl */ 283c0e85e96SFrançois Tigeot /* WaEnableForceRestoreInCtxtDescForVCS:bxt */ 2848621f407SFrançois Tigeot if (engine->disable_lite_restore_wa) 2858621f407SFrançois Tigeot engine->ctx_desc_template |= GEN8_CTX_FORCE_RESTORE; 286c0e85e96SFrançois Tigeot } 287c0e85e96SFrançois Tigeot 288c0e85e96SFrançois Tigeot /** 289c0e85e96SFrançois Tigeot * intel_lr_context_descriptor_update() - calculate & cache the descriptor 290c0e85e96SFrançois Tigeot * descriptor for a pinned context 291c0e85e96SFrançois Tigeot * 292c0e85e96SFrançois Tigeot * @ctx: Context to work on 293*1487f786SFrançois Tigeot * @engine: Engine the descriptor will be used with 294c0e85e96SFrançois Tigeot * 295c0e85e96SFrançois Tigeot * The context descriptor encodes various attributes of a context, 296c0e85e96SFrançois Tigeot * including its GTT address and some flags. Because it's fairly 297c0e85e96SFrançois Tigeot * expensive to calculate, we'll just do it once and cache the result, 298c0e85e96SFrançois Tigeot * which remains valid until the context is unpinned. 299c0e85e96SFrançois Tigeot * 300c0e85e96SFrançois Tigeot * This is what a descriptor looks like, from LSB to MSB: 301c0e85e96SFrançois Tigeot * bits 0-11: flags, GEN8_CTX_* (cached in ctx_desc_template) 302c0e85e96SFrançois Tigeot * bits 12-31: LRCA, GTT address of (the HWSP of) this context 303*1487f786SFrançois Tigeot * bits 32-52: ctx ID, a globally unique tag 304*1487f786SFrançois Tigeot * bits 53-54: mbz, reserved for use by hardware 305*1487f786SFrançois Tigeot * bits 55-63: group ID, currently unused and set to 0 306c0e85e96SFrançois Tigeot */ 307c0e85e96SFrançois Tigeot static void 308*1487f786SFrançois Tigeot intel_lr_context_descriptor_update(struct i915_gem_context *ctx, 3098621f407SFrançois Tigeot struct intel_engine_cs *engine) 310c0e85e96SFrançois Tigeot { 311*1487f786SFrançois Tigeot struct intel_context *ce = &ctx->engine[engine->id]; 312*1487f786SFrançois Tigeot u64 desc; 313c0e85e96SFrançois Tigeot 314*1487f786SFrançois Tigeot BUILD_BUG_ON(MAX_CONTEXT_HW_ID > (1<<GEN8_CTX_ID_WIDTH)); 315c0e85e96SFrançois Tigeot 316*1487f786SFrançois Tigeot desc = ctx->desc_template; /* bits 3-4 */ 317*1487f786SFrançois Tigeot desc |= engine->ctx_desc_template; /* bits 0-11 */ 318*1487f786SFrançois Tigeot desc |= ce->lrc_vma->node.start + LRC_PPHWSP_PN * PAGE_SIZE; 319*1487f786SFrançois Tigeot /* bits 12-31 */ 320*1487f786SFrançois Tigeot desc |= (u64)ctx->hw_id << GEN8_CTX_ID_SHIFT; /* bits 32-52 */ 321c0e85e96SFrançois Tigeot 322*1487f786SFrançois Tigeot ce->lrc_desc = desc; 323c0e85e96SFrançois Tigeot } 324c0e85e96SFrançois Tigeot 325*1487f786SFrançois Tigeot uint64_t intel_lr_context_descriptor(struct i915_gem_context *ctx, 3268621f407SFrançois Tigeot struct intel_engine_cs *engine) 327c0e85e96SFrançois Tigeot { 3288621f407SFrançois Tigeot return ctx->engine[engine->id].lrc_desc; 329c0e85e96SFrançois Tigeot } 330c0e85e96SFrançois Tigeot 331a05eeebfSFrançois Tigeot static void execlists_elsp_write(struct drm_i915_gem_request *rq0, 332a05eeebfSFrançois Tigeot struct drm_i915_gem_request *rq1) 3331b13d190SFrançois Tigeot { 334a05eeebfSFrançois Tigeot 3358621f407SFrançois Tigeot struct intel_engine_cs *engine = rq0->engine; 336*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = rq0->i915; 337a05eeebfSFrançois Tigeot uint64_t desc[2]; 3381b13d190SFrançois Tigeot 339a05eeebfSFrançois Tigeot if (rq1) { 3408621f407SFrançois Tigeot desc[1] = intel_lr_context_descriptor(rq1->ctx, rq1->engine); 341a05eeebfSFrançois Tigeot rq1->elsp_submitted++; 342a05eeebfSFrançois Tigeot } else { 343a05eeebfSFrançois Tigeot desc[1] = 0; 344a05eeebfSFrançois Tigeot } 3451b13d190SFrançois Tigeot 3468621f407SFrançois Tigeot desc[0] = intel_lr_context_descriptor(rq0->ctx, rq0->engine); 347a05eeebfSFrançois Tigeot rq0->elsp_submitted++; 3481b13d190SFrançois Tigeot 349a05eeebfSFrançois Tigeot /* You must always write both descriptors in the order below. */ 3508621f407SFrançois Tigeot I915_WRITE_FW(RING_ELSP(engine), upper_32_bits(desc[1])); 3518621f407SFrançois Tigeot I915_WRITE_FW(RING_ELSP(engine), lower_32_bits(desc[1])); 3522c9916cdSFrançois Tigeot 3538621f407SFrançois Tigeot I915_WRITE_FW(RING_ELSP(engine), upper_32_bits(desc[0])); 3541b13d190SFrançois Tigeot /* The context is automatically loaded after the following */ 3558621f407SFrançois Tigeot I915_WRITE_FW(RING_ELSP(engine), lower_32_bits(desc[0])); 3561b13d190SFrançois Tigeot 357a05eeebfSFrançois Tigeot /* ELSP is a wo register, use another nearby reg for posting */ 3588621f407SFrançois Tigeot POSTING_READ_FW(RING_EXECLIST_STATUS_LO(engine)); 3591b13d190SFrançois Tigeot } 3601b13d190SFrançois Tigeot 3618621f407SFrançois Tigeot static void 3628621f407SFrançois Tigeot execlists_update_context_pdps(struct i915_hw_ppgtt *ppgtt, u32 *reg_state) 3631b13d190SFrançois Tigeot { 36419c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 3); 36519c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 2); 36619c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 1); 36719c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 0); 36819c468b4SFrançois Tigeot } 36919c468b4SFrançois Tigeot 3708621f407SFrançois Tigeot static void execlists_update_context(struct drm_i915_gem_request *rq) 3718621f407SFrançois Tigeot { 3728621f407SFrançois Tigeot struct intel_engine_cs *engine = rq->engine; 3738621f407SFrançois Tigeot struct i915_hw_ppgtt *ppgtt = rq->ctx->ppgtt; 3748621f407SFrançois Tigeot uint32_t *reg_state = rq->ctx->engine[engine->id].lrc_reg_state; 3758621f407SFrançois Tigeot 3768621f407SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = rq->tail; 3778621f407SFrançois Tigeot 3788621f407SFrançois Tigeot /* True 32b PPGTT with dynamic page allocation: update PDP 3798621f407SFrançois Tigeot * registers and point the unallocated PDPs to scratch page. 3808621f407SFrançois Tigeot * PML4 is allocated during ppgtt init, so this is not needed 3818621f407SFrançois Tigeot * in 48-bit mode. 3828621f407SFrançois Tigeot */ 3838621f407SFrançois Tigeot if (ppgtt && !USES_FULL_48BIT_PPGTT(ppgtt->base.dev)) 3848621f407SFrançois Tigeot execlists_update_context_pdps(ppgtt, reg_state); 3851b13d190SFrançois Tigeot } 3861b13d190SFrançois Tigeot 387a05eeebfSFrançois Tigeot static void execlists_submit_requests(struct drm_i915_gem_request *rq0, 388a05eeebfSFrançois Tigeot struct drm_i915_gem_request *rq1) 3891b13d190SFrançois Tigeot { 3908621f407SFrançois Tigeot struct drm_i915_private *dev_priv = rq0->i915; 3918621f407SFrançois Tigeot unsigned int fw_domains = rq0->engine->fw_domains; 3928621f407SFrançois Tigeot 393a05eeebfSFrançois Tigeot execlists_update_context(rq0); 3941b13d190SFrançois Tigeot 395a05eeebfSFrançois Tigeot if (rq1) 396a05eeebfSFrançois Tigeot execlists_update_context(rq1); 3971b13d190SFrançois Tigeot 3988621f407SFrançois Tigeot spin_lock_irq(&dev_priv->uncore.lock); 3998621f407SFrançois Tigeot intel_uncore_forcewake_get__locked(dev_priv, fw_domains); 4008621f407SFrançois Tigeot 401a05eeebfSFrançois Tigeot execlists_elsp_write(rq0, rq1); 4028621f407SFrançois Tigeot 4038621f407SFrançois Tigeot intel_uncore_forcewake_put__locked(dev_priv, fw_domains); 4048621f407SFrançois Tigeot spin_unlock_irq(&dev_priv->uncore.lock); 4051b13d190SFrançois Tigeot } 4061b13d190SFrançois Tigeot 407*1487f786SFrançois Tigeot static inline void execlists_context_status_change( 408*1487f786SFrançois Tigeot struct drm_i915_gem_request *rq, 409*1487f786SFrançois Tigeot unsigned long status) 410*1487f786SFrançois Tigeot { 411*1487f786SFrançois Tigeot /* 412*1487f786SFrançois Tigeot * Only used when GVT-g is enabled now. When GVT-g is disabled, 413*1487f786SFrançois Tigeot * The compiler should eliminate this function as dead-code. 414*1487f786SFrançois Tigeot */ 415*1487f786SFrançois Tigeot if (!IS_ENABLED(CONFIG_DRM_I915_GVT)) 416*1487f786SFrançois Tigeot return; 417*1487f786SFrançois Tigeot 418*1487f786SFrançois Tigeot atomic_notifier_call_chain(&rq->ctx->status_notifier, status, rq); 419*1487f786SFrançois Tigeot } 420*1487f786SFrançois Tigeot 4218621f407SFrançois Tigeot static void execlists_context_unqueue(struct intel_engine_cs *engine) 4221b13d190SFrançois Tigeot { 4232c9916cdSFrançois Tigeot struct drm_i915_gem_request *req0 = NULL, *req1 = NULL; 4248621f407SFrançois Tigeot struct drm_i915_gem_request *cursor, *tmp; 4251b13d190SFrançois Tigeot 4268621f407SFrançois Tigeot assert_spin_locked(&engine->execlist_lock); 4271b13d190SFrançois Tigeot 42819c468b4SFrançois Tigeot /* 42919c468b4SFrançois Tigeot * If irqs are not active generate a warning as batches that finish 43019c468b4SFrançois Tigeot * without the irqs may get lost and a GPU Hang may occur. 43119c468b4SFrançois Tigeot */ 432*1487f786SFrançois Tigeot WARN_ON(!intel_irqs_enabled(engine->i915)); 4331b13d190SFrançois Tigeot 4341b13d190SFrançois Tigeot /* Try to read in pairs */ 4358621f407SFrançois Tigeot list_for_each_entry_safe(cursor, tmp, &engine->execlist_queue, 4361b13d190SFrançois Tigeot execlist_link) { 4371b13d190SFrançois Tigeot if (!req0) { 4381b13d190SFrançois Tigeot req0 = cursor; 4391b13d190SFrançois Tigeot } else if (req0->ctx == cursor->ctx) { 4401b13d190SFrançois Tigeot /* Same ctx: ignore first request, as second request 4411b13d190SFrançois Tigeot * will update tail past first request's workload */ 4421b13d190SFrançois Tigeot cursor->elsp_submitted = req0->elsp_submitted; 443*1487f786SFrançois Tigeot list_del(&req0->execlist_link); 444*1487f786SFrançois Tigeot i915_gem_request_unreference(req0); 4451b13d190SFrançois Tigeot req0 = cursor; 4461b13d190SFrançois Tigeot } else { 447*1487f786SFrançois Tigeot if (IS_ENABLED(CONFIG_DRM_I915_GVT)) { 448*1487f786SFrançois Tigeot /* 449*1487f786SFrançois Tigeot * req0 (after merged) ctx requires single 450*1487f786SFrançois Tigeot * submission, stop picking 451*1487f786SFrançois Tigeot */ 452*1487f786SFrançois Tigeot if (req0->ctx->execlists_force_single_submission) 453*1487f786SFrançois Tigeot break; 454*1487f786SFrançois Tigeot /* 455*1487f786SFrançois Tigeot * req0 ctx doesn't require single submission, 456*1487f786SFrançois Tigeot * but next req ctx requires, stop picking 457*1487f786SFrançois Tigeot */ 458*1487f786SFrançois Tigeot if (cursor->ctx->execlists_force_single_submission) 459*1487f786SFrançois Tigeot break; 460*1487f786SFrançois Tigeot } 4611b13d190SFrançois Tigeot req1 = cursor; 4628621f407SFrançois Tigeot WARN_ON(req1->elsp_submitted); 4631b13d190SFrançois Tigeot break; 4641b13d190SFrançois Tigeot } 4651b13d190SFrançois Tigeot } 4661b13d190SFrançois Tigeot 4678621f407SFrançois Tigeot if (unlikely(!req0)) 4688621f407SFrançois Tigeot return; 4698621f407SFrançois Tigeot 470*1487f786SFrançois Tigeot execlists_context_status_change(req0, INTEL_CONTEXT_SCHEDULE_IN); 471*1487f786SFrançois Tigeot 472*1487f786SFrançois Tigeot if (req1) 473*1487f786SFrançois Tigeot execlists_context_status_change(req1, 474*1487f786SFrançois Tigeot INTEL_CONTEXT_SCHEDULE_IN); 475*1487f786SFrançois Tigeot 4768621f407SFrançois Tigeot if (req0->elsp_submitted & engine->idle_lite_restore_wa) { 477477eb7f9SFrançois Tigeot /* 4788621f407SFrançois Tigeot * WaIdleLiteRestore: make sure we never cause a lite restore 4798621f407SFrançois Tigeot * with HEAD==TAIL. 4808621f407SFrançois Tigeot * 4818621f407SFrançois Tigeot * Apply the wa NOOPS to prevent ring:HEAD == req:TAIL as we 4828621f407SFrançois Tigeot * resubmit the request. See gen8_emit_request() for where we 4838621f407SFrançois Tigeot * prepare the padding after the end of the request. 484477eb7f9SFrançois Tigeot */ 485477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf; 486477eb7f9SFrançois Tigeot 4878621f407SFrançois Tigeot ringbuf = req0->ctx->engine[engine->id].ringbuf; 488477eb7f9SFrançois Tigeot req0->tail += 8; 489477eb7f9SFrançois Tigeot req0->tail &= ringbuf->size - 1; 490477eb7f9SFrançois Tigeot } 4911b13d190SFrançois Tigeot 492a05eeebfSFrançois Tigeot execlists_submit_requests(req0, req1); 4931b13d190SFrançois Tigeot } 4941b13d190SFrançois Tigeot 4958621f407SFrançois Tigeot static unsigned int 496*1487f786SFrançois Tigeot execlists_check_remove_request(struct intel_engine_cs *engine, u32 ctx_id) 4971b13d190SFrançois Tigeot { 4982c9916cdSFrançois Tigeot struct drm_i915_gem_request *head_req; 4991b13d190SFrançois Tigeot 5008621f407SFrançois Tigeot assert_spin_locked(&engine->execlist_lock); 5011b13d190SFrançois Tigeot 5028621f407SFrançois Tigeot head_req = list_first_entry_or_null(&engine->execlist_queue, 5032c9916cdSFrançois Tigeot struct drm_i915_gem_request, 5041b13d190SFrançois Tigeot execlist_link); 5051b13d190SFrançois Tigeot 506*1487f786SFrançois Tigeot if (WARN_ON(!head_req || (head_req->ctx_hw_id != ctx_id))) 5078621f407SFrançois Tigeot return 0; 5088621f407SFrançois Tigeot 5098621f407SFrançois Tigeot WARN(head_req->elsp_submitted == 0, "Never submitted head request\n"); 5108621f407SFrançois Tigeot 5118621f407SFrançois Tigeot if (--head_req->elsp_submitted > 0) 5128621f407SFrançois Tigeot return 0; 5138621f407SFrançois Tigeot 514*1487f786SFrançois Tigeot execlists_context_status_change(head_req, INTEL_CONTEXT_SCHEDULE_OUT); 515*1487f786SFrançois Tigeot 516*1487f786SFrançois Tigeot list_del(&head_req->execlist_link); 517*1487f786SFrançois Tigeot i915_gem_request_unreference(head_req); 5188621f407SFrançois Tigeot 5198621f407SFrançois Tigeot return 1; 5201b13d190SFrançois Tigeot } 5211b13d190SFrançois Tigeot 5228621f407SFrançois Tigeot static u32 5238621f407SFrançois Tigeot get_context_status(struct intel_engine_cs *engine, unsigned int read_pointer, 5248621f407SFrançois Tigeot u32 *context_id) 525c0e85e96SFrançois Tigeot { 526*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 5278621f407SFrançois Tigeot u32 status; 528c0e85e96SFrançois Tigeot 5298621f407SFrançois Tigeot read_pointer %= GEN8_CSB_ENTRIES; 530c0e85e96SFrançois Tigeot 5318621f407SFrançois Tigeot status = I915_READ_FW(RING_CONTEXT_STATUS_BUF_LO(engine, read_pointer)); 5328621f407SFrançois Tigeot 5338621f407SFrançois Tigeot if (status & GEN8_CTX_STATUS_IDLE_ACTIVE) 5348621f407SFrançois Tigeot return 0; 5358621f407SFrançois Tigeot 5368621f407SFrançois Tigeot *context_id = I915_READ_FW(RING_CONTEXT_STATUS_BUF_HI(engine, 5378621f407SFrançois Tigeot read_pointer)); 5388621f407SFrançois Tigeot 5398621f407SFrançois Tigeot return status; 540c0e85e96SFrançois Tigeot } 541c0e85e96SFrançois Tigeot 5421b13d190SFrançois Tigeot /** 5432c9916cdSFrançois Tigeot * intel_lrc_irq_handler() - handle Context Switch interrupts 544*1487f786SFrançois Tigeot * @data: tasklet handler passed in unsigned long 5451b13d190SFrançois Tigeot * 5461b13d190SFrançois Tigeot * Check the unread Context Status Buffers and manage the submission of new 5471b13d190SFrançois Tigeot * contexts to the ELSP accordingly. 5481b13d190SFrançois Tigeot */ 5498621f407SFrançois Tigeot static void intel_lrc_irq_handler(unsigned long data) 5501b13d190SFrançois Tigeot { 5518621f407SFrançois Tigeot struct intel_engine_cs *engine = (struct intel_engine_cs *)data; 552*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 5531b13d190SFrançois Tigeot u32 status_pointer; 5548621f407SFrançois Tigeot unsigned int read_pointer, write_pointer; 5558621f407SFrançois Tigeot u32 csb[GEN8_CSB_ENTRIES][2]; 5568621f407SFrançois Tigeot unsigned int csb_read = 0, i; 5578621f407SFrançois Tigeot unsigned int submit_contexts = 0; 5581b13d190SFrançois Tigeot 5598621f407SFrançois Tigeot intel_uncore_forcewake_get(dev_priv, engine->fw_domains); 5601b13d190SFrançois Tigeot 5618621f407SFrançois Tigeot status_pointer = I915_READ_FW(RING_CONTEXT_STATUS_PTR(engine)); 5628621f407SFrançois Tigeot 5638621f407SFrançois Tigeot read_pointer = engine->next_context_status_buffer; 564c0e85e96SFrançois Tigeot write_pointer = GEN8_CSB_WRITE_PTR(status_pointer); 5651b13d190SFrançois Tigeot if (read_pointer > write_pointer) 566a05eeebfSFrançois Tigeot write_pointer += GEN8_CSB_ENTRIES; 5671b13d190SFrançois Tigeot 5681b13d190SFrançois Tigeot while (read_pointer < write_pointer) { 5698621f407SFrançois Tigeot if (WARN_ON_ONCE(csb_read == GEN8_CSB_ENTRIES)) 5708621f407SFrançois Tigeot break; 5718621f407SFrançois Tigeot csb[csb_read][0] = get_context_status(engine, ++read_pointer, 5728621f407SFrançois Tigeot &csb[csb_read][1]); 5738621f407SFrançois Tigeot csb_read++; 5748621f407SFrançois Tigeot } 575c0e85e96SFrançois Tigeot 5768621f407SFrançois Tigeot engine->next_context_status_buffer = write_pointer % GEN8_CSB_ENTRIES; 577a05eeebfSFrançois Tigeot 5788621f407SFrançois Tigeot /* Update the read pointer to the old write pointer. Manual ringbuffer 5798621f407SFrançois Tigeot * management ftw </sarcasm> */ 5808621f407SFrançois Tigeot I915_WRITE_FW(RING_CONTEXT_STATUS_PTR(engine), 5818621f407SFrançois Tigeot _MASKED_FIELD(GEN8_CSB_READ_PTR_MASK, 5828621f407SFrançois Tigeot engine->next_context_status_buffer << 8)); 5831b13d190SFrançois Tigeot 5848621f407SFrançois Tigeot intel_uncore_forcewake_put(dev_priv, engine->fw_domains); 5858621f407SFrançois Tigeot 5868621f407SFrançois Tigeot lockmgr(&engine->execlist_lock, LK_EXCLUSIVE); 5878621f407SFrançois Tigeot 5888621f407SFrançois Tigeot for (i = 0; i < csb_read; i++) { 5898621f407SFrançois Tigeot if (unlikely(csb[i][0] & GEN8_CTX_STATUS_PREEMPTED)) { 5908621f407SFrançois Tigeot if (csb[i][0] & GEN8_CTX_STATUS_LITE_RESTORE) { 5918621f407SFrançois Tigeot if (execlists_check_remove_request(engine, csb[i][1])) 5921b13d190SFrançois Tigeot WARN(1, "Lite Restored request removed from queue\n"); 5931b13d190SFrançois Tigeot } else 5941b13d190SFrançois Tigeot WARN(1, "Preemption without Lite Restore\n"); 5951b13d190SFrançois Tigeot } 5961b13d190SFrançois Tigeot 5978621f407SFrançois Tigeot if (csb[i][0] & (GEN8_CTX_STATUS_ACTIVE_IDLE | 5988621f407SFrançois Tigeot GEN8_CTX_STATUS_ELEMENT_SWITCH)) 5998621f407SFrançois Tigeot submit_contexts += 6008621f407SFrançois Tigeot execlists_check_remove_request(engine, csb[i][1]); 6011b13d190SFrançois Tigeot } 6021b13d190SFrançois Tigeot 6038621f407SFrançois Tigeot if (submit_contexts) { 6048621f407SFrançois Tigeot if (!engine->disable_lite_restore_wa || 6058621f407SFrançois Tigeot (csb[i][0] & GEN8_CTX_STATUS_ACTIVE_IDLE)) 6068621f407SFrançois Tigeot execlists_context_unqueue(engine); 607352ff8bdSFrançois Tigeot } 6081b13d190SFrançois Tigeot 6098621f407SFrançois Tigeot lockmgr(&engine->execlist_lock, LK_RELEASE); 6101b13d190SFrançois Tigeot 611c0e85e96SFrançois Tigeot if (unlikely(submit_contexts > 2)) 612c0e85e96SFrançois Tigeot DRM_ERROR("More than two context complete events?\n"); 6131b13d190SFrançois Tigeot } 6141b13d190SFrançois Tigeot 6158621f407SFrançois Tigeot static void execlists_context_queue(struct drm_i915_gem_request *request) 6161b13d190SFrançois Tigeot { 6178621f407SFrançois Tigeot struct intel_engine_cs *engine = request->engine; 6182c9916cdSFrançois Tigeot struct drm_i915_gem_request *cursor; 6191b13d190SFrançois Tigeot int num_elements = 0; 6201b13d190SFrançois Tigeot 6218621f407SFrançois Tigeot spin_lock_bh(&engine->execlist_lock); 6221b13d190SFrançois Tigeot 6238621f407SFrançois Tigeot list_for_each_entry(cursor, &engine->execlist_queue, execlist_link) 6241b13d190SFrançois Tigeot if (++num_elements > 2) 6251b13d190SFrançois Tigeot break; 6261b13d190SFrançois Tigeot 6271b13d190SFrançois Tigeot if (num_elements > 2) { 6282c9916cdSFrançois Tigeot struct drm_i915_gem_request *tail_req; 6291b13d190SFrançois Tigeot 6308621f407SFrançois Tigeot tail_req = list_last_entry(&engine->execlist_queue, 6312c9916cdSFrançois Tigeot struct drm_i915_gem_request, 6321b13d190SFrançois Tigeot execlist_link); 6331b13d190SFrançois Tigeot 634a05eeebfSFrançois Tigeot if (request->ctx == tail_req->ctx) { 6351b13d190SFrançois Tigeot WARN(tail_req->elsp_submitted != 0, 6361b13d190SFrançois Tigeot "More than 2 already-submitted reqs queued\n"); 637*1487f786SFrançois Tigeot list_del(&tail_req->execlist_link); 638*1487f786SFrançois Tigeot i915_gem_request_unreference(tail_req); 6391b13d190SFrançois Tigeot } 6401b13d190SFrançois Tigeot } 6411b13d190SFrançois Tigeot 642*1487f786SFrançois Tigeot i915_gem_request_reference(request); 6438621f407SFrançois Tigeot list_add_tail(&request->execlist_link, &engine->execlist_queue); 644*1487f786SFrançois Tigeot request->ctx_hw_id = request->ctx->hw_id; 6451b13d190SFrançois Tigeot if (num_elements == 0) 6468621f407SFrançois Tigeot execlists_context_unqueue(engine); 6471b13d190SFrançois Tigeot 6488621f407SFrançois Tigeot spin_unlock_bh(&engine->execlist_lock); 6491b13d190SFrançois Tigeot } 6501b13d190SFrançois Tigeot 651a05eeebfSFrançois Tigeot static int logical_ring_invalidate_all_caches(struct drm_i915_gem_request *req) 6521b13d190SFrançois Tigeot { 6538621f407SFrançois Tigeot struct intel_engine_cs *engine = req->engine; 6541b13d190SFrançois Tigeot uint32_t flush_domains; 6551b13d190SFrançois Tigeot int ret; 6561b13d190SFrançois Tigeot 6571b13d190SFrançois Tigeot flush_domains = 0; 6588621f407SFrançois Tigeot if (engine->gpu_caches_dirty) 6591b13d190SFrançois Tigeot flush_domains = I915_GEM_GPU_DOMAINS; 6601b13d190SFrançois Tigeot 6618621f407SFrançois Tigeot ret = engine->emit_flush(req, I915_GEM_GPU_DOMAINS, flush_domains); 6621b13d190SFrançois Tigeot if (ret) 6631b13d190SFrançois Tigeot return ret; 6641b13d190SFrançois Tigeot 6658621f407SFrançois Tigeot engine->gpu_caches_dirty = false; 6661b13d190SFrançois Tigeot return 0; 6671b13d190SFrançois Tigeot } 6681b13d190SFrançois Tigeot 669a05eeebfSFrançois Tigeot static int execlists_move_to_gpu(struct drm_i915_gem_request *req, 6701b13d190SFrançois Tigeot struct list_head *vmas) 6711b13d190SFrançois Tigeot { 6728621f407SFrançois Tigeot const unsigned other_rings = ~intel_engine_flag(req->engine); 6731b13d190SFrançois Tigeot struct i915_vma *vma; 6741b13d190SFrançois Tigeot uint32_t flush_domains = 0; 6751b13d190SFrançois Tigeot bool flush_chipset = false; 6761b13d190SFrançois Tigeot int ret; 6771b13d190SFrançois Tigeot 6781b13d190SFrançois Tigeot list_for_each_entry(vma, vmas, exec_list) { 6791b13d190SFrançois Tigeot struct drm_i915_gem_object *obj = vma->obj; 6801b13d190SFrançois Tigeot 68119c468b4SFrançois Tigeot if (obj->active & other_rings) { 6828621f407SFrançois Tigeot ret = i915_gem_object_sync(obj, req->engine, &req); 6831b13d190SFrançois Tigeot if (ret) 6841b13d190SFrançois Tigeot return ret; 68519c468b4SFrançois Tigeot } 6861b13d190SFrançois Tigeot 6871b13d190SFrançois Tigeot if (obj->base.write_domain & I915_GEM_DOMAIN_CPU) 6881b13d190SFrançois Tigeot flush_chipset |= i915_gem_clflush_object(obj, false); 6891b13d190SFrançois Tigeot 6901b13d190SFrançois Tigeot flush_domains |= obj->base.write_domain; 6911b13d190SFrançois Tigeot } 6921b13d190SFrançois Tigeot 6931b13d190SFrançois Tigeot if (flush_domains & I915_GEM_DOMAIN_GTT) 6941b13d190SFrançois Tigeot wmb(); 6951b13d190SFrançois Tigeot 6961b13d190SFrançois Tigeot /* Unconditionally invalidate gpu caches and ensure that we do flush 6971b13d190SFrançois Tigeot * any residual writes from the previous batch. 6981b13d190SFrançois Tigeot */ 699a05eeebfSFrançois Tigeot return logical_ring_invalidate_all_caches(req); 7001b13d190SFrançois Tigeot } 7011b13d190SFrançois Tigeot 702a05eeebfSFrançois Tigeot int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request) 70319c468b4SFrançois Tigeot { 704*1487f786SFrançois Tigeot struct intel_engine_cs *engine = request->engine; 705*1487f786SFrançois Tigeot struct intel_context *ce = &request->ctx->engine[engine->id]; 706*1487f786SFrançois Tigeot int ret; 70719c468b4SFrançois Tigeot 708*1487f786SFrançois Tigeot /* Flush enough space to reduce the likelihood of waiting after 709*1487f786SFrançois Tigeot * we start building the request - in which case we will just 710*1487f786SFrançois Tigeot * have to repeat work. 711*1487f786SFrançois Tigeot */ 712*1487f786SFrançois Tigeot request->reserved_space += EXECLISTS_REQUEST_SIZE; 713*1487f786SFrançois Tigeot 714*1487f786SFrançois Tigeot if (!ce->state) { 715*1487f786SFrançois Tigeot ret = execlists_context_deferred_alloc(request->ctx, engine); 716*1487f786SFrançois Tigeot if (ret) 717*1487f786SFrançois Tigeot return ret; 718*1487f786SFrançois Tigeot } 719*1487f786SFrançois Tigeot 720*1487f786SFrançois Tigeot request->ringbuf = ce->ringbuf; 721a05eeebfSFrançois Tigeot 722c0e85e96SFrançois Tigeot if (i915.enable_guc_submission) { 723c0e85e96SFrançois Tigeot /* 724c0e85e96SFrançois Tigeot * Check that the GuC has space for the request before 725c0e85e96SFrançois Tigeot * going any further, as the i915_add_request() call 726c0e85e96SFrançois Tigeot * later on mustn't fail ... 727c0e85e96SFrançois Tigeot */ 728*1487f786SFrançois Tigeot ret = i915_guc_wq_check_space(request); 72919c468b4SFrançois Tigeot if (ret) 73019c468b4SFrançois Tigeot return ret; 73119c468b4SFrançois Tigeot } 73219c468b4SFrançois Tigeot 733*1487f786SFrançois Tigeot ret = intel_lr_context_pin(request->ctx, engine); 734*1487f786SFrançois Tigeot if (ret) 735*1487f786SFrançois Tigeot return ret; 736c0e85e96SFrançois Tigeot 737*1487f786SFrançois Tigeot ret = intel_ring_begin(request, 0); 738*1487f786SFrançois Tigeot if (ret) 739*1487f786SFrançois Tigeot goto err_unpin; 740*1487f786SFrançois Tigeot 741*1487f786SFrançois Tigeot if (!ce->initialised) { 742*1487f786SFrançois Tigeot ret = engine->init_context(request); 743*1487f786SFrançois Tigeot if (ret) 744*1487f786SFrançois Tigeot goto err_unpin; 745*1487f786SFrançois Tigeot 746*1487f786SFrançois Tigeot ce->initialised = true; 747*1487f786SFrançois Tigeot } 748*1487f786SFrançois Tigeot 749*1487f786SFrançois Tigeot /* Note that after this point, we have committed to using 750*1487f786SFrançois Tigeot * this request as it is being used to both track the 751*1487f786SFrançois Tigeot * state of engine initialisation and liveness of the 752*1487f786SFrançois Tigeot * golden renderstate above. Think twice before you try 753*1487f786SFrançois Tigeot * to cancel/unwind this request now. 754*1487f786SFrançois Tigeot */ 755*1487f786SFrançois Tigeot 756*1487f786SFrançois Tigeot request->reserved_space -= EXECLISTS_REQUEST_SIZE; 757*1487f786SFrançois Tigeot return 0; 758*1487f786SFrançois Tigeot 759*1487f786SFrançois Tigeot err_unpin: 760*1487f786SFrançois Tigeot intel_lr_context_unpin(request->ctx, engine); 761c0e85e96SFrançois Tigeot return ret; 76219c468b4SFrançois Tigeot } 76319c468b4SFrançois Tigeot 76419c468b4SFrançois Tigeot /* 76519c468b4SFrançois Tigeot * intel_logical_ring_advance_and_submit() - advance the tail and submit the workload 766a05eeebfSFrançois Tigeot * @request: Request to advance the logical ringbuffer of. 76719c468b4SFrançois Tigeot * 76819c468b4SFrançois Tigeot * The tail is updated in our logical ringbuffer struct, not in the actual context. What 76919c468b4SFrançois Tigeot * really happens during submission is that the context and current tail will be placed 77019c468b4SFrançois Tigeot * on a queue waiting for the ELSP to be ready to accept a new context submission. At that 77119c468b4SFrançois Tigeot * point, the tail *inside* the context is updated and the ELSP written to. 77219c468b4SFrançois Tigeot */ 773c0e85e96SFrançois Tigeot static int 774a05eeebfSFrançois Tigeot intel_logical_ring_advance_and_submit(struct drm_i915_gem_request *request) 77519c468b4SFrançois Tigeot { 776c0e85e96SFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 7778621f407SFrançois Tigeot struct intel_engine_cs *engine = request->engine; 77819c468b4SFrançois Tigeot 779c0e85e96SFrançois Tigeot intel_logical_ring_advance(ringbuf); 780c0e85e96SFrançois Tigeot request->tail = ringbuf->tail; 78119c468b4SFrançois Tigeot 782c0e85e96SFrançois Tigeot /* 783c0e85e96SFrançois Tigeot * Here we add two extra NOOPs as padding to avoid 784c0e85e96SFrançois Tigeot * lite restore of a context with HEAD==TAIL. 785c0e85e96SFrançois Tigeot * 786c0e85e96SFrançois Tigeot * Caller must reserve WA_TAIL_DWORDS for us! 787c0e85e96SFrançois Tigeot */ 788c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 789c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 790c0e85e96SFrançois Tigeot intel_logical_ring_advance(ringbuf); 791352ff8bdSFrançois Tigeot 7928621f407SFrançois Tigeot if (intel_engine_stopped(engine)) 793c0e85e96SFrançois Tigeot return 0; 794c0e85e96SFrançois Tigeot 795*1487f786SFrançois Tigeot /* We keep the previous context alive until we retire the following 796*1487f786SFrançois Tigeot * request. This ensures that any the context object is still pinned 797*1487f786SFrançois Tigeot * for any residual writes the HW makes into it on the context switch 798*1487f786SFrançois Tigeot * into the next object following the breadcrumb. Otherwise, we may 799*1487f786SFrançois Tigeot * retire the context too early. 800*1487f786SFrançois Tigeot */ 801*1487f786SFrançois Tigeot request->previous_context = engine->last_context; 802c0e85e96SFrançois Tigeot engine->last_context = request->ctx; 80319c468b4SFrançois Tigeot 804*1487f786SFrançois Tigeot if (i915.enable_guc_submission) 805*1487f786SFrançois Tigeot i915_guc_submit(request); 806352ff8bdSFrançois Tigeot else 807a05eeebfSFrançois Tigeot execlists_context_queue(request); 808c0e85e96SFrançois Tigeot 809c0e85e96SFrançois Tigeot return 0; 81019c468b4SFrançois Tigeot } 81119c468b4SFrançois Tigeot 8121b13d190SFrançois Tigeot /** 8131b13d190SFrançois Tigeot * execlists_submission() - submit a batchbuffer for execution, Execlists style 814*1487f786SFrançois Tigeot * @params: execbuffer call parameters. 8151b13d190SFrançois Tigeot * @args: execbuffer call arguments. 8161b13d190SFrançois Tigeot * @vmas: list of vmas. 8171b13d190SFrançois Tigeot * 8181b13d190SFrançois Tigeot * This is the evil twin version of i915_gem_ringbuffer_submission. It abstracts 8191b13d190SFrançois Tigeot * away the submission details of the execbuffer ioctl call. 8201b13d190SFrançois Tigeot * 8211b13d190SFrançois Tigeot * Return: non-zero if the submission fails. 8221b13d190SFrançois Tigeot */ 823a05eeebfSFrançois Tigeot int intel_execlists_submission(struct i915_execbuffer_params *params, 8241b13d190SFrançois Tigeot struct drm_i915_gem_execbuffer2 *args, 825a05eeebfSFrançois Tigeot struct list_head *vmas) 8261b13d190SFrançois Tigeot { 827a05eeebfSFrançois Tigeot struct drm_device *dev = params->dev; 8288621f407SFrançois Tigeot struct intel_engine_cs *engine = params->engine; 8291b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 8308621f407SFrançois Tigeot struct intel_ringbuffer *ringbuf = params->ctx->engine[engine->id].ringbuf; 831a05eeebfSFrançois Tigeot u64 exec_start; 8321b13d190SFrançois Tigeot int instp_mode; 8331b13d190SFrançois Tigeot u32 instp_mask; 8341b13d190SFrançois Tigeot int ret; 8351b13d190SFrançois Tigeot 8361b13d190SFrançois Tigeot instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK; 8371b13d190SFrançois Tigeot instp_mask = I915_EXEC_CONSTANTS_MASK; 8381b13d190SFrançois Tigeot switch (instp_mode) { 8391b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_GENERAL: 8401b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_ABSOLUTE: 8411b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_SURFACE: 8428621f407SFrançois Tigeot if (instp_mode != 0 && engine != &dev_priv->engine[RCS]) { 8431b13d190SFrançois Tigeot DRM_DEBUG("non-0 rel constants mode on non-RCS\n"); 8441b13d190SFrançois Tigeot return -EINVAL; 8451b13d190SFrançois Tigeot } 8461b13d190SFrançois Tigeot 8471b13d190SFrançois Tigeot if (instp_mode != dev_priv->relative_constants_mode) { 8481b13d190SFrançois Tigeot if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) { 8491b13d190SFrançois Tigeot DRM_DEBUG("rel surface constants mode invalid on gen5+\n"); 8501b13d190SFrançois Tigeot return -EINVAL; 8511b13d190SFrançois Tigeot } 8521b13d190SFrançois Tigeot 8531b13d190SFrançois Tigeot /* The HW changed the meaning on this bit on gen6 */ 8541b13d190SFrançois Tigeot instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE; 8551b13d190SFrançois Tigeot } 8561b13d190SFrançois Tigeot break; 8571b13d190SFrançois Tigeot default: 8581b13d190SFrançois Tigeot DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode); 8591b13d190SFrançois Tigeot return -EINVAL; 8601b13d190SFrançois Tigeot } 8611b13d190SFrançois Tigeot 8621b13d190SFrançois Tigeot if (args->flags & I915_EXEC_GEN7_SOL_RESET) { 8631b13d190SFrançois Tigeot DRM_DEBUG("sol reset is gen7 only\n"); 8641b13d190SFrançois Tigeot return -EINVAL; 8651b13d190SFrançois Tigeot } 8661b13d190SFrançois Tigeot 867a05eeebfSFrançois Tigeot ret = execlists_move_to_gpu(params->request, vmas); 8681b13d190SFrançois Tigeot if (ret) 8691b13d190SFrançois Tigeot return ret; 8701b13d190SFrançois Tigeot 8718621f407SFrançois Tigeot if (engine == &dev_priv->engine[RCS] && 8721b13d190SFrançois Tigeot instp_mode != dev_priv->relative_constants_mode) { 8738621f407SFrançois Tigeot ret = intel_ring_begin(params->request, 4); 8741b13d190SFrançois Tigeot if (ret) 8751b13d190SFrançois Tigeot return ret; 8761b13d190SFrançois Tigeot 8771b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 8781b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1)); 879aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, INSTPM); 8801b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode); 8811b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 8821b13d190SFrançois Tigeot 8831b13d190SFrançois Tigeot dev_priv->relative_constants_mode = instp_mode; 8841b13d190SFrançois Tigeot } 8851b13d190SFrançois Tigeot 886a05eeebfSFrançois Tigeot exec_start = params->batch_obj_vm_offset + 887a05eeebfSFrançois Tigeot args->batch_start_offset; 888a05eeebfSFrançois Tigeot 8898621f407SFrançois Tigeot ret = engine->emit_bb_start(params->request, exec_start, params->dispatch_flags); 8901b13d190SFrançois Tigeot if (ret) 8911b13d190SFrançois Tigeot return ret; 8921b13d190SFrançois Tigeot 893a05eeebfSFrançois Tigeot trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags); 894477eb7f9SFrançois Tigeot 895a05eeebfSFrançois Tigeot i915_gem_execbuffer_move_to_active(vmas, params->request); 8961b13d190SFrançois Tigeot 8971b13d190SFrançois Tigeot return 0; 8981b13d190SFrançois Tigeot } 8991b13d190SFrançois Tigeot 900*1487f786SFrançois Tigeot void intel_execlists_cancel_requests(struct intel_engine_cs *engine) 9012c9916cdSFrançois Tigeot { 9022c9916cdSFrançois Tigeot struct drm_i915_gem_request *req, *tmp; 903*1487f786SFrançois Tigeot LINUX_LIST_HEAD(cancel_list); 9042c9916cdSFrançois Tigeot 905*1487f786SFrançois Tigeot WARN_ON(!mutex_is_locked(&engine->i915->dev->struct_mutex)); 9062c9916cdSFrançois Tigeot 9078621f407SFrançois Tigeot spin_lock_bh(&engine->execlist_lock); 908*1487f786SFrançois Tigeot list_replace_init(&engine->execlist_queue, &cancel_list); 9098621f407SFrançois Tigeot spin_unlock_bh(&engine->execlist_lock); 9102c9916cdSFrançois Tigeot 911*1487f786SFrançois Tigeot list_for_each_entry_safe(req, tmp, &cancel_list, execlist_link) { 9122c9916cdSFrançois Tigeot list_del(&req->execlist_link); 9132c9916cdSFrançois Tigeot i915_gem_request_unreference(req); 9142c9916cdSFrançois Tigeot } 9152c9916cdSFrançois Tigeot } 9162c9916cdSFrançois Tigeot 9178621f407SFrançois Tigeot void intel_logical_ring_stop(struct intel_engine_cs *engine) 9181b13d190SFrançois Tigeot { 919*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 9201b13d190SFrançois Tigeot int ret; 9211b13d190SFrançois Tigeot 9228621f407SFrançois Tigeot if (!intel_engine_initialized(engine)) 9231b13d190SFrançois Tigeot return; 9241b13d190SFrançois Tigeot 9258621f407SFrançois Tigeot ret = intel_engine_idle(engine); 9268621f407SFrançois Tigeot if (ret) 9271b13d190SFrançois Tigeot DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n", 9288621f407SFrançois Tigeot engine->name, ret); 9291b13d190SFrançois Tigeot 9301b13d190SFrançois Tigeot /* TODO: Is this correct with Execlists enabled? */ 9318621f407SFrançois Tigeot I915_WRITE_MODE(engine, _MASKED_BIT_ENABLE(STOP_RING)); 932*1487f786SFrançois Tigeot if (intel_wait_for_register(dev_priv, 933*1487f786SFrançois Tigeot RING_MI_MODE(engine->mmio_base), 934*1487f786SFrançois Tigeot MODE_IDLE, MODE_IDLE, 935*1487f786SFrançois Tigeot 1000)) { 9368621f407SFrançois Tigeot DRM_ERROR("%s :timed out trying to stop ring\n", engine->name); 9371b13d190SFrançois Tigeot return; 9381b13d190SFrançois Tigeot } 9398621f407SFrançois Tigeot I915_WRITE_MODE(engine, _MASKED_BIT_DISABLE(STOP_RING)); 9401b13d190SFrançois Tigeot } 9411b13d190SFrançois Tigeot 942a05eeebfSFrançois Tigeot int logical_ring_flush_all_caches(struct drm_i915_gem_request *req) 9431b13d190SFrançois Tigeot { 9448621f407SFrançois Tigeot struct intel_engine_cs *engine = req->engine; 9451b13d190SFrançois Tigeot int ret; 9461b13d190SFrançois Tigeot 9478621f407SFrançois Tigeot if (!engine->gpu_caches_dirty) 9481b13d190SFrançois Tigeot return 0; 9491b13d190SFrançois Tigeot 9508621f407SFrançois Tigeot ret = engine->emit_flush(req, 0, I915_GEM_GPU_DOMAINS); 9511b13d190SFrançois Tigeot if (ret) 9521b13d190SFrançois Tigeot return ret; 9531b13d190SFrançois Tigeot 9548621f407SFrançois Tigeot engine->gpu_caches_dirty = false; 9551b13d190SFrançois Tigeot return 0; 9561b13d190SFrançois Tigeot } 9571b13d190SFrançois Tigeot 958*1487f786SFrançois Tigeot static int intel_lr_context_pin(struct i915_gem_context *ctx, 9598621f407SFrançois Tigeot struct intel_engine_cs *engine) 9601b13d190SFrançois Tigeot { 961*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = ctx->i915; 962*1487f786SFrançois Tigeot struct intel_context *ce = &ctx->engine[engine->id]; 963*1487f786SFrançois Tigeot void *vaddr; 9648621f407SFrançois Tigeot u32 *lrc_reg_state; 965c0e85e96SFrançois Tigeot int ret; 9662c9916cdSFrançois Tigeot 967*1487f786SFrançois Tigeot lockdep_assert_held(&ctx->i915->dev->struct_mutex); 968c0e85e96SFrançois Tigeot 969*1487f786SFrançois Tigeot if (ce->pin_count++) 970*1487f786SFrançois Tigeot return 0; 971*1487f786SFrançois Tigeot 972*1487f786SFrançois Tigeot ret = i915_gem_obj_ggtt_pin(ce->state, GEN8_LR_CONTEXT_ALIGN, 973352ff8bdSFrançois Tigeot PIN_OFFSET_BIAS | GUC_WOPCM_TOP); 9742c9916cdSFrançois Tigeot if (ret) 975*1487f786SFrançois Tigeot goto err; 9762c9916cdSFrançois Tigeot 977*1487f786SFrançois Tigeot vaddr = i915_gem_object_pin_map(ce->state); 9788621f407SFrançois Tigeot if (IS_ERR(vaddr)) { 9798621f407SFrançois Tigeot ret = PTR_ERR(vaddr); 980c0e85e96SFrançois Tigeot goto unpin_ctx_obj; 981c0e85e96SFrançois Tigeot } 982c0e85e96SFrançois Tigeot 983*1487f786SFrançois Tigeot lrc_reg_state = vaddr + LRC_STATE_PN * PAGE_SIZE; 98419c468b4SFrançois Tigeot 985*1487f786SFrançois Tigeot ret = intel_pin_and_map_ringbuffer_obj(dev_priv, ce->ringbuf); 9868621f407SFrançois Tigeot if (ret) 9878621f407SFrançois Tigeot goto unpin_map; 9888621f407SFrançois Tigeot 989*1487f786SFrançois Tigeot i915_gem_context_reference(ctx); 990*1487f786SFrançois Tigeot ce->lrc_vma = i915_gem_obj_to_ggtt(ce->state); 9918621f407SFrançois Tigeot intel_lr_context_descriptor_update(ctx, engine); 992*1487f786SFrançois Tigeot 993*1487f786SFrançois Tigeot lrc_reg_state[CTX_RING_BUFFER_START+1] = ce->ringbuf->vma->node.start; 994*1487f786SFrançois Tigeot ce->lrc_reg_state = lrc_reg_state; 995*1487f786SFrançois Tigeot ce->state->dirty = true; 996352ff8bdSFrançois Tigeot 997352ff8bdSFrançois Tigeot /* Invalidate GuC TLB. */ 998352ff8bdSFrançois Tigeot if (i915.enable_guc_submission) 999352ff8bdSFrançois Tigeot I915_WRITE(GEN8_GTCR, GEN8_GTCR_INVALIDATE); 10002c9916cdSFrançois Tigeot 1001*1487f786SFrançois Tigeot return 0; 10022c9916cdSFrançois Tigeot 10038621f407SFrançois Tigeot unpin_map: 1004*1487f786SFrançois Tigeot i915_gem_object_unpin_map(ce->state); 10052c9916cdSFrançois Tigeot unpin_ctx_obj: 1006*1487f786SFrançois Tigeot i915_gem_object_ggtt_unpin(ce->state); 1007*1487f786SFrançois Tigeot err: 1008*1487f786SFrançois Tigeot ce->pin_count = 0; 1009352ff8bdSFrançois Tigeot return ret; 1010352ff8bdSFrançois Tigeot } 1011352ff8bdSFrançois Tigeot 1012*1487f786SFrançois Tigeot void intel_lr_context_unpin(struct i915_gem_context *ctx, 1013c0e85e96SFrançois Tigeot struct intel_engine_cs *engine) 1014352ff8bdSFrançois Tigeot { 1015*1487f786SFrançois Tigeot struct intel_context *ce = &ctx->engine[engine->id]; 1016352ff8bdSFrançois Tigeot 1017*1487f786SFrançois Tigeot lockdep_assert_held(&ctx->i915->dev->struct_mutex); 1018*1487f786SFrançois Tigeot GEM_BUG_ON(ce->pin_count == 0); 1019c0e85e96SFrançois Tigeot 1020*1487f786SFrançois Tigeot if (--ce->pin_count) 1021*1487f786SFrançois Tigeot return; 1022352ff8bdSFrançois Tigeot 1023*1487f786SFrançois Tigeot intel_unpin_ringbuffer_obj(ce->ringbuf); 10242c9916cdSFrançois Tigeot 1025*1487f786SFrançois Tigeot i915_gem_object_unpin_map(ce->state); 1026*1487f786SFrançois Tigeot i915_gem_object_ggtt_unpin(ce->state); 10272c9916cdSFrançois Tigeot 1028*1487f786SFrançois Tigeot ce->lrc_vma = NULL; 1029*1487f786SFrançois Tigeot ce->lrc_desc = 0; 1030*1487f786SFrançois Tigeot ce->lrc_reg_state = NULL; 1031c0e85e96SFrançois Tigeot 1032c0e85e96SFrançois Tigeot i915_gem_context_unreference(ctx); 10332c9916cdSFrançois Tigeot } 10342c9916cdSFrançois Tigeot 1035a05eeebfSFrançois Tigeot static int intel_logical_ring_workarounds_emit(struct drm_i915_gem_request *req) 10362c9916cdSFrançois Tigeot { 10372c9916cdSFrançois Tigeot int ret, i; 10388621f407SFrançois Tigeot struct intel_engine_cs *engine = req->engine; 1039a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 1040*1487f786SFrançois Tigeot struct i915_workarounds *w = &req->i915->workarounds; 10412c9916cdSFrançois Tigeot 1042c0e85e96SFrançois Tigeot if (w->count == 0) 10432c9916cdSFrançois Tigeot return 0; 10442c9916cdSFrançois Tigeot 10458621f407SFrançois Tigeot engine->gpu_caches_dirty = true; 1046a05eeebfSFrançois Tigeot ret = logical_ring_flush_all_caches(req); 10472c9916cdSFrançois Tigeot if (ret) 10482c9916cdSFrançois Tigeot return ret; 10492c9916cdSFrançois Tigeot 10508621f407SFrançois Tigeot ret = intel_ring_begin(req, w->count * 2 + 2); 10512c9916cdSFrançois Tigeot if (ret) 10522c9916cdSFrançois Tigeot return ret; 10532c9916cdSFrançois Tigeot 10542c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(w->count)); 10552c9916cdSFrançois Tigeot for (i = 0; i < w->count; i++) { 1056aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, w->reg[i].addr); 10572c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, w->reg[i].value); 10582c9916cdSFrançois Tigeot } 10592c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 10602c9916cdSFrançois Tigeot 10612c9916cdSFrançois Tigeot intel_logical_ring_advance(ringbuf); 10622c9916cdSFrançois Tigeot 10638621f407SFrançois Tigeot engine->gpu_caches_dirty = true; 1064a05eeebfSFrançois Tigeot ret = logical_ring_flush_all_caches(req); 10652c9916cdSFrançois Tigeot if (ret) 10662c9916cdSFrançois Tigeot return ret; 10672c9916cdSFrançois Tigeot 10682c9916cdSFrançois Tigeot return 0; 10692c9916cdSFrançois Tigeot } 10702c9916cdSFrançois Tigeot 1071a05eeebfSFrançois Tigeot #define wa_ctx_emit(batch, index, cmd) \ 1072a05eeebfSFrançois Tigeot do { \ 1073a05eeebfSFrançois Tigeot int __index = (index)++; \ 1074a05eeebfSFrançois Tigeot if (WARN_ON(__index >= (PAGE_SIZE / sizeof(uint32_t)))) { \ 1075a05eeebfSFrançois Tigeot return -ENOSPC; \ 1076a05eeebfSFrançois Tigeot } \ 1077a05eeebfSFrançois Tigeot batch[__index] = (cmd); \ 1078a05eeebfSFrançois Tigeot } while (0) 1079a05eeebfSFrançois Tigeot 1080aee94f86SFrançois Tigeot #define wa_ctx_emit_reg(batch, index, reg) \ 1081aee94f86SFrançois Tigeot wa_ctx_emit((batch), (index), i915_mmio_reg_offset(reg)) 1082a05eeebfSFrançois Tigeot 1083a05eeebfSFrançois Tigeot /* 1084a05eeebfSFrançois Tigeot * In this WA we need to set GEN8_L3SQCREG4[21:21] and reset it after 1085a05eeebfSFrançois Tigeot * PIPE_CONTROL instruction. This is required for the flush to happen correctly 1086a05eeebfSFrançois Tigeot * but there is a slight complication as this is applied in WA batch where the 1087a05eeebfSFrançois Tigeot * values are only initialized once so we cannot take register value at the 1088a05eeebfSFrançois Tigeot * beginning and reuse it further; hence we save its value to memory, upload a 1089a05eeebfSFrançois Tigeot * constant value with bit21 set and then we restore it back with the saved value. 1090a05eeebfSFrançois Tigeot * To simplify the WA, a constant value is formed by using the default value 1091a05eeebfSFrançois Tigeot * of this register. This shouldn't be a problem because we are only modifying 1092a05eeebfSFrançois Tigeot * it for a short period and this batch in non-premptible. We can ofcourse 1093a05eeebfSFrançois Tigeot * use additional instructions that read the actual value of the register 1094a05eeebfSFrançois Tigeot * at that time and set our bit of interest but it makes the WA complicated. 1095a05eeebfSFrançois Tigeot * 1096a05eeebfSFrançois Tigeot * This WA is also required for Gen9 so extracting as a function avoids 1097a05eeebfSFrançois Tigeot * code duplication. 1098a05eeebfSFrançois Tigeot */ 10998621f407SFrançois Tigeot static inline int gen8_emit_flush_coherentl3_wa(struct intel_engine_cs *engine, 1100a05eeebfSFrançois Tigeot uint32_t *const batch, 1101a05eeebfSFrançois Tigeot uint32_t index) 1102a05eeebfSFrançois Tigeot { 1103a05eeebfSFrançois Tigeot uint32_t l3sqc4_flush = (0x40400000 | GEN8_LQSC_FLUSH_COHERENT_LINES); 1104a05eeebfSFrançois Tigeot 1105a05eeebfSFrançois Tigeot /* 11068621f407SFrançois Tigeot * WaDisableLSQCROPERFforOCL:skl,kbl 1107a05eeebfSFrançois Tigeot * This WA is implemented in skl_init_clock_gating() but since 1108a05eeebfSFrançois Tigeot * this batch updates GEN8_L3SQCREG4 with default value we need to 1109a05eeebfSFrançois Tigeot * set this bit here to retain the WA during flush. 1110a05eeebfSFrançois Tigeot */ 1111*1487f786SFrançois Tigeot if (IS_SKL_REVID(engine->i915, 0, SKL_REVID_E0) || 1112*1487f786SFrançois Tigeot IS_KBL_REVID(engine->i915, 0, KBL_REVID_E0)) 1113a05eeebfSFrançois Tigeot l3sqc4_flush |= GEN8_LQSC_RO_PERF_DIS; 1114a05eeebfSFrançois Tigeot 1115352ff8bdSFrançois Tigeot wa_ctx_emit(batch, index, (MI_STORE_REGISTER_MEM_GEN8 | 1116a05eeebfSFrançois Tigeot MI_SRM_LRM_GLOBAL_GTT)); 1117aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 11188621f407SFrançois Tigeot wa_ctx_emit(batch, index, engine->scratch.gtt_offset + 256); 1119a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1120a05eeebfSFrançois Tigeot 1121a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_LOAD_REGISTER_IMM(1)); 1122aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 1123a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, l3sqc4_flush); 1124a05eeebfSFrançois Tigeot 1125a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, GFX_OP_PIPE_CONTROL(6)); 1126a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, (PIPE_CONTROL_CS_STALL | 1127a05eeebfSFrançois Tigeot PIPE_CONTROL_DC_FLUSH_ENABLE)); 1128a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1129a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1130a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1131a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1132a05eeebfSFrançois Tigeot 1133352ff8bdSFrançois Tigeot wa_ctx_emit(batch, index, (MI_LOAD_REGISTER_MEM_GEN8 | 1134a05eeebfSFrançois Tigeot MI_SRM_LRM_GLOBAL_GTT)); 1135aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 11368621f407SFrançois Tigeot wa_ctx_emit(batch, index, engine->scratch.gtt_offset + 256); 1137a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1138a05eeebfSFrançois Tigeot 1139a05eeebfSFrançois Tigeot return index; 1140a05eeebfSFrançois Tigeot } 1141a05eeebfSFrançois Tigeot 1142a05eeebfSFrançois Tigeot static inline uint32_t wa_ctx_start(struct i915_wa_ctx_bb *wa_ctx, 1143a05eeebfSFrançois Tigeot uint32_t offset, 1144a05eeebfSFrançois Tigeot uint32_t start_alignment) 1145a05eeebfSFrançois Tigeot { 1146a05eeebfSFrançois Tigeot return wa_ctx->offset = ALIGN(offset, start_alignment); 1147a05eeebfSFrançois Tigeot } 1148a05eeebfSFrançois Tigeot 1149a05eeebfSFrançois Tigeot static inline int wa_ctx_end(struct i915_wa_ctx_bb *wa_ctx, 1150a05eeebfSFrançois Tigeot uint32_t offset, 1151a05eeebfSFrançois Tigeot uint32_t size_alignment) 1152a05eeebfSFrançois Tigeot { 1153a05eeebfSFrançois Tigeot wa_ctx->size = offset - wa_ctx->offset; 1154a05eeebfSFrançois Tigeot 1155a05eeebfSFrançois Tigeot WARN(wa_ctx->size % size_alignment, 1156a05eeebfSFrançois Tigeot "wa_ctx_bb failed sanity checks: size %d is not aligned to %d\n", 1157a05eeebfSFrançois Tigeot wa_ctx->size, size_alignment); 1158a05eeebfSFrançois Tigeot return 0; 1159a05eeebfSFrançois Tigeot } 1160a05eeebfSFrançois Tigeot 1161a05eeebfSFrançois Tigeot /** 1162a05eeebfSFrançois Tigeot * gen8_init_indirectctx_bb() - initialize indirect ctx batch with WA 1163a05eeebfSFrançois Tigeot * 1164*1487f786SFrançois Tigeot * @engine: only applicable for RCS 1165a05eeebfSFrançois Tigeot * @wa_ctx: structure representing wa_ctx 1166a05eeebfSFrançois Tigeot * offset: specifies start of the batch, should be cache-aligned. This is updated 1167a05eeebfSFrançois Tigeot * with the offset value received as input. 1168a05eeebfSFrançois Tigeot * size: size of the batch in DWORDS but HW expects in terms of cachelines 1169a05eeebfSFrançois Tigeot * @batch: page in which WA are loaded 1170a05eeebfSFrançois Tigeot * @offset: This field specifies the start of the batch, it should be 1171a05eeebfSFrançois Tigeot * cache-aligned otherwise it is adjusted accordingly. 1172a05eeebfSFrançois Tigeot * Typically we only have one indirect_ctx and per_ctx batch buffer which are 1173a05eeebfSFrançois Tigeot * initialized at the beginning and shared across all contexts but this field 1174a05eeebfSFrançois Tigeot * helps us to have multiple batches at different offsets and select them based 1175a05eeebfSFrançois Tigeot * on a criteria. At the moment this batch always start at the beginning of the page 1176a05eeebfSFrançois Tigeot * and at this point we don't have multiple wa_ctx batch buffers. 1177a05eeebfSFrançois Tigeot * 1178a05eeebfSFrançois Tigeot * The number of WA applied are not known at the beginning; we use this field 1179a05eeebfSFrançois Tigeot * to return the no of DWORDS written. 1180a05eeebfSFrançois Tigeot * 1181a05eeebfSFrançois Tigeot * It is to be noted that this batch does not contain MI_BATCH_BUFFER_END 1182a05eeebfSFrançois Tigeot * so it adds NOOPs as padding to make it cacheline aligned. 1183a05eeebfSFrançois Tigeot * MI_BATCH_BUFFER_END will be added to perctx batch and both of them together 1184a05eeebfSFrançois Tigeot * makes a complete batch buffer. 1185a05eeebfSFrançois Tigeot * 1186a05eeebfSFrançois Tigeot * Return: non-zero if we exceed the PAGE_SIZE limit. 1187a05eeebfSFrançois Tigeot */ 1188a05eeebfSFrançois Tigeot 11898621f407SFrançois Tigeot static int gen8_init_indirectctx_bb(struct intel_engine_cs *engine, 1190a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1191a05eeebfSFrançois Tigeot uint32_t *const batch, 1192a05eeebfSFrançois Tigeot uint32_t *offset) 1193a05eeebfSFrançois Tigeot { 1194a05eeebfSFrançois Tigeot uint32_t scratch_addr; 1195a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1196a05eeebfSFrançois Tigeot 1197a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:bdw,chv */ 1198a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_DISABLE); 1199a05eeebfSFrançois Tigeot 1200a05eeebfSFrançois Tigeot /* WaFlushCoherentL3CacheLinesAtContextSwitch:bdw */ 1201*1487f786SFrançois Tigeot if (IS_BROADWELL(engine->i915)) { 12028621f407SFrançois Tigeot int rc = gen8_emit_flush_coherentl3_wa(engine, batch, index); 1203352ff8bdSFrançois Tigeot if (rc < 0) 1204352ff8bdSFrançois Tigeot return rc; 1205352ff8bdSFrançois Tigeot index = rc; 1206a05eeebfSFrançois Tigeot } 1207a05eeebfSFrançois Tigeot 1208a05eeebfSFrançois Tigeot /* WaClearSlmSpaceAtContextSwitch:bdw,chv */ 1209a05eeebfSFrançois Tigeot /* Actual scratch location is at 128 bytes offset */ 12108621f407SFrançois Tigeot scratch_addr = engine->scratch.gtt_offset + 2*CACHELINE_BYTES; 1211a05eeebfSFrançois Tigeot 1212a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, GFX_OP_PIPE_CONTROL(6)); 1213a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, (PIPE_CONTROL_FLUSH_L3 | 1214a05eeebfSFrançois Tigeot PIPE_CONTROL_GLOBAL_GTT_IVB | 1215a05eeebfSFrançois Tigeot PIPE_CONTROL_CS_STALL | 1216a05eeebfSFrançois Tigeot PIPE_CONTROL_QW_WRITE)); 1217a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, scratch_addr); 1218a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1219a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1220a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1221a05eeebfSFrançois Tigeot 1222a05eeebfSFrançois Tigeot /* Pad to end of cacheline */ 1223a05eeebfSFrançois Tigeot while (index % CACHELINE_DWORDS) 1224a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1225a05eeebfSFrançois Tigeot 1226a05eeebfSFrançois Tigeot /* 1227a05eeebfSFrançois Tigeot * MI_BATCH_BUFFER_END is not required in Indirect ctx BB because 1228a05eeebfSFrançois Tigeot * execution depends on the length specified in terms of cache lines 1229a05eeebfSFrançois Tigeot * in the register CTX_RCS_INDIRECT_CTX 1230a05eeebfSFrançois Tigeot */ 1231a05eeebfSFrançois Tigeot 1232a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, CACHELINE_DWORDS); 1233a05eeebfSFrançois Tigeot } 1234a05eeebfSFrançois Tigeot 1235a05eeebfSFrançois Tigeot /** 1236a05eeebfSFrançois Tigeot * gen8_init_perctx_bb() - initialize per ctx batch with WA 1237a05eeebfSFrançois Tigeot * 1238*1487f786SFrançois Tigeot * @engine: only applicable for RCS 1239a05eeebfSFrançois Tigeot * @wa_ctx: structure representing wa_ctx 1240a05eeebfSFrançois Tigeot * offset: specifies start of the batch, should be cache-aligned. 1241a05eeebfSFrançois Tigeot * size: size of the batch in DWORDS but HW expects in terms of cachelines 1242a05eeebfSFrançois Tigeot * @batch: page in which WA are loaded 1243a05eeebfSFrançois Tigeot * @offset: This field specifies the start of this batch. 1244a05eeebfSFrançois Tigeot * This batch is started immediately after indirect_ctx batch. Since we ensure 1245a05eeebfSFrançois Tigeot * that indirect_ctx ends on a cacheline this batch is aligned automatically. 1246a05eeebfSFrançois Tigeot * 1247a05eeebfSFrançois Tigeot * The number of DWORDS written are returned using this field. 1248a05eeebfSFrançois Tigeot * 1249a05eeebfSFrançois Tigeot * This batch is terminated with MI_BATCH_BUFFER_END and so we need not add padding 1250a05eeebfSFrançois Tigeot * to align it with cacheline as padding after MI_BATCH_BUFFER_END is redundant. 1251a05eeebfSFrançois Tigeot */ 12528621f407SFrançois Tigeot static int gen8_init_perctx_bb(struct intel_engine_cs *engine, 1253a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1254a05eeebfSFrançois Tigeot uint32_t *const batch, 1255a05eeebfSFrançois Tigeot uint32_t *offset) 1256a05eeebfSFrançois Tigeot { 1257a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1258a05eeebfSFrançois Tigeot 1259a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:bdw,chv */ 1260a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_ENABLE); 1261a05eeebfSFrançois Tigeot 1262a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_BATCH_BUFFER_END); 1263a05eeebfSFrançois Tigeot 1264a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, 1); 1265a05eeebfSFrançois Tigeot } 1266a05eeebfSFrançois Tigeot 12678621f407SFrançois Tigeot static int gen9_init_indirectctx_bb(struct intel_engine_cs *engine, 1268a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1269a05eeebfSFrançois Tigeot uint32_t *const batch, 1270a05eeebfSFrançois Tigeot uint32_t *offset) 1271a05eeebfSFrançois Tigeot { 1272a05eeebfSFrançois Tigeot int ret; 1273a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1274a05eeebfSFrançois Tigeot 1275a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:skl,bxt */ 1276*1487f786SFrançois Tigeot if (IS_SKL_REVID(engine->i915, 0, SKL_REVID_D0) || 1277*1487f786SFrançois Tigeot IS_BXT_REVID(engine->i915, 0, BXT_REVID_A1)) 1278a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_DISABLE); 1279a05eeebfSFrançois Tigeot 1280a05eeebfSFrançois Tigeot /* WaFlushCoherentL3CacheLinesAtContextSwitch:skl,bxt */ 12818621f407SFrançois Tigeot ret = gen8_emit_flush_coherentl3_wa(engine, batch, index); 1282a05eeebfSFrançois Tigeot if (ret < 0) 1283a05eeebfSFrançois Tigeot return ret; 1284a05eeebfSFrançois Tigeot index = ret; 1285a05eeebfSFrançois Tigeot 12868621f407SFrançois Tigeot /* WaClearSlmSpaceAtContextSwitch:kbl */ 12878621f407SFrançois Tigeot /* Actual scratch location is at 128 bytes offset */ 1288*1487f786SFrançois Tigeot if (IS_KBL_REVID(engine->i915, 0, KBL_REVID_A0)) { 12898621f407SFrançois Tigeot uint32_t scratch_addr 12908621f407SFrançois Tigeot = engine->scratch.gtt_offset + 2*CACHELINE_BYTES; 12918621f407SFrançois Tigeot 12928621f407SFrançois Tigeot wa_ctx_emit(batch, index, GFX_OP_PIPE_CONTROL(6)); 12938621f407SFrançois Tigeot wa_ctx_emit(batch, index, (PIPE_CONTROL_FLUSH_L3 | 12948621f407SFrançois Tigeot PIPE_CONTROL_GLOBAL_GTT_IVB | 12958621f407SFrançois Tigeot PIPE_CONTROL_CS_STALL | 12968621f407SFrançois Tigeot PIPE_CONTROL_QW_WRITE)); 12978621f407SFrançois Tigeot wa_ctx_emit(batch, index, scratch_addr); 12988621f407SFrançois Tigeot wa_ctx_emit(batch, index, 0); 12998621f407SFrançois Tigeot wa_ctx_emit(batch, index, 0); 13008621f407SFrançois Tigeot wa_ctx_emit(batch, index, 0); 13018621f407SFrançois Tigeot } 1302a05eeebfSFrançois Tigeot /* Pad to end of cacheline */ 1303a05eeebfSFrançois Tigeot while (index % CACHELINE_DWORDS) 1304a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1305a05eeebfSFrançois Tigeot 1306a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, CACHELINE_DWORDS); 1307a05eeebfSFrançois Tigeot } 1308a05eeebfSFrançois Tigeot 13098621f407SFrançois Tigeot static int gen9_init_perctx_bb(struct intel_engine_cs *engine, 1310a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1311a05eeebfSFrançois Tigeot uint32_t *const batch, 1312a05eeebfSFrançois Tigeot uint32_t *offset) 1313a05eeebfSFrançois Tigeot { 1314a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1315a05eeebfSFrançois Tigeot 1316a05eeebfSFrançois Tigeot /* WaSetDisablePixMaskCammingAndRhwoInCommonSliceChicken:skl,bxt */ 1317*1487f786SFrançois Tigeot if (IS_SKL_REVID(engine->i915, 0, SKL_REVID_B0) || 1318*1487f786SFrançois Tigeot IS_BXT_REVID(engine->i915, 0, BXT_REVID_A1)) { 1319a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_LOAD_REGISTER_IMM(1)); 1320aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN9_SLICE_COMMON_ECO_CHICKEN0); 1321a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 1322a05eeebfSFrançois Tigeot _MASKED_BIT_ENABLE(DISABLE_PIXEL_MASK_CAMMING)); 1323a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1324a05eeebfSFrançois Tigeot } 1325a05eeebfSFrançois Tigeot 13268621f407SFrançois Tigeot /* WaClearTdlStateAckDirtyBits:bxt */ 1327*1487f786SFrançois Tigeot if (IS_BXT_REVID(engine->i915, 0, BXT_REVID_B0)) { 13288621f407SFrançois Tigeot wa_ctx_emit(batch, index, MI_LOAD_REGISTER_IMM(4)); 13298621f407SFrançois Tigeot 13308621f407SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_STATE_ACK); 13318621f407SFrançois Tigeot wa_ctx_emit(batch, index, _MASKED_BIT_DISABLE(GEN9_SUBSLICE_TDL_ACK_BITS)); 13328621f407SFrançois Tigeot 13338621f407SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN9_STATE_ACK_SLICE1); 13348621f407SFrançois Tigeot wa_ctx_emit(batch, index, _MASKED_BIT_DISABLE(GEN9_SUBSLICE_TDL_ACK_BITS)); 13358621f407SFrançois Tigeot 13368621f407SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN9_STATE_ACK_SLICE2); 13378621f407SFrançois Tigeot wa_ctx_emit(batch, index, _MASKED_BIT_DISABLE(GEN9_SUBSLICE_TDL_ACK_BITS)); 13388621f407SFrançois Tigeot 13398621f407SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN7_ROW_CHICKEN2); 13408621f407SFrançois Tigeot /* dummy write to CS, mask bits are 0 to ensure the register is not modified */ 13418621f407SFrançois Tigeot wa_ctx_emit(batch, index, 0x0); 13428621f407SFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 13438621f407SFrançois Tigeot } 13448621f407SFrançois Tigeot 1345a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:skl,bxt */ 1346*1487f786SFrançois Tigeot if (IS_SKL_REVID(engine->i915, 0, SKL_REVID_D0) || 1347*1487f786SFrançois Tigeot IS_BXT_REVID(engine->i915, 0, BXT_REVID_A1)) 1348a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_ENABLE); 1349a05eeebfSFrançois Tigeot 1350a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_BATCH_BUFFER_END); 1351a05eeebfSFrançois Tigeot 1352a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, 1); 1353a05eeebfSFrançois Tigeot } 1354a05eeebfSFrançois Tigeot 13558621f407SFrançois Tigeot static int lrc_setup_wa_ctx_obj(struct intel_engine_cs *engine, u32 size) 1356a05eeebfSFrançois Tigeot { 1357a05eeebfSFrançois Tigeot int ret; 1358a05eeebfSFrançois Tigeot 1359*1487f786SFrançois Tigeot engine->wa_ctx.obj = i915_gem_object_create(engine->i915->dev, 13608621f407SFrançois Tigeot PAGE_ALIGN(size)); 1361*1487f786SFrançois Tigeot if (IS_ERR(engine->wa_ctx.obj)) { 1362a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("alloc LRC WA ctx backing obj failed.\n"); 1363*1487f786SFrançois Tigeot ret = PTR_ERR(engine->wa_ctx.obj); 1364*1487f786SFrançois Tigeot engine->wa_ctx.obj = NULL; 1365*1487f786SFrançois Tigeot return ret; 1366a05eeebfSFrançois Tigeot } 1367a05eeebfSFrançois Tigeot 13688621f407SFrançois Tigeot ret = i915_gem_obj_ggtt_pin(engine->wa_ctx.obj, PAGE_SIZE, 0); 1369a05eeebfSFrançois Tigeot if (ret) { 1370a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("pin LRC WA ctx backing obj failed: %d\n", 1371a05eeebfSFrançois Tigeot ret); 13728621f407SFrançois Tigeot drm_gem_object_unreference(&engine->wa_ctx.obj->base); 1373a05eeebfSFrançois Tigeot return ret; 1374a05eeebfSFrançois Tigeot } 1375a05eeebfSFrançois Tigeot 1376a05eeebfSFrançois Tigeot return 0; 1377a05eeebfSFrançois Tigeot } 1378a05eeebfSFrançois Tigeot 13798621f407SFrançois Tigeot static void lrc_destroy_wa_ctx_obj(struct intel_engine_cs *engine) 1380a05eeebfSFrançois Tigeot { 13818621f407SFrançois Tigeot if (engine->wa_ctx.obj) { 13828621f407SFrançois Tigeot i915_gem_object_ggtt_unpin(engine->wa_ctx.obj); 13838621f407SFrançois Tigeot drm_gem_object_unreference(&engine->wa_ctx.obj->base); 13848621f407SFrançois Tigeot engine->wa_ctx.obj = NULL; 1385a05eeebfSFrançois Tigeot } 1386a05eeebfSFrançois Tigeot } 1387a05eeebfSFrançois Tigeot 13888621f407SFrançois Tigeot static int intel_init_workaround_bb(struct intel_engine_cs *engine) 1389a05eeebfSFrançois Tigeot { 1390a05eeebfSFrançois Tigeot int ret; 1391a05eeebfSFrançois Tigeot uint32_t *batch; 1392a05eeebfSFrançois Tigeot uint32_t offset; 1393f0bba3d1SFrançois Tigeot struct page *page; 13948621f407SFrançois Tigeot struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx; 1395a05eeebfSFrançois Tigeot 13968621f407SFrançois Tigeot WARN_ON(engine->id != RCS); 1397a05eeebfSFrançois Tigeot 1398a05eeebfSFrançois Tigeot /* update this when WA for higher Gen are added */ 1399*1487f786SFrançois Tigeot if (INTEL_GEN(engine->i915) > 9) { 1400a05eeebfSFrançois Tigeot DRM_ERROR("WA batch buffer is not initialized for Gen%d\n", 1401*1487f786SFrançois Tigeot INTEL_GEN(engine->i915)); 1402a05eeebfSFrançois Tigeot return 0; 1403a05eeebfSFrançois Tigeot } 1404a05eeebfSFrançois Tigeot 1405a05eeebfSFrançois Tigeot /* some WA perform writes to scratch page, ensure it is valid */ 14068621f407SFrançois Tigeot if (engine->scratch.obj == NULL) { 14078621f407SFrançois Tigeot DRM_ERROR("scratch page not allocated for %s\n", engine->name); 1408a05eeebfSFrançois Tigeot return -EINVAL; 1409a05eeebfSFrançois Tigeot } 1410a05eeebfSFrançois Tigeot 14118621f407SFrançois Tigeot ret = lrc_setup_wa_ctx_obj(engine, PAGE_SIZE); 1412a05eeebfSFrançois Tigeot if (ret) { 1413a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("Failed to setup context WA page: %d\n", ret); 1414a05eeebfSFrançois Tigeot return ret; 1415a05eeebfSFrançois Tigeot } 1416a05eeebfSFrançois Tigeot 1417aee94f86SFrançois Tigeot page = i915_gem_object_get_dirty_page(wa_ctx->obj, 0); 1418a05eeebfSFrançois Tigeot batch = kmap_atomic(page); 1419a05eeebfSFrançois Tigeot offset = 0; 1420a05eeebfSFrançois Tigeot 1421*1487f786SFrançois Tigeot if (IS_GEN8(engine->i915)) { 14228621f407SFrançois Tigeot ret = gen8_init_indirectctx_bb(engine, 1423a05eeebfSFrançois Tigeot &wa_ctx->indirect_ctx, 1424a05eeebfSFrançois Tigeot batch, 1425a05eeebfSFrançois Tigeot &offset); 1426a05eeebfSFrançois Tigeot if (ret) 1427a05eeebfSFrançois Tigeot goto out; 1428a05eeebfSFrançois Tigeot 14298621f407SFrançois Tigeot ret = gen8_init_perctx_bb(engine, 1430a05eeebfSFrançois Tigeot &wa_ctx->per_ctx, 1431a05eeebfSFrançois Tigeot batch, 1432a05eeebfSFrançois Tigeot &offset); 1433a05eeebfSFrançois Tigeot if (ret) 1434a05eeebfSFrançois Tigeot goto out; 1435*1487f786SFrançois Tigeot } else if (IS_GEN9(engine->i915)) { 14368621f407SFrançois Tigeot ret = gen9_init_indirectctx_bb(engine, 1437a05eeebfSFrançois Tigeot &wa_ctx->indirect_ctx, 1438a05eeebfSFrançois Tigeot batch, 1439a05eeebfSFrançois Tigeot &offset); 1440a05eeebfSFrançois Tigeot if (ret) 1441a05eeebfSFrançois Tigeot goto out; 1442a05eeebfSFrançois Tigeot 14438621f407SFrançois Tigeot ret = gen9_init_perctx_bb(engine, 1444a05eeebfSFrançois Tigeot &wa_ctx->per_ctx, 1445a05eeebfSFrançois Tigeot batch, 1446a05eeebfSFrançois Tigeot &offset); 1447a05eeebfSFrançois Tigeot if (ret) 1448a05eeebfSFrançois Tigeot goto out; 1449a05eeebfSFrançois Tigeot } 1450a05eeebfSFrançois Tigeot 1451a05eeebfSFrançois Tigeot out: 1452a05eeebfSFrançois Tigeot kunmap_atomic(batch); 1453a05eeebfSFrançois Tigeot if (ret) 14548621f407SFrançois Tigeot lrc_destroy_wa_ctx_obj(engine); 1455a05eeebfSFrançois Tigeot 1456a05eeebfSFrançois Tigeot return ret; 1457a05eeebfSFrançois Tigeot } 1458a05eeebfSFrançois Tigeot 14598621f407SFrançois Tigeot static void lrc_init_hws(struct intel_engine_cs *engine) 14601b13d190SFrançois Tigeot { 1461*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 14628621f407SFrançois Tigeot 14638621f407SFrançois Tigeot I915_WRITE(RING_HWS_PGA(engine->mmio_base), 14648621f407SFrançois Tigeot (u32)engine->status_page.gfx_addr); 14658621f407SFrançois Tigeot POSTING_READ(RING_HWS_PGA(engine->mmio_base)); 14668621f407SFrançois Tigeot } 14678621f407SFrançois Tigeot 14688621f407SFrançois Tigeot static int gen8_init_common_ring(struct intel_engine_cs *engine) 14698621f407SFrançois Tigeot { 1470*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 14718621f407SFrançois Tigeot unsigned int next_context_status_buffer_hw; 14721b13d190SFrançois Tigeot 14738621f407SFrançois Tigeot lrc_init_hws(engine); 1474352ff8bdSFrançois Tigeot 14758621f407SFrançois Tigeot I915_WRITE_IMR(engine, 14768621f407SFrançois Tigeot ~(engine->irq_enable_mask | engine->irq_keep_mask)); 14778621f407SFrançois Tigeot I915_WRITE(RING_HWSTAM(engine->mmio_base), 0xffffffff); 14781b13d190SFrançois Tigeot 14798621f407SFrançois Tigeot I915_WRITE(RING_MODE_GEN7(engine), 14801b13d190SFrançois Tigeot _MASKED_BIT_DISABLE(GFX_REPLAY_MODE) | 14811b13d190SFrançois Tigeot _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE)); 14828621f407SFrançois Tigeot POSTING_READ(RING_MODE_GEN7(engine)); 1483a05eeebfSFrançois Tigeot 1484a05eeebfSFrançois Tigeot /* 1485a05eeebfSFrançois Tigeot * Instead of resetting the Context Status Buffer (CSB) read pointer to 1486a05eeebfSFrançois Tigeot * zero, we need to read the write pointer from hardware and use its 1487a05eeebfSFrançois Tigeot * value because "this register is power context save restored". 1488a05eeebfSFrançois Tigeot * Effectively, these states have been observed: 1489a05eeebfSFrançois Tigeot * 1490a05eeebfSFrançois Tigeot * | Suspend-to-idle (freeze) | Suspend-to-RAM (mem) | 1491a05eeebfSFrançois Tigeot * BDW | CSB regs not reset | CSB regs reset | 1492a05eeebfSFrançois Tigeot * CHT | CSB regs not reset | CSB regs not reset | 1493c0e85e96SFrançois Tigeot * SKL | ? | ? | 1494c0e85e96SFrançois Tigeot * BXT | ? | ? | 1495a05eeebfSFrançois Tigeot */ 1496c0e85e96SFrançois Tigeot next_context_status_buffer_hw = 14978621f407SFrançois Tigeot GEN8_CSB_WRITE_PTR(I915_READ(RING_CONTEXT_STATUS_PTR(engine))); 1498a05eeebfSFrançois Tigeot 1499a05eeebfSFrançois Tigeot /* 1500a05eeebfSFrançois Tigeot * When the CSB registers are reset (also after power-up / gpu reset), 1501a05eeebfSFrançois Tigeot * CSB write pointer is set to all 1's, which is not valid, use '5' in 1502a05eeebfSFrançois Tigeot * this special case, so the first element read is CSB[0]. 1503a05eeebfSFrançois Tigeot */ 1504a05eeebfSFrançois Tigeot if (next_context_status_buffer_hw == GEN8_CSB_PTR_MASK) 1505a05eeebfSFrançois Tigeot next_context_status_buffer_hw = (GEN8_CSB_ENTRIES - 1); 1506a05eeebfSFrançois Tigeot 15078621f407SFrançois Tigeot engine->next_context_status_buffer = next_context_status_buffer_hw; 15088621f407SFrançois Tigeot DRM_DEBUG_DRIVER("Execlists enabled for %s\n", engine->name); 15091b13d190SFrançois Tigeot 15108621f407SFrançois Tigeot intel_engine_init_hangcheck(engine); 15111b13d190SFrançois Tigeot 15128621f407SFrançois Tigeot return intel_mocs_init_engine(engine); 15131b13d190SFrançois Tigeot } 15141b13d190SFrançois Tigeot 15158621f407SFrançois Tigeot static int gen8_init_render_ring(struct intel_engine_cs *engine) 15161b13d190SFrançois Tigeot { 1517*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 15181b13d190SFrançois Tigeot int ret; 15191b13d190SFrançois Tigeot 15208621f407SFrançois Tigeot ret = gen8_init_common_ring(engine); 15211b13d190SFrançois Tigeot if (ret) 15221b13d190SFrançois Tigeot return ret; 15231b13d190SFrançois Tigeot 15241b13d190SFrançois Tigeot /* We need to disable the AsyncFlip performance optimisations in order 15251b13d190SFrançois Tigeot * to use MI_WAIT_FOR_EVENT within the CS. It should already be 15261b13d190SFrançois Tigeot * programmed to '1' on all products. 15271b13d190SFrançois Tigeot * 15281b13d190SFrançois Tigeot * WaDisableAsyncFlipPerfMode:snb,ivb,hsw,vlv,bdw,chv 15291b13d190SFrançois Tigeot */ 15301b13d190SFrançois Tigeot I915_WRITE(MI_MODE, _MASKED_BIT_ENABLE(ASYNC_FLIP_PERF_DISABLE)); 15311b13d190SFrançois Tigeot 15321b13d190SFrançois Tigeot I915_WRITE(INSTPM, _MASKED_BIT_ENABLE(INSTPM_FORCE_ORDERING)); 15331b13d190SFrançois Tigeot 15348621f407SFrançois Tigeot return init_workarounds_ring(engine); 15351b13d190SFrançois Tigeot } 15361b13d190SFrançois Tigeot 15378621f407SFrançois Tigeot static int gen9_init_render_ring(struct intel_engine_cs *engine) 1538477eb7f9SFrançois Tigeot { 1539477eb7f9SFrançois Tigeot int ret; 1540477eb7f9SFrançois Tigeot 15418621f407SFrançois Tigeot ret = gen8_init_common_ring(engine); 1542477eb7f9SFrançois Tigeot if (ret) 1543477eb7f9SFrançois Tigeot return ret; 1544477eb7f9SFrançois Tigeot 15458621f407SFrançois Tigeot return init_workarounds_ring(engine); 1546477eb7f9SFrançois Tigeot } 1547477eb7f9SFrançois Tigeot 1548a05eeebfSFrançois Tigeot static int intel_logical_ring_emit_pdps(struct drm_i915_gem_request *req) 1549a05eeebfSFrançois Tigeot { 1550a05eeebfSFrançois Tigeot struct i915_hw_ppgtt *ppgtt = req->ctx->ppgtt; 15518621f407SFrançois Tigeot struct intel_engine_cs *engine = req->engine; 1552a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 1553a05eeebfSFrançois Tigeot const int num_lri_cmds = GEN8_LEGACY_PDPES * 2; 1554a05eeebfSFrançois Tigeot int i, ret; 1555a05eeebfSFrançois Tigeot 15568621f407SFrançois Tigeot ret = intel_ring_begin(req, num_lri_cmds * 2 + 2); 1557a05eeebfSFrançois Tigeot if (ret) 1558a05eeebfSFrançois Tigeot return ret; 1559a05eeebfSFrançois Tigeot 1560a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(num_lri_cmds)); 1561a05eeebfSFrançois Tigeot for (i = GEN8_LEGACY_PDPES - 1; i >= 0; i--) { 1562a05eeebfSFrançois Tigeot const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i); 1563a05eeebfSFrançois Tigeot 15648621f407SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, 15658621f407SFrançois Tigeot GEN8_RING_PDP_UDW(engine, i)); 1566a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, upper_32_bits(pd_daddr)); 15678621f407SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, 15688621f407SFrançois Tigeot GEN8_RING_PDP_LDW(engine, i)); 1569a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, lower_32_bits(pd_daddr)); 1570a05eeebfSFrançois Tigeot } 1571a05eeebfSFrançois Tigeot 1572a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1573a05eeebfSFrançois Tigeot intel_logical_ring_advance(ringbuf); 1574a05eeebfSFrançois Tigeot 1575a05eeebfSFrançois Tigeot return 0; 1576a05eeebfSFrançois Tigeot } 1577a05eeebfSFrançois Tigeot 1578a05eeebfSFrançois Tigeot static int gen8_emit_bb_start(struct drm_i915_gem_request *req, 1579477eb7f9SFrançois Tigeot u64 offset, unsigned dispatch_flags) 15801b13d190SFrançois Tigeot { 1581a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 1582477eb7f9SFrançois Tigeot bool ppgtt = !(dispatch_flags & I915_DISPATCH_SECURE); 15831b13d190SFrançois Tigeot int ret; 15841b13d190SFrançois Tigeot 1585a05eeebfSFrançois Tigeot /* Don't rely in hw updating PDPs, specially in lite-restore. 1586a05eeebfSFrançois Tigeot * Ideally, we should set Force PD Restore in ctx descriptor, 1587a05eeebfSFrançois Tigeot * but we can't. Force Restore would be a second option, but 1588a05eeebfSFrançois Tigeot * it is unsafe in case of lite-restore (because the ctx is 1589352ff8bdSFrançois Tigeot * not idle). PML4 is allocated during ppgtt init so this is 1590352ff8bdSFrançois Tigeot * not needed in 48-bit.*/ 1591a05eeebfSFrançois Tigeot if (req->ctx->ppgtt && 15928621f407SFrançois Tigeot (intel_engine_flag(req->engine) & req->ctx->ppgtt->pd_dirty_rings)) { 1593352ff8bdSFrançois Tigeot if (!USES_FULL_48BIT_PPGTT(req->i915) && 1594*1487f786SFrançois Tigeot !intel_vgpu_active(req->i915)) { 1595a05eeebfSFrançois Tigeot ret = intel_logical_ring_emit_pdps(req); 1596a05eeebfSFrançois Tigeot if (ret) 1597a05eeebfSFrançois Tigeot return ret; 1598352ff8bdSFrançois Tigeot } 1599a05eeebfSFrançois Tigeot 16008621f407SFrançois Tigeot req->ctx->ppgtt->pd_dirty_rings &= ~intel_engine_flag(req->engine); 1601a05eeebfSFrançois Tigeot } 1602a05eeebfSFrançois Tigeot 16038621f407SFrançois Tigeot ret = intel_ring_begin(req, 4); 16041b13d190SFrançois Tigeot if (ret) 16051b13d190SFrançois Tigeot return ret; 16061b13d190SFrançois Tigeot 16071b13d190SFrançois Tigeot /* FIXME(BDW): Address space and security selectors. */ 1608a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_BATCH_BUFFER_START_GEN8 | 1609a05eeebfSFrançois Tigeot (ppgtt<<8) | 1610a05eeebfSFrançois Tigeot (dispatch_flags & I915_DISPATCH_RS ? 1611a05eeebfSFrançois Tigeot MI_BATCH_RESOURCE_STREAMER : 0)); 16121b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, lower_32_bits(offset)); 16131b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, upper_32_bits(offset)); 16141b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 16151b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 16161b13d190SFrançois Tigeot 16171b13d190SFrançois Tigeot return 0; 16181b13d190SFrançois Tigeot } 16191b13d190SFrançois Tigeot 16208621f407SFrançois Tigeot static bool gen8_logical_ring_get_irq(struct intel_engine_cs *engine) 16211b13d190SFrançois Tigeot { 1622*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 16235e269720SFrançois Tigeot unsigned long flags; 16241b13d190SFrançois Tigeot 16252c9916cdSFrançois Tigeot if (WARN_ON(!intel_irqs_enabled(dev_priv))) 16261b13d190SFrançois Tigeot return false; 16271b13d190SFrançois Tigeot 16285e269720SFrançois Tigeot spin_lock_irqsave(&dev_priv->irq_lock, flags); 16298621f407SFrançois Tigeot if (engine->irq_refcount++ == 0) { 16308621f407SFrançois Tigeot I915_WRITE_IMR(engine, 16318621f407SFrançois Tigeot ~(engine->irq_enable_mask | engine->irq_keep_mask)); 16328621f407SFrançois Tigeot POSTING_READ(RING_IMR(engine->mmio_base)); 16331b13d190SFrançois Tigeot } 16345e269720SFrançois Tigeot spin_unlock_irqrestore(&dev_priv->irq_lock, flags); 16351b13d190SFrançois Tigeot 16361b13d190SFrançois Tigeot return true; 16371b13d190SFrançois Tigeot } 16381b13d190SFrançois Tigeot 16398621f407SFrançois Tigeot static void gen8_logical_ring_put_irq(struct intel_engine_cs *engine) 16401b13d190SFrançois Tigeot { 1641*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 16425e269720SFrançois Tigeot unsigned long flags; 16431b13d190SFrançois Tigeot 16445e269720SFrançois Tigeot spin_lock_irqsave(&dev_priv->irq_lock, flags); 16458621f407SFrançois Tigeot if (--engine->irq_refcount == 0) { 16468621f407SFrançois Tigeot I915_WRITE_IMR(engine, ~engine->irq_keep_mask); 16478621f407SFrançois Tigeot POSTING_READ(RING_IMR(engine->mmio_base)); 16481b13d190SFrançois Tigeot } 16495e269720SFrançois Tigeot spin_unlock_irqrestore(&dev_priv->irq_lock, flags); 16501b13d190SFrançois Tigeot } 16511b13d190SFrançois Tigeot 1652a05eeebfSFrançois Tigeot static int gen8_emit_flush(struct drm_i915_gem_request *request, 16531b13d190SFrançois Tigeot u32 invalidate_domains, 16541b13d190SFrançois Tigeot u32 unused) 16551b13d190SFrançois Tigeot { 1656a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 16578621f407SFrançois Tigeot struct intel_engine_cs *engine = ringbuf->engine; 1658*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = request->i915; 16591b13d190SFrançois Tigeot uint32_t cmd; 16601b13d190SFrançois Tigeot int ret; 16611b13d190SFrançois Tigeot 16628621f407SFrançois Tigeot ret = intel_ring_begin(request, 4); 16631b13d190SFrançois Tigeot if (ret) 16641b13d190SFrançois Tigeot return ret; 16651b13d190SFrançois Tigeot 16661b13d190SFrançois Tigeot cmd = MI_FLUSH_DW + 1; 16671b13d190SFrançois Tigeot 16682c9916cdSFrançois Tigeot /* We always require a command barrier so that subsequent 16692c9916cdSFrançois Tigeot * commands, such as breadcrumb interrupts, are strictly ordered 16702c9916cdSFrançois Tigeot * wrt the contents of the write cache being flushed to memory 16712c9916cdSFrançois Tigeot * (and thus being coherent from the CPU). 16722c9916cdSFrançois Tigeot */ 16732c9916cdSFrançois Tigeot cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW; 16742c9916cdSFrançois Tigeot 16752c9916cdSFrançois Tigeot if (invalidate_domains & I915_GEM_GPU_DOMAINS) { 16762c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_TLB; 16778621f407SFrançois Tigeot if (engine == &dev_priv->engine[VCS]) 16782c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_BSD; 16791b13d190SFrançois Tigeot } 16801b13d190SFrançois Tigeot 16811b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, cmd); 16821b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 16831b13d190SFrançois Tigeot I915_GEM_HWS_SCRATCH_ADDR | 16841b13d190SFrançois Tigeot MI_FLUSH_DW_USE_GTT); 16851b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* upper addr */ 16861b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* value */ 16871b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 16881b13d190SFrançois Tigeot 16891b13d190SFrançois Tigeot return 0; 16901b13d190SFrançois Tigeot } 16911b13d190SFrançois Tigeot 1692a05eeebfSFrançois Tigeot static int gen8_emit_flush_render(struct drm_i915_gem_request *request, 16931b13d190SFrançois Tigeot u32 invalidate_domains, 16941b13d190SFrançois Tigeot u32 flush_domains) 16951b13d190SFrançois Tigeot { 1696a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 16978621f407SFrançois Tigeot struct intel_engine_cs *engine = ringbuf->engine; 16988621f407SFrançois Tigeot u32 scratch_addr = engine->scratch.gtt_offset + 2 * CACHELINE_BYTES; 16998621f407SFrançois Tigeot bool vf_flush_wa = false, dc_flush_wa = false; 17001b13d190SFrançois Tigeot u32 flags = 0; 17011b13d190SFrançois Tigeot int ret; 17028621f407SFrançois Tigeot int len; 17031b13d190SFrançois Tigeot 17041b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CS_STALL; 17051b13d190SFrançois Tigeot 17061b13d190SFrançois Tigeot if (flush_domains) { 17071b13d190SFrançois Tigeot flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH; 17081b13d190SFrançois Tigeot flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH; 1709aee94f86SFrançois Tigeot flags |= PIPE_CONTROL_DC_FLUSH_ENABLE; 1710b49c8cf9SFrançois Tigeot flags |= PIPE_CONTROL_FLUSH_ENABLE; 17111b13d190SFrançois Tigeot } 17121b13d190SFrançois Tigeot 17131b13d190SFrançois Tigeot if (invalidate_domains) { 17141b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TLB_INVALIDATE; 17151b13d190SFrançois Tigeot flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE; 17161b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE; 17171b13d190SFrançois Tigeot flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE; 17181b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE; 17191b13d190SFrançois Tigeot flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE; 17201b13d190SFrançois Tigeot flags |= PIPE_CONTROL_QW_WRITE; 17211b13d190SFrançois Tigeot flags |= PIPE_CONTROL_GLOBAL_GTT_IVB; 17221b13d190SFrançois Tigeot 172319c468b4SFrançois Tigeot /* 1724c0e85e96SFrançois Tigeot * On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL 1725c0e85e96SFrançois Tigeot * pipe control. 172619c468b4SFrançois Tigeot */ 1727*1487f786SFrançois Tigeot if (IS_GEN9(request->i915)) 1728c0e85e96SFrançois Tigeot vf_flush_wa = true; 17298621f407SFrançois Tigeot 17308621f407SFrançois Tigeot /* WaForGAMHang:kbl */ 17318621f407SFrançois Tigeot if (IS_KBL_REVID(request->i915, 0, KBL_REVID_B0)) 17328621f407SFrançois Tigeot dc_flush_wa = true; 1733c0e85e96SFrançois Tigeot } 173419c468b4SFrançois Tigeot 17358621f407SFrançois Tigeot len = 6; 17368621f407SFrançois Tigeot 17378621f407SFrançois Tigeot if (vf_flush_wa) 17388621f407SFrançois Tigeot len += 6; 17398621f407SFrançois Tigeot 17408621f407SFrançois Tigeot if (dc_flush_wa) 17418621f407SFrançois Tigeot len += 12; 17428621f407SFrançois Tigeot 17438621f407SFrançois Tigeot ret = intel_ring_begin(request, len); 17441b13d190SFrançois Tigeot if (ret) 17451b13d190SFrançois Tigeot return ret; 17461b13d190SFrançois Tigeot 174719c468b4SFrançois Tigeot if (vf_flush_wa) { 174819c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 174919c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 175019c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 175119c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 175219c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 175319c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 175419c468b4SFrançois Tigeot } 175519c468b4SFrançois Tigeot 17568621f407SFrançois Tigeot if (dc_flush_wa) { 17578621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 17588621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, PIPE_CONTROL_DC_FLUSH_ENABLE); 17598621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17608621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17618621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17628621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17638621f407SFrançois Tigeot } 17648621f407SFrançois Tigeot 17651b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 17661b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, flags); 17671b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, scratch_addr); 17681b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17691b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17701b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17718621f407SFrançois Tigeot 17728621f407SFrançois Tigeot if (dc_flush_wa) { 17738621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 17748621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, PIPE_CONTROL_CS_STALL); 17758621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17768621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17778621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17788621f407SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17798621f407SFrançois Tigeot } 17808621f407SFrançois Tigeot 17811b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 17821b13d190SFrançois Tigeot 17831b13d190SFrançois Tigeot return 0; 17841b13d190SFrançois Tigeot } 17851b13d190SFrançois Tigeot 17868621f407SFrançois Tigeot static u32 gen8_get_seqno(struct intel_engine_cs *engine) 17871b13d190SFrançois Tigeot { 17888621f407SFrançois Tigeot return intel_read_status_page(engine, I915_GEM_HWS_INDEX); 17891b13d190SFrançois Tigeot } 17901b13d190SFrançois Tigeot 17918621f407SFrançois Tigeot static void gen8_set_seqno(struct intel_engine_cs *engine, u32 seqno) 17921b13d190SFrançois Tigeot { 17938621f407SFrançois Tigeot intel_write_status_page(engine, I915_GEM_HWS_INDEX, seqno); 17941b13d190SFrançois Tigeot } 17951b13d190SFrançois Tigeot 17968621f407SFrançois Tigeot static void bxt_a_seqno_barrier(struct intel_engine_cs *engine) 1797352ff8bdSFrançois Tigeot { 1798352ff8bdSFrançois Tigeot /* 1799352ff8bdSFrançois Tigeot * On BXT A steppings there is a HW coherency issue whereby the 1800352ff8bdSFrançois Tigeot * MI_STORE_DATA_IMM storing the completed request's seqno 1801352ff8bdSFrançois Tigeot * occasionally doesn't invalidate the CPU cache. Work around this by 1802352ff8bdSFrançois Tigeot * clflushing the corresponding cacheline whenever the caller wants 1803352ff8bdSFrançois Tigeot * the coherency to be guaranteed. Note that this cacheline is known 1804352ff8bdSFrançois Tigeot * to be clean at this point, since we only write it in 1805352ff8bdSFrançois Tigeot * bxt_a_set_seqno(), where we also do a clflush after the write. So 1806352ff8bdSFrançois Tigeot * this clflush in practice becomes an invalidate operation. 1807352ff8bdSFrançois Tigeot */ 18088621f407SFrançois Tigeot intel_flush_status_page(engine, I915_GEM_HWS_INDEX); 1809352ff8bdSFrançois Tigeot } 1810352ff8bdSFrançois Tigeot 18118621f407SFrançois Tigeot static void bxt_a_set_seqno(struct intel_engine_cs *engine, u32 seqno) 1812352ff8bdSFrançois Tigeot { 18138621f407SFrançois Tigeot intel_write_status_page(engine, I915_GEM_HWS_INDEX, seqno); 1814352ff8bdSFrançois Tigeot 1815352ff8bdSFrançois Tigeot /* See bxt_a_get_seqno() explaining the reason for the clflush. */ 18168621f407SFrançois Tigeot intel_flush_status_page(engine, I915_GEM_HWS_INDEX); 1817352ff8bdSFrançois Tigeot } 1818352ff8bdSFrançois Tigeot 1819477eb7f9SFrançois Tigeot /* 1820477eb7f9SFrançois Tigeot * Reserve space for 2 NOOPs at the end of each request to be 1821477eb7f9SFrançois Tigeot * used as a workaround for not being allowed to do lite 1822477eb7f9SFrançois Tigeot * restore with HEAD==TAIL (WaIdleLiteRestore). 1823477eb7f9SFrançois Tigeot */ 1824c0e85e96SFrançois Tigeot #define WA_TAIL_DWORDS 2 1825c0e85e96SFrançois Tigeot 1826c0e85e96SFrançois Tigeot static int gen8_emit_request(struct drm_i915_gem_request *request) 1827c0e85e96SFrançois Tigeot { 1828c0e85e96SFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 1829c0e85e96SFrançois Tigeot int ret; 1830c0e85e96SFrançois Tigeot 18318621f407SFrançois Tigeot ret = intel_ring_begin(request, 6 + WA_TAIL_DWORDS); 18321b13d190SFrançois Tigeot if (ret) 18331b13d190SFrançois Tigeot return ret; 18341b13d190SFrançois Tigeot 1835c0e85e96SFrançois Tigeot /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */ 1836c0e85e96SFrançois Tigeot BUILD_BUG_ON(I915_GEM_HWS_INDEX_ADDR & (1 << 5)); 18371b13d190SFrançois Tigeot 18381b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 1839c0e85e96SFrançois Tigeot (MI_FLUSH_DW + 1) | MI_FLUSH_DW_OP_STOREDW); 1840c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, 1841*1487f786SFrançois Tigeot intel_hws_seqno_address(request->engine) | 1842c0e85e96SFrançois Tigeot MI_FLUSH_DW_USE_GTT); 18431b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1844a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, i915_gem_request_get_seqno(request)); 18451b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_USER_INTERRUPT); 18461b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1847c0e85e96SFrançois Tigeot return intel_logical_ring_advance_and_submit(request); 1848c0e85e96SFrançois Tigeot } 18491b13d190SFrançois Tigeot 1850c0e85e96SFrançois Tigeot static int gen8_emit_request_render(struct drm_i915_gem_request *request) 1851c0e85e96SFrançois Tigeot { 1852c0e85e96SFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 1853c0e85e96SFrançois Tigeot int ret; 1854c0e85e96SFrançois Tigeot 18558621f407SFrançois Tigeot ret = intel_ring_begin(request, 8 + WA_TAIL_DWORDS); 1856c0e85e96SFrançois Tigeot if (ret) 1857c0e85e96SFrançois Tigeot return ret; 1858c0e85e96SFrançois Tigeot 1859c0e85e96SFrançois Tigeot /* We're using qword write, seqno should be aligned to 8 bytes. */ 1860c0e85e96SFrançois Tigeot BUILD_BUG_ON(I915_GEM_HWS_INDEX & 1); 1861c0e85e96SFrançois Tigeot 1862c0e85e96SFrançois Tigeot /* w/a for post sync ops following a GPGPU operation we 1863c0e85e96SFrançois Tigeot * need a prior CS_STALL, which is emitted by the flush 1864c0e85e96SFrançois Tigeot * following the batch. 1865477eb7f9SFrançois Tigeot */ 1866c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 1867c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, 1868c0e85e96SFrançois Tigeot (PIPE_CONTROL_GLOBAL_GTT_IVB | 1869c0e85e96SFrançois Tigeot PIPE_CONTROL_CS_STALL | 1870c0e85e96SFrançois Tigeot PIPE_CONTROL_QW_WRITE)); 1871*1487f786SFrançois Tigeot intel_logical_ring_emit(ringbuf, 1872*1487f786SFrançois Tigeot intel_hws_seqno_address(request->engine)); 1873c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1874c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, i915_gem_request_get_seqno(request)); 1875c0e85e96SFrançois Tigeot /* We're thrashing one dword of HWS. */ 1876c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1877c0e85e96SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_USER_INTERRUPT); 1878477eb7f9SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1879c0e85e96SFrançois Tigeot return intel_logical_ring_advance_and_submit(request); 18801b13d190SFrançois Tigeot } 18811b13d190SFrançois Tigeot 1882a05eeebfSFrançois Tigeot static int intel_lr_context_render_state_init(struct drm_i915_gem_request *req) 1883477eb7f9SFrançois Tigeot { 1884477eb7f9SFrançois Tigeot struct render_state so; 1885477eb7f9SFrançois Tigeot int ret; 1886477eb7f9SFrançois Tigeot 18878621f407SFrançois Tigeot ret = i915_gem_render_state_prepare(req->engine, &so); 1888477eb7f9SFrançois Tigeot if (ret) 1889477eb7f9SFrançois Tigeot return ret; 1890477eb7f9SFrançois Tigeot 1891477eb7f9SFrançois Tigeot if (so.rodata == NULL) 1892477eb7f9SFrançois Tigeot return 0; 1893477eb7f9SFrançois Tigeot 18948621f407SFrançois Tigeot ret = req->engine->emit_bb_start(req, so.ggtt_offset, 1895477eb7f9SFrançois Tigeot I915_DISPATCH_SECURE); 1896477eb7f9SFrançois Tigeot if (ret) 1897477eb7f9SFrançois Tigeot goto out; 1898477eb7f9SFrançois Tigeot 18998621f407SFrançois Tigeot ret = req->engine->emit_bb_start(req, 1900a05eeebfSFrançois Tigeot (so.ggtt_offset + so.aux_batch_offset), 1901a05eeebfSFrançois Tigeot I915_DISPATCH_SECURE); 1902a05eeebfSFrançois Tigeot if (ret) 1903a05eeebfSFrançois Tigeot goto out; 1904477eb7f9SFrançois Tigeot 1905a05eeebfSFrançois Tigeot i915_vma_move_to_active(i915_gem_obj_to_ggtt(so.obj), req); 1906a05eeebfSFrançois Tigeot 1907477eb7f9SFrançois Tigeot out: 1908477eb7f9SFrançois Tigeot i915_gem_render_state_fini(&so); 1909477eb7f9SFrançois Tigeot return ret; 1910477eb7f9SFrançois Tigeot } 1911477eb7f9SFrançois Tigeot 1912a05eeebfSFrançois Tigeot static int gen8_init_rcs_context(struct drm_i915_gem_request *req) 19132c9916cdSFrançois Tigeot { 19142c9916cdSFrançois Tigeot int ret; 19152c9916cdSFrançois Tigeot 1916a05eeebfSFrançois Tigeot ret = intel_logical_ring_workarounds_emit(req); 19172c9916cdSFrançois Tigeot if (ret) 19182c9916cdSFrançois Tigeot return ret; 19192c9916cdSFrançois Tigeot 1920a05eeebfSFrançois Tigeot ret = intel_rcs_context_init_mocs(req); 1921a05eeebfSFrançois Tigeot /* 1922a05eeebfSFrançois Tigeot * Failing to program the MOCS is non-fatal.The system will not 1923a05eeebfSFrançois Tigeot * run at peak performance. So generate an error and carry on. 1924a05eeebfSFrançois Tigeot */ 1925a05eeebfSFrançois Tigeot if (ret) 1926a05eeebfSFrançois Tigeot DRM_ERROR("MOCS failed to program: expect performance issues.\n"); 1927a05eeebfSFrançois Tigeot 1928a05eeebfSFrançois Tigeot return intel_lr_context_render_state_init(req); 19292c9916cdSFrançois Tigeot } 19302c9916cdSFrançois Tigeot 19311b13d190SFrançois Tigeot /** 19321b13d190SFrançois Tigeot * intel_logical_ring_cleanup() - deallocate the Engine Command Streamer 19331b13d190SFrançois Tigeot * 1934*1487f786SFrançois Tigeot * @engine: Engine Command Streamer. 19351b13d190SFrançois Tigeot * 19361b13d190SFrançois Tigeot */ 19378621f407SFrançois Tigeot void intel_logical_ring_cleanup(struct intel_engine_cs *engine) 19381b13d190SFrançois Tigeot { 19392c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv; 19401b13d190SFrançois Tigeot 19418621f407SFrançois Tigeot if (!intel_engine_initialized(engine)) 19421b13d190SFrançois Tigeot return; 19431b13d190SFrançois Tigeot 19448621f407SFrançois Tigeot /* 19458621f407SFrançois Tigeot * Tasklet cannot be active at this point due intel_mark_active/idle 19468621f407SFrançois Tigeot * so this is just for documentation. 19478621f407SFrançois Tigeot */ 19488621f407SFrançois Tigeot if (WARN_ON(test_bit(TASKLET_STATE_SCHED, &engine->irq_tasklet.state))) 19498621f407SFrançois Tigeot tasklet_kill(&engine->irq_tasklet); 19502c9916cdSFrançois Tigeot 1951*1487f786SFrançois Tigeot dev_priv = engine->i915; 19528621f407SFrançois Tigeot 19538621f407SFrançois Tigeot if (engine->buffer) { 19548621f407SFrançois Tigeot intel_logical_ring_stop(engine); 19558621f407SFrançois Tigeot WARN_ON((I915_READ_MODE(engine) & MODE_IDLE) == 0); 1956aee94f86SFrançois Tigeot } 19571b13d190SFrançois Tigeot 19588621f407SFrançois Tigeot if (engine->cleanup) 19598621f407SFrançois Tigeot engine->cleanup(engine); 19601b13d190SFrançois Tigeot 19618621f407SFrançois Tigeot i915_cmd_parser_fini_ring(engine); 19628621f407SFrançois Tigeot i915_gem_batch_pool_fini(&engine->batch_pool); 19631b13d190SFrançois Tigeot 19648621f407SFrançois Tigeot if (engine->status_page.obj) { 19658621f407SFrançois Tigeot i915_gem_object_unpin_map(engine->status_page.obj); 19668621f407SFrançois Tigeot engine->status_page.obj = NULL; 19671b13d190SFrançois Tigeot } 1968*1487f786SFrançois Tigeot intel_lr_context_unpin(dev_priv->kernel_context, engine); 1969a05eeebfSFrançois Tigeot 19708621f407SFrançois Tigeot engine->idle_lite_restore_wa = 0; 19718621f407SFrançois Tigeot engine->disable_lite_restore_wa = false; 19728621f407SFrançois Tigeot engine->ctx_desc_template = 0; 1973c0e85e96SFrançois Tigeot 19748621f407SFrançois Tigeot lrc_destroy_wa_ctx_obj(engine); 1975*1487f786SFrançois Tigeot engine->i915 = NULL; 19761b13d190SFrançois Tigeot } 19771b13d190SFrançois Tigeot 1978c0e85e96SFrançois Tigeot static void 1979*1487f786SFrançois Tigeot logical_ring_default_vfuncs(struct intel_engine_cs *engine) 19801b13d190SFrançois Tigeot { 1981c0e85e96SFrançois Tigeot /* Default vfuncs which can be overriden by each engine. */ 19828621f407SFrançois Tigeot engine->init_hw = gen8_init_common_ring; 19838621f407SFrançois Tigeot engine->emit_request = gen8_emit_request; 19848621f407SFrançois Tigeot engine->emit_flush = gen8_emit_flush; 19858621f407SFrançois Tigeot engine->irq_get = gen8_logical_ring_get_irq; 19868621f407SFrançois Tigeot engine->irq_put = gen8_logical_ring_put_irq; 19878621f407SFrançois Tigeot engine->emit_bb_start = gen8_emit_bb_start; 19888621f407SFrançois Tigeot engine->get_seqno = gen8_get_seqno; 19898621f407SFrançois Tigeot engine->set_seqno = gen8_set_seqno; 1990*1487f786SFrançois Tigeot if (IS_BXT_REVID(engine->i915, 0, BXT_REVID_A1)) { 19918621f407SFrançois Tigeot engine->irq_seqno_barrier = bxt_a_seqno_barrier; 19928621f407SFrançois Tigeot engine->set_seqno = bxt_a_set_seqno; 1993c0e85e96SFrançois Tigeot } 1994c0e85e96SFrançois Tigeot } 1995c0e85e96SFrançois Tigeot 1996c0e85e96SFrançois Tigeot static inline void 19978621f407SFrançois Tigeot logical_ring_default_irqs(struct intel_engine_cs *engine, unsigned shift) 1998c0e85e96SFrançois Tigeot { 19998621f407SFrançois Tigeot engine->irq_enable_mask = GT_RENDER_USER_INTERRUPT << shift; 20008621f407SFrançois Tigeot engine->irq_keep_mask = GT_CONTEXT_SWITCH_INTERRUPT << shift; 2001*1487f786SFrançois Tigeot init_waitqueue_head(&engine->irq_queue); 2002c0e85e96SFrançois Tigeot } 2003c0e85e96SFrançois Tigeot 2004c0e85e96SFrançois Tigeot static int 20058621f407SFrançois Tigeot lrc_setup_hws(struct intel_engine_cs *engine, 20068621f407SFrançois Tigeot struct drm_i915_gem_object *dctx_obj) 2007c0e85e96SFrançois Tigeot { 2008*1487f786SFrançois Tigeot void *hws; 20098621f407SFrançois Tigeot 20108621f407SFrançois Tigeot /* The HWSP is part of the default context object in LRC mode. */ 20118621f407SFrançois Tigeot engine->status_page.gfx_addr = i915_gem_obj_ggtt_offset(dctx_obj) + 20128621f407SFrançois Tigeot LRC_PPHWSP_PN * PAGE_SIZE; 20138621f407SFrançois Tigeot hws = i915_gem_object_pin_map(dctx_obj); 20148621f407SFrançois Tigeot if (IS_ERR(hws)) 20158621f407SFrançois Tigeot return PTR_ERR(hws); 2016*1487f786SFrançois Tigeot engine->status_page.page_addr = hws + LRC_PPHWSP_PN * PAGE_SIZE; 20178621f407SFrançois Tigeot engine->status_page.obj = dctx_obj; 20188621f407SFrançois Tigeot 20198621f407SFrançois Tigeot return 0; 20208621f407SFrançois Tigeot } 20218621f407SFrançois Tigeot 20228621f407SFrançois Tigeot static int 2023*1487f786SFrançois Tigeot logical_ring_init(struct intel_engine_cs *engine) 20248621f407SFrançois Tigeot { 2025*1487f786SFrançois Tigeot struct i915_gem_context *dctx = engine->i915->kernel_context; 20261b13d190SFrançois Tigeot int ret; 20271b13d190SFrançois Tigeot 20288621f407SFrançois Tigeot ret = i915_cmd_parser_init_ring(engine); 20291b13d190SFrançois Tigeot if (ret) 2030aee94f86SFrançois Tigeot goto error; 20311b13d190SFrançois Tigeot 2032*1487f786SFrançois Tigeot ret = execlists_context_deferred_alloc(dctx, engine); 2033352ff8bdSFrançois Tigeot if (ret) 2034aee94f86SFrançois Tigeot goto error; 2035352ff8bdSFrançois Tigeot 2036352ff8bdSFrançois Tigeot /* As this is the default context, always pin it */ 2037*1487f786SFrançois Tigeot ret = intel_lr_context_pin(dctx, engine); 2038352ff8bdSFrançois Tigeot if (ret) { 2039*1487f786SFrançois Tigeot DRM_ERROR("Failed to pin context for %s: %d\n", 20408621f407SFrançois Tigeot engine->name, ret); 20418621f407SFrançois Tigeot goto error; 20428621f407SFrançois Tigeot } 20438621f407SFrançois Tigeot 20448621f407SFrançois Tigeot /* And setup the hardware status page. */ 20458621f407SFrançois Tigeot ret = lrc_setup_hws(engine, dctx->engine[engine->id].state); 20468621f407SFrançois Tigeot if (ret) { 20478621f407SFrançois Tigeot DRM_ERROR("Failed to set up hws %s: %d\n", engine->name, ret); 2048aee94f86SFrançois Tigeot goto error; 2049352ff8bdSFrançois Tigeot } 20501b13d190SFrançois Tigeot 2051aee94f86SFrançois Tigeot return 0; 2052aee94f86SFrançois Tigeot 2053aee94f86SFrançois Tigeot error: 20548621f407SFrançois Tigeot intel_logical_ring_cleanup(engine); 20551b13d190SFrançois Tigeot return ret; 20561b13d190SFrançois Tigeot } 20571b13d190SFrançois Tigeot 2058*1487f786SFrançois Tigeot static int logical_render_ring_init(struct intel_engine_cs *engine) 20591b13d190SFrançois Tigeot { 2060*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = engine->i915; 20612c9916cdSFrançois Tigeot int ret; 20621b13d190SFrançois Tigeot 2063*1487f786SFrançois Tigeot if (HAS_L3_DPF(dev_priv)) 20648621f407SFrançois Tigeot engine->irq_keep_mask |= GT_RENDER_L3_PARITY_ERROR_INTERRUPT; 20651b13d190SFrançois Tigeot 2066c0e85e96SFrançois Tigeot /* Override some for render ring. */ 2067*1487f786SFrançois Tigeot if (INTEL_GEN(dev_priv) >= 9) 20688621f407SFrançois Tigeot engine->init_hw = gen9_init_render_ring; 2069477eb7f9SFrançois Tigeot else 20708621f407SFrançois Tigeot engine->init_hw = gen8_init_render_ring; 20718621f407SFrançois Tigeot engine->init_context = gen8_init_rcs_context; 20728621f407SFrançois Tigeot engine->cleanup = intel_fini_pipe_control; 20738621f407SFrançois Tigeot engine->emit_flush = gen8_emit_flush_render; 20748621f407SFrançois Tigeot engine->emit_request = gen8_emit_request_render; 20751b13d190SFrançois Tigeot 20768621f407SFrançois Tigeot ret = intel_init_pipe_control(engine); 20772c9916cdSFrançois Tigeot if (ret) 20782c9916cdSFrançois Tigeot return ret; 20792c9916cdSFrançois Tigeot 20808621f407SFrançois Tigeot ret = intel_init_workaround_bb(engine); 2081a05eeebfSFrançois Tigeot if (ret) { 2082a05eeebfSFrançois Tigeot /* 2083a05eeebfSFrançois Tigeot * We continue even if we fail to initialize WA batch 2084a05eeebfSFrançois Tigeot * because we only expect rare glitches but nothing 2085a05eeebfSFrançois Tigeot * critical to prevent us from using GPU 2086a05eeebfSFrançois Tigeot */ 2087a05eeebfSFrançois Tigeot DRM_ERROR("WA batch buffer initialization failed: %d\n", 2088a05eeebfSFrançois Tigeot ret); 2089a05eeebfSFrançois Tigeot } 2090a05eeebfSFrançois Tigeot 2091*1487f786SFrançois Tigeot ret = logical_ring_init(engine); 2092a05eeebfSFrançois Tigeot if (ret) { 20938621f407SFrançois Tigeot lrc_destroy_wa_ctx_obj(engine); 2094a05eeebfSFrançois Tigeot } 2095a05eeebfSFrançois Tigeot 2096a05eeebfSFrançois Tigeot return ret; 20971b13d190SFrançois Tigeot } 20981b13d190SFrançois Tigeot 2099*1487f786SFrançois Tigeot static const struct logical_ring_info { 2100*1487f786SFrançois Tigeot const char *name; 2101*1487f786SFrançois Tigeot unsigned exec_id; 2102*1487f786SFrançois Tigeot unsigned guc_id; 2103*1487f786SFrançois Tigeot u32 mmio_base; 2104*1487f786SFrançois Tigeot unsigned irq_shift; 2105*1487f786SFrançois Tigeot int (*init)(struct intel_engine_cs *engine); 2106*1487f786SFrançois Tigeot } logical_rings[] = { 2107*1487f786SFrançois Tigeot [RCS] = { 2108*1487f786SFrançois Tigeot .name = "render ring", 2109*1487f786SFrançois Tigeot .exec_id = I915_EXEC_RENDER, 2110*1487f786SFrançois Tigeot .guc_id = GUC_RENDER_ENGINE, 2111*1487f786SFrançois Tigeot .mmio_base = RENDER_RING_BASE, 2112*1487f786SFrançois Tigeot .irq_shift = GEN8_RCS_IRQ_SHIFT, 2113*1487f786SFrançois Tigeot .init = logical_render_ring_init, 2114*1487f786SFrançois Tigeot }, 2115*1487f786SFrançois Tigeot [BCS] = { 2116*1487f786SFrançois Tigeot .name = "blitter ring", 2117*1487f786SFrançois Tigeot .exec_id = I915_EXEC_BLT, 2118*1487f786SFrançois Tigeot .guc_id = GUC_BLITTER_ENGINE, 2119*1487f786SFrançois Tigeot .mmio_base = BLT_RING_BASE, 2120*1487f786SFrançois Tigeot .irq_shift = GEN8_BCS_IRQ_SHIFT, 2121*1487f786SFrançois Tigeot .init = logical_ring_init, 2122*1487f786SFrançois Tigeot }, 2123*1487f786SFrançois Tigeot [VCS] = { 2124*1487f786SFrançois Tigeot .name = "bsd ring", 2125*1487f786SFrançois Tigeot .exec_id = I915_EXEC_BSD, 2126*1487f786SFrançois Tigeot .guc_id = GUC_VIDEO_ENGINE, 2127*1487f786SFrançois Tigeot .mmio_base = GEN6_BSD_RING_BASE, 2128*1487f786SFrançois Tigeot .irq_shift = GEN8_VCS1_IRQ_SHIFT, 2129*1487f786SFrançois Tigeot .init = logical_ring_init, 2130*1487f786SFrançois Tigeot }, 2131*1487f786SFrançois Tigeot [VCS2] = { 2132*1487f786SFrançois Tigeot .name = "bsd2 ring", 2133*1487f786SFrançois Tigeot .exec_id = I915_EXEC_BSD, 2134*1487f786SFrançois Tigeot .guc_id = GUC_VIDEO_ENGINE2, 2135*1487f786SFrançois Tigeot .mmio_base = GEN8_BSD2_RING_BASE, 2136*1487f786SFrançois Tigeot .irq_shift = GEN8_VCS2_IRQ_SHIFT, 2137*1487f786SFrançois Tigeot .init = logical_ring_init, 2138*1487f786SFrançois Tigeot }, 2139*1487f786SFrançois Tigeot [VECS] = { 2140*1487f786SFrançois Tigeot .name = "video enhancement ring", 2141*1487f786SFrançois Tigeot .exec_id = I915_EXEC_VEBOX, 2142*1487f786SFrançois Tigeot .guc_id = GUC_VIDEOENHANCE_ENGINE, 2143*1487f786SFrançois Tigeot .mmio_base = VEBOX_RING_BASE, 2144*1487f786SFrançois Tigeot .irq_shift = GEN8_VECS_IRQ_SHIFT, 2145*1487f786SFrançois Tigeot .init = logical_ring_init, 2146*1487f786SFrançois Tigeot }, 2147*1487f786SFrançois Tigeot }; 2148*1487f786SFrançois Tigeot 2149*1487f786SFrançois Tigeot static struct intel_engine_cs * 2150*1487f786SFrançois Tigeot logical_ring_setup(struct drm_i915_private *dev_priv, enum intel_engine_id id) 21511b13d190SFrançois Tigeot { 2152*1487f786SFrançois Tigeot const struct logical_ring_info *info = &logical_rings[id]; 2153*1487f786SFrançois Tigeot struct intel_engine_cs *engine = &dev_priv->engine[id]; 2154*1487f786SFrançois Tigeot enum forcewake_domains fw_domains; 21551b13d190SFrançois Tigeot 2156*1487f786SFrançois Tigeot engine->id = id; 2157*1487f786SFrançois Tigeot engine->name = info->name; 2158*1487f786SFrançois Tigeot engine->exec_id = info->exec_id; 2159*1487f786SFrançois Tigeot engine->guc_id = info->guc_id; 2160*1487f786SFrançois Tigeot engine->mmio_base = info->mmio_base; 21611b13d190SFrançois Tigeot 2162*1487f786SFrançois Tigeot engine->i915 = dev_priv; 21631b13d190SFrançois Tigeot 2164*1487f786SFrançois Tigeot /* Intentionally left blank. */ 2165*1487f786SFrançois Tigeot engine->buffer = NULL; 21661b13d190SFrançois Tigeot 2167*1487f786SFrançois Tigeot fw_domains = intel_uncore_forcewake_for_reg(dev_priv, 2168*1487f786SFrançois Tigeot RING_ELSP(engine), 2169*1487f786SFrançois Tigeot FW_REG_WRITE); 21701b13d190SFrançois Tigeot 2171*1487f786SFrançois Tigeot fw_domains |= intel_uncore_forcewake_for_reg(dev_priv, 2172*1487f786SFrançois Tigeot RING_CONTEXT_STATUS_PTR(engine), 2173*1487f786SFrançois Tigeot FW_REG_READ | FW_REG_WRITE); 21741b13d190SFrançois Tigeot 2175*1487f786SFrançois Tigeot fw_domains |= intel_uncore_forcewake_for_reg(dev_priv, 2176*1487f786SFrançois Tigeot RING_CONTEXT_STATUS_BUF_BASE(engine), 2177*1487f786SFrançois Tigeot FW_REG_READ); 21781b13d190SFrançois Tigeot 2179*1487f786SFrançois Tigeot engine->fw_domains = fw_domains; 21801b13d190SFrançois Tigeot 2181*1487f786SFrançois Tigeot INIT_LIST_HEAD(&engine->active_list); 2182*1487f786SFrançois Tigeot INIT_LIST_HEAD(&engine->request_list); 2183*1487f786SFrançois Tigeot INIT_LIST_HEAD(&engine->buffers); 2184*1487f786SFrançois Tigeot INIT_LIST_HEAD(&engine->execlist_queue); 2185*1487f786SFrançois Tigeot lockinit(&engine->execlist_lock, "i915el", 0, LK_CANRECURSE); 21861b13d190SFrançois Tigeot 2187*1487f786SFrançois Tigeot tasklet_init(&engine->irq_tasklet, 2188*1487f786SFrançois Tigeot intel_lrc_irq_handler, (unsigned long)engine); 21891b13d190SFrançois Tigeot 2190*1487f786SFrançois Tigeot logical_ring_init_platform_invariants(engine); 2191*1487f786SFrançois Tigeot logical_ring_default_vfuncs(engine); 2192*1487f786SFrançois Tigeot logical_ring_default_irqs(engine, info->irq_shift); 21931b13d190SFrançois Tigeot 2194*1487f786SFrançois Tigeot intel_engine_init_hangcheck(engine); 2195*1487f786SFrançois Tigeot i915_gem_batch_pool_init(dev_priv->dev, &engine->batch_pool); 21961b13d190SFrançois Tigeot 2197*1487f786SFrançois Tigeot return engine; 21981b13d190SFrançois Tigeot } 21991b13d190SFrançois Tigeot 22001b13d190SFrançois Tigeot /** 22011b13d190SFrançois Tigeot * intel_logical_rings_init() - allocate, populate and init the Engine Command Streamers 22021b13d190SFrançois Tigeot * @dev: DRM device. 22031b13d190SFrançois Tigeot * 2204*1487f786SFrançois Tigeot * This function inits the engines for an Execlists submission style (the 2205*1487f786SFrançois Tigeot * equivalent in the legacy ringbuffer submission world would be 2206*1487f786SFrançois Tigeot * i915_gem_init_engines). It does it only for those engines that are present in 2207*1487f786SFrançois Tigeot * the hardware. 22081b13d190SFrançois Tigeot * 22091b13d190SFrançois Tigeot * Return: non-zero if the initialization failed. 22101b13d190SFrançois Tigeot */ 22111b13d190SFrançois Tigeot int intel_logical_rings_init(struct drm_device *dev) 22121b13d190SFrançois Tigeot { 22131b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 2214*1487f786SFrançois Tigeot unsigned int mask = 0; 2215*1487f786SFrançois Tigeot unsigned int i; 22161b13d190SFrançois Tigeot int ret; 22171b13d190SFrançois Tigeot 2218*1487f786SFrançois Tigeot WARN_ON(INTEL_INFO(dev_priv)->ring_mask & 2219*1487f786SFrançois Tigeot GENMASK(sizeof(mask) * BITS_PER_BYTE - 1, I915_NUM_ENGINES)); 22201b13d190SFrançois Tigeot 2221*1487f786SFrançois Tigeot for (i = 0; i < ARRAY_SIZE(logical_rings); i++) { 2222*1487f786SFrançois Tigeot if (!HAS_ENGINE(dev_priv, i)) 2223*1487f786SFrançois Tigeot continue; 2224*1487f786SFrançois Tigeot 2225*1487f786SFrançois Tigeot if (!logical_rings[i].init) 2226*1487f786SFrançois Tigeot continue; 2227*1487f786SFrançois Tigeot 2228*1487f786SFrançois Tigeot ret = logical_rings[i].init(logical_ring_setup(dev_priv, i)); 22291b13d190SFrançois Tigeot if (ret) 2230*1487f786SFrançois Tigeot goto cleanup; 2231*1487f786SFrançois Tigeot 2232*1487f786SFrançois Tigeot mask |= ENGINE_MASK(i); 22331b13d190SFrançois Tigeot } 22341b13d190SFrançois Tigeot 2235*1487f786SFrançois Tigeot /* 2236*1487f786SFrançois Tigeot * Catch failures to update logical_rings table when the new engines 2237*1487f786SFrançois Tigeot * are added to the driver by a warning and disabling the forgotten 2238*1487f786SFrançois Tigeot * engines. 2239*1487f786SFrançois Tigeot */ 2240*1487f786SFrançois Tigeot if (WARN_ON(mask != INTEL_INFO(dev_priv)->ring_mask)) { 2241*1487f786SFrançois Tigeot struct intel_device_info *info = 2242*1487f786SFrançois Tigeot (struct intel_device_info *)&dev_priv->info; 2243*1487f786SFrançois Tigeot info->ring_mask = mask; 22441b13d190SFrançois Tigeot } 22451b13d190SFrançois Tigeot 22461b13d190SFrançois Tigeot return 0; 22471b13d190SFrançois Tigeot 2248*1487f786SFrançois Tigeot cleanup: 2249*1487f786SFrançois Tigeot for (i = 0; i < I915_NUM_ENGINES; i++) 2250*1487f786SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->engine[i]); 22511b13d190SFrançois Tigeot 22521b13d190SFrançois Tigeot return ret; 22531b13d190SFrançois Tigeot } 22541b13d190SFrançois Tigeot 2255477eb7f9SFrançois Tigeot static u32 2256*1487f786SFrançois Tigeot make_rpcs(struct drm_i915_private *dev_priv) 22571b13d190SFrançois Tigeot { 2258477eb7f9SFrançois Tigeot u32 rpcs = 0; 22591b13d190SFrançois Tigeot 2260477eb7f9SFrançois Tigeot /* 2261477eb7f9SFrançois Tigeot * No explicit RPCS request is needed to ensure full 2262477eb7f9SFrançois Tigeot * slice/subslice/EU enablement prior to Gen9. 2263477eb7f9SFrançois Tigeot */ 2264*1487f786SFrançois Tigeot if (INTEL_GEN(dev_priv) < 9) 22651b13d190SFrançois Tigeot return 0; 22661b13d190SFrançois Tigeot 2267477eb7f9SFrançois Tigeot /* 2268477eb7f9SFrançois Tigeot * Starting in Gen9, render power gating can leave 2269477eb7f9SFrançois Tigeot * slice/subslice/EU in a partially enabled state. We 2270477eb7f9SFrançois Tigeot * must make an explicit request through RPCS for full 2271477eb7f9SFrançois Tigeot * enablement. 2272477eb7f9SFrançois Tigeot */ 2273*1487f786SFrançois Tigeot if (INTEL_INFO(dev_priv)->has_slice_pg) { 2274477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_S_CNT_ENABLE; 2275*1487f786SFrançois Tigeot rpcs |= INTEL_INFO(dev_priv)->slice_total << 2276477eb7f9SFrançois Tigeot GEN8_RPCS_S_CNT_SHIFT; 2277477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2278477eb7f9SFrançois Tigeot } 22791b13d190SFrançois Tigeot 2280*1487f786SFrançois Tigeot if (INTEL_INFO(dev_priv)->has_subslice_pg) { 2281477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_SS_CNT_ENABLE; 2282*1487f786SFrançois Tigeot rpcs |= INTEL_INFO(dev_priv)->subslice_per_slice << 2283477eb7f9SFrançois Tigeot GEN8_RPCS_SS_CNT_SHIFT; 2284477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2285477eb7f9SFrançois Tigeot } 22861b13d190SFrançois Tigeot 2287*1487f786SFrançois Tigeot if (INTEL_INFO(dev_priv)->has_eu_pg) { 2288*1487f786SFrançois Tigeot rpcs |= INTEL_INFO(dev_priv)->eu_per_subslice << 2289477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MIN_SHIFT; 2290*1487f786SFrançois Tigeot rpcs |= INTEL_INFO(dev_priv)->eu_per_subslice << 2291477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MAX_SHIFT; 2292477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2293477eb7f9SFrançois Tigeot } 2294477eb7f9SFrançois Tigeot 2295477eb7f9SFrançois Tigeot return rpcs; 22961b13d190SFrançois Tigeot } 22971b13d190SFrançois Tigeot 22988621f407SFrançois Tigeot static u32 intel_lr_indirect_ctx_offset(struct intel_engine_cs *engine) 2299c0e85e96SFrançois Tigeot { 2300c0e85e96SFrançois Tigeot u32 indirect_ctx_offset; 2301c0e85e96SFrançois Tigeot 2302*1487f786SFrançois Tigeot switch (INTEL_GEN(engine->i915)) { 2303c0e85e96SFrançois Tigeot default: 2304*1487f786SFrançois Tigeot MISSING_CASE(INTEL_GEN(engine->i915)); 2305c0e85e96SFrançois Tigeot /* fall through */ 2306c0e85e96SFrançois Tigeot case 9: 2307c0e85e96SFrançois Tigeot indirect_ctx_offset = 2308c0e85e96SFrançois Tigeot GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; 2309c0e85e96SFrançois Tigeot break; 2310c0e85e96SFrançois Tigeot case 8: 2311c0e85e96SFrançois Tigeot indirect_ctx_offset = 2312c0e85e96SFrançois Tigeot GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; 2313c0e85e96SFrançois Tigeot break; 2314c0e85e96SFrançois Tigeot } 2315c0e85e96SFrançois Tigeot 2316c0e85e96SFrançois Tigeot return indirect_ctx_offset; 2317c0e85e96SFrançois Tigeot } 2318c0e85e96SFrançois Tigeot 23191b13d190SFrançois Tigeot static int 2320*1487f786SFrançois Tigeot populate_lr_context(struct i915_gem_context *ctx, 23218621f407SFrançois Tigeot struct drm_i915_gem_object *ctx_obj, 23228621f407SFrançois Tigeot struct intel_engine_cs *engine, 23238621f407SFrançois Tigeot struct intel_ringbuffer *ringbuf) 23241b13d190SFrançois Tigeot { 2325*1487f786SFrançois Tigeot struct drm_i915_private *dev_priv = ctx->i915; 23261b13d190SFrançois Tigeot struct i915_hw_ppgtt *ppgtt = ctx->ppgtt; 2327*1487f786SFrançois Tigeot void *vaddr; 23288621f407SFrançois Tigeot u32 *reg_state; 23291b13d190SFrançois Tigeot int ret; 23301b13d190SFrançois Tigeot 23311b13d190SFrançois Tigeot if (!ppgtt) 23321b13d190SFrançois Tigeot ppgtt = dev_priv->mm.aliasing_ppgtt; 23331b13d190SFrançois Tigeot 23341b13d190SFrançois Tigeot ret = i915_gem_object_set_to_cpu_domain(ctx_obj, true); 23351b13d190SFrançois Tigeot if (ret) { 23361b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Could not set to CPU domain\n"); 23371b13d190SFrançois Tigeot return ret; 23381b13d190SFrançois Tigeot } 23391b13d190SFrançois Tigeot 23408621f407SFrançois Tigeot vaddr = i915_gem_object_pin_map(ctx_obj); 23418621f407SFrançois Tigeot if (IS_ERR(vaddr)) { 23428621f407SFrançois Tigeot ret = PTR_ERR(vaddr); 23438621f407SFrançois Tigeot DRM_DEBUG_DRIVER("Could not map object pages! (%d)\n", ret); 23441b13d190SFrançois Tigeot return ret; 23451b13d190SFrançois Tigeot } 23468621f407SFrançois Tigeot ctx_obj->dirty = true; 23471b13d190SFrançois Tigeot 23481b13d190SFrançois Tigeot /* The second page of the context object contains some fields which must 23491b13d190SFrançois Tigeot * be set up prior to the first execution. */ 2350*1487f786SFrançois Tigeot reg_state = vaddr + LRC_STATE_PN * PAGE_SIZE; 23511b13d190SFrançois Tigeot 23521b13d190SFrançois Tigeot /* A context is actually a big batch buffer with several MI_LOAD_REGISTER_IMM 23531b13d190SFrançois Tigeot * commands followed by (reg, value) pairs. The values we are setting here are 23541b13d190SFrançois Tigeot * only for the first context restore: on a subsequent save, the GPU will 23551b13d190SFrançois Tigeot * recreate this batchbuffer with new values (including all the missing 23561b13d190SFrançois Tigeot * MI_LOAD_REGISTER_IMM commands that we are not initializing here). */ 2357aee94f86SFrançois Tigeot reg_state[CTX_LRI_HEADER_0] = 23588621f407SFrançois Tigeot MI_LOAD_REGISTER_IMM(engine->id == RCS ? 14 : 11) | MI_LRI_FORCE_POSTED; 23598621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_CONTEXT_CONTROL, 23608621f407SFrançois Tigeot RING_CONTEXT_CONTROL(engine), 2361477eb7f9SFrançois Tigeot _MASKED_BIT_ENABLE(CTX_CTRL_INHIBIT_SYN_CTX_SWITCH | 2362a05eeebfSFrançois Tigeot CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT | 2363*1487f786SFrançois Tigeot (HAS_RESOURCE_STREAMER(dev_priv) ? 2364c0e85e96SFrançois Tigeot CTX_CTRL_RS_CTX_ENABLE : 0))); 23658621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_HEAD, RING_HEAD(engine->mmio_base), 23668621f407SFrançois Tigeot 0); 23678621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_TAIL, RING_TAIL(engine->mmio_base), 23688621f407SFrançois Tigeot 0); 23692c9916cdSFrançois Tigeot /* Ring buffer start address is not known until the buffer is pinned. 23702c9916cdSFrançois Tigeot * It is written to the context image in execlists_update_context() 23712c9916cdSFrançois Tigeot */ 23728621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_BUFFER_START, 23738621f407SFrançois Tigeot RING_START(engine->mmio_base), 0); 23748621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_BUFFER_CONTROL, 23758621f407SFrançois Tigeot RING_CTL(engine->mmio_base), 2376aee94f86SFrançois Tigeot ((ringbuf->size - PAGE_SIZE) & RING_NR_PAGES) | RING_VALID); 23778621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_HEAD_U, 23788621f407SFrançois Tigeot RING_BBADDR_UDW(engine->mmio_base), 0); 23798621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_HEAD_L, 23808621f407SFrançois Tigeot RING_BBADDR(engine->mmio_base), 0); 23818621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_STATE, 23828621f407SFrançois Tigeot RING_BBSTATE(engine->mmio_base), 2383aee94f86SFrançois Tigeot RING_BB_PPGTT); 23848621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_HEAD_U, 23858621f407SFrançois Tigeot RING_SBBADDR_UDW(engine->mmio_base), 0); 23868621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_HEAD_L, 23878621f407SFrançois Tigeot RING_SBBADDR(engine->mmio_base), 0); 23888621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_STATE, 23898621f407SFrançois Tigeot RING_SBBSTATE(engine->mmio_base), 0); 23908621f407SFrançois Tigeot if (engine->id == RCS) { 23918621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_PER_CTX_PTR, 23928621f407SFrançois Tigeot RING_BB_PER_CTX_PTR(engine->mmio_base), 0); 23938621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RCS_INDIRECT_CTX, 23948621f407SFrançois Tigeot RING_INDIRECT_CTX(engine->mmio_base), 0); 23958621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RCS_INDIRECT_CTX_OFFSET, 23968621f407SFrançois Tigeot RING_INDIRECT_CTX_OFFSET(engine->mmio_base), 0); 23978621f407SFrançois Tigeot if (engine->wa_ctx.obj) { 23988621f407SFrançois Tigeot struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx; 2399a05eeebfSFrançois Tigeot uint32_t ggtt_offset = i915_gem_obj_ggtt_offset(wa_ctx->obj); 2400a05eeebfSFrançois Tigeot 2401a05eeebfSFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX+1] = 2402a05eeebfSFrançois Tigeot (ggtt_offset + wa_ctx->indirect_ctx.offset * sizeof(uint32_t)) | 2403a05eeebfSFrançois Tigeot (wa_ctx->indirect_ctx.size / CACHELINE_DWORDS); 2404a05eeebfSFrançois Tigeot 2405a05eeebfSFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX_OFFSET+1] = 24068621f407SFrançois Tigeot intel_lr_indirect_ctx_offset(engine) << 6; 2407a05eeebfSFrançois Tigeot 2408a05eeebfSFrançois Tigeot reg_state[CTX_BB_PER_CTX_PTR+1] = 2409a05eeebfSFrançois Tigeot (ggtt_offset + wa_ctx->per_ctx.offset * sizeof(uint32_t)) | 2410a05eeebfSFrançois Tigeot 0x01; 2411a05eeebfSFrançois Tigeot } 24121b13d190SFrançois Tigeot } 2413aee94f86SFrançois Tigeot reg_state[CTX_LRI_HEADER_1] = MI_LOAD_REGISTER_IMM(9) | MI_LRI_FORCE_POSTED; 24148621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_CTX_TIMESTAMP, 24158621f407SFrançois Tigeot RING_CTX_TIMESTAMP(engine->mmio_base), 0); 2416aee94f86SFrançois Tigeot /* PDP values well be assigned later if needed */ 24178621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP3_UDW, GEN8_RING_PDP_UDW(engine, 3), 24188621f407SFrançois Tigeot 0); 24198621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP3_LDW, GEN8_RING_PDP_LDW(engine, 3), 24208621f407SFrançois Tigeot 0); 24218621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP2_UDW, GEN8_RING_PDP_UDW(engine, 2), 24228621f407SFrançois Tigeot 0); 24238621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP2_LDW, GEN8_RING_PDP_LDW(engine, 2), 24248621f407SFrançois Tigeot 0); 24258621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP1_UDW, GEN8_RING_PDP_UDW(engine, 1), 24268621f407SFrançois Tigeot 0); 24278621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP1_LDW, GEN8_RING_PDP_LDW(engine, 1), 24288621f407SFrançois Tigeot 0); 24298621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP0_UDW, GEN8_RING_PDP_UDW(engine, 0), 24308621f407SFrançois Tigeot 0); 24318621f407SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP0_LDW, GEN8_RING_PDP_LDW(engine, 0), 24328621f407SFrançois Tigeot 0); 243319c468b4SFrançois Tigeot 2434352ff8bdSFrançois Tigeot if (USES_FULL_48BIT_PPGTT(ppgtt->base.dev)) { 2435352ff8bdSFrançois Tigeot /* 64b PPGTT (48bit canonical) 2436352ff8bdSFrançois Tigeot * PDP0_DESCRIPTOR contains the base address to PML4 and 2437352ff8bdSFrançois Tigeot * other PDP Descriptors are ignored. 2438352ff8bdSFrançois Tigeot */ 2439352ff8bdSFrançois Tigeot ASSIGN_CTX_PML4(ppgtt, reg_state); 2440352ff8bdSFrançois Tigeot } else { 2441352ff8bdSFrançois Tigeot /* 32b PPGTT 2442352ff8bdSFrançois Tigeot * PDP*_DESCRIPTOR contains the base address of space supported. 2443352ff8bdSFrançois Tigeot * With dynamic page allocation, PDPs may not be allocated at 2444352ff8bdSFrançois Tigeot * this point. Point the unallocated PDPs to the scratch page 244519c468b4SFrançois Tigeot */ 24468621f407SFrançois Tigeot execlists_update_context_pdps(ppgtt, reg_state); 2447352ff8bdSFrançois Tigeot } 2448352ff8bdSFrançois Tigeot 24498621f407SFrançois Tigeot if (engine->id == RCS) { 24501b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1); 2451aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE, 2452*1487f786SFrançois Tigeot make_rpcs(dev_priv)); 24531b13d190SFrançois Tigeot } 24541b13d190SFrançois Tigeot 24558621f407SFrançois Tigeot i915_gem_object_unpin_map(ctx_obj); 24561b13d190SFrançois Tigeot 24571b13d190SFrançois Tigeot return 0; 24581b13d190SFrançois Tigeot } 24591b13d190SFrançois Tigeot 24601b13d190SFrançois Tigeot /** 2461c0e85e96SFrançois Tigeot * intel_lr_context_size() - return the size of the context for an engine 2462*1487f786SFrançois Tigeot * @engine: which engine to find the context size for 2463c0e85e96SFrançois Tigeot * 2464c0e85e96SFrançois Tigeot * Each engine may require a different amount of space for a context image, 2465c0e85e96SFrançois Tigeot * so when allocating (or copying) an image, this function can be used to 2466c0e85e96SFrançois Tigeot * find the right size for the specific engine. 2467c0e85e96SFrançois Tigeot * 2468c0e85e96SFrançois Tigeot * Return: size (in bytes) of an engine-specific context image 2469c0e85e96SFrançois Tigeot * 2470c0e85e96SFrançois Tigeot * Note: this size includes the HWSP, which is part of the context image 2471c0e85e96SFrançois Tigeot * in LRC mode, but does not include the "shared data page" used with 2472c0e85e96SFrançois Tigeot * GuC submission. The caller should account for this if using the GuC. 2473c0e85e96SFrançois Tigeot */ 24748621f407SFrançois Tigeot uint32_t intel_lr_context_size(struct intel_engine_cs *engine) 24751b13d190SFrançois Tigeot { 24761b13d190SFrançois Tigeot int ret = 0; 24771b13d190SFrançois Tigeot 2478*1487f786SFrançois Tigeot WARN_ON(INTEL_GEN(engine->i915) < 8); 24791b13d190SFrançois Tigeot 24808621f407SFrançois Tigeot switch (engine->id) { 24811b13d190SFrançois Tigeot case RCS: 2482*1487f786SFrançois Tigeot if (INTEL_GEN(engine->i915) >= 9) 24832c9916cdSFrançois Tigeot ret = GEN9_LR_CONTEXT_RENDER_SIZE; 24842c9916cdSFrançois Tigeot else 24851b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_RENDER_SIZE; 24861b13d190SFrançois Tigeot break; 24871b13d190SFrançois Tigeot case VCS: 24881b13d190SFrançois Tigeot case BCS: 24891b13d190SFrançois Tigeot case VECS: 24901b13d190SFrançois Tigeot case VCS2: 24911b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_OTHER_SIZE; 24921b13d190SFrançois Tigeot break; 24931b13d190SFrançois Tigeot } 24941b13d190SFrançois Tigeot 24951b13d190SFrançois Tigeot return ret; 24961b13d190SFrançois Tigeot } 24971b13d190SFrançois Tigeot 24981b13d190SFrançois Tigeot /** 2499*1487f786SFrançois Tigeot * execlists_context_deferred_alloc() - create the LRC specific bits of a context 25001b13d190SFrançois Tigeot * @ctx: LR context to create. 2501*1487f786SFrançois Tigeot * @engine: engine to be used with the context. 25021b13d190SFrançois Tigeot * 25031b13d190SFrançois Tigeot * This function can be called more than once, with different engines, if we plan 25041b13d190SFrançois Tigeot * to use the context with them. The context backing objects and the ringbuffers 25051b13d190SFrançois Tigeot * (specially the ringbuffer backing objects) suck a lot of memory up, and that's why 25061b13d190SFrançois Tigeot * the creation is a deferred call: it's better to make sure first that we need to use 25071b13d190SFrançois Tigeot * a given ring with the context. 25081b13d190SFrançois Tigeot * 25092c9916cdSFrançois Tigeot * Return: non-zero on error. 25101b13d190SFrançois Tigeot */ 2511*1487f786SFrançois Tigeot static int execlists_context_deferred_alloc(struct i915_gem_context *ctx, 25128621f407SFrançois Tigeot struct intel_engine_cs *engine) 25131b13d190SFrançois Tigeot { 25141b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj; 2515*1487f786SFrançois Tigeot struct intel_context *ce = &ctx->engine[engine->id]; 25161b13d190SFrançois Tigeot uint32_t context_size; 25171b13d190SFrançois Tigeot struct intel_ringbuffer *ringbuf; 25181b13d190SFrançois Tigeot int ret; 25191b13d190SFrançois Tigeot 2520*1487f786SFrançois Tigeot WARN_ON(ce->state); 25211b13d190SFrançois Tigeot 25228621f407SFrançois Tigeot context_size = round_up(intel_lr_context_size(engine), 4096); 25231b13d190SFrançois Tigeot 2524352ff8bdSFrançois Tigeot /* One extra page as the sharing data between driver and GuC */ 2525352ff8bdSFrançois Tigeot context_size += PAGE_SIZE * LRC_PPHWSP_PN; 2526352ff8bdSFrançois Tigeot 2527*1487f786SFrançois Tigeot ctx_obj = i915_gem_object_create(ctx->i915->dev, context_size); 2528*1487f786SFrançois Tigeot if (IS_ERR(ctx_obj)) { 252919c468b4SFrançois Tigeot DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n"); 2530*1487f786SFrançois Tigeot return PTR_ERR(ctx_obj); 25311b13d190SFrançois Tigeot } 25321b13d190SFrançois Tigeot 2533*1487f786SFrançois Tigeot ringbuf = intel_engine_create_ringbuffer(engine, ctx->ring_size); 2534352ff8bdSFrançois Tigeot if (IS_ERR(ringbuf)) { 2535352ff8bdSFrançois Tigeot ret = PTR_ERR(ringbuf); 2536352ff8bdSFrançois Tigeot goto error_deref_obj; 25371b13d190SFrançois Tigeot } 25381b13d190SFrançois Tigeot 25398621f407SFrançois Tigeot ret = populate_lr_context(ctx, ctx_obj, engine, ringbuf); 25401b13d190SFrançois Tigeot if (ret) { 25411b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Failed to populate LRC: %d\n", ret); 2542352ff8bdSFrançois Tigeot goto error_ringbuf; 25431b13d190SFrançois Tigeot } 25441b13d190SFrançois Tigeot 2545*1487f786SFrançois Tigeot ce->ringbuf = ringbuf; 2546*1487f786SFrançois Tigeot ce->state = ctx_obj; 2547*1487f786SFrançois Tigeot ce->initialised = engine->init_context == NULL; 25481b13d190SFrançois Tigeot 25491b13d190SFrançois Tigeot return 0; 25501b13d190SFrançois Tigeot 2551352ff8bdSFrançois Tigeot error_ringbuf: 2552352ff8bdSFrançois Tigeot intel_ringbuffer_free(ringbuf); 2553352ff8bdSFrançois Tigeot error_deref_obj: 25541b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 2555*1487f786SFrançois Tigeot ce->ringbuf = NULL; 2556*1487f786SFrançois Tigeot ce->state = NULL; 25571b13d190SFrançois Tigeot return ret; 25581b13d190SFrançois Tigeot } 2559477eb7f9SFrançois Tigeot 25608621f407SFrançois Tigeot void intel_lr_context_reset(struct drm_i915_private *dev_priv, 2561*1487f786SFrançois Tigeot struct i915_gem_context *ctx) 2562477eb7f9SFrançois Tigeot { 25638621f407SFrançois Tigeot struct intel_engine_cs *engine; 2564477eb7f9SFrançois Tigeot 25658621f407SFrançois Tigeot for_each_engine(engine, dev_priv) { 2566*1487f786SFrançois Tigeot struct intel_context *ce = &ctx->engine[engine->id]; 2567*1487f786SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ce->state; 2568*1487f786SFrançois Tigeot void *vaddr; 2569477eb7f9SFrançois Tigeot uint32_t *reg_state; 2570477eb7f9SFrançois Tigeot 2571477eb7f9SFrançois Tigeot if (!ctx_obj) 2572477eb7f9SFrançois Tigeot continue; 2573477eb7f9SFrançois Tigeot 25748621f407SFrançois Tigeot vaddr = i915_gem_object_pin_map(ctx_obj); 25758621f407SFrançois Tigeot if (WARN_ON(IS_ERR(vaddr))) 2576477eb7f9SFrançois Tigeot continue; 25778621f407SFrançois Tigeot 2578*1487f786SFrançois Tigeot reg_state = vaddr + LRC_STATE_PN * PAGE_SIZE; 25798621f407SFrançois Tigeot ctx_obj->dirty = true; 2580477eb7f9SFrançois Tigeot 2581477eb7f9SFrançois Tigeot reg_state[CTX_RING_HEAD+1] = 0; 2582477eb7f9SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = 0; 2583477eb7f9SFrançois Tigeot 25848621f407SFrançois Tigeot i915_gem_object_unpin_map(ctx_obj); 2585477eb7f9SFrançois Tigeot 2586*1487f786SFrançois Tigeot ce->ringbuf->head = 0; 2587*1487f786SFrançois Tigeot ce->ringbuf->tail = 0; 2588477eb7f9SFrançois Tigeot } 2589477eb7f9SFrançois Tigeot } 2590