11b13d190SFrançois Tigeot /* 21b13d190SFrançois Tigeot * Copyright © 2014 Intel Corporation 31b13d190SFrançois Tigeot * 41b13d190SFrançois Tigeot * Permission is hereby granted, free of charge, to any person obtaining a 51b13d190SFrançois Tigeot * copy of this software and associated documentation files (the "Software"), 61b13d190SFrançois Tigeot * to deal in the Software without restriction, including without limitation 71b13d190SFrançois Tigeot * the rights to use, copy, modify, merge, publish, distribute, sublicense, 81b13d190SFrançois Tigeot * and/or sell copies of the Software, and to permit persons to whom the 91b13d190SFrançois Tigeot * Software is furnished to do so, subject to the following conditions: 101b13d190SFrançois Tigeot * 111b13d190SFrançois Tigeot * The above copyright notice and this permission notice (including the next 121b13d190SFrançois Tigeot * paragraph) shall be included in all copies or substantial portions of the 131b13d190SFrançois Tigeot * Software. 141b13d190SFrançois Tigeot * 151b13d190SFrançois Tigeot * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 161b13d190SFrançois Tigeot * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 171b13d190SFrançois Tigeot * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 181b13d190SFrançois Tigeot * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 191b13d190SFrançois Tigeot * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 201b13d190SFrançois Tigeot * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 211b13d190SFrançois Tigeot * IN THE SOFTWARE. 221b13d190SFrançois Tigeot * 231b13d190SFrançois Tigeot * Authors: 241b13d190SFrançois Tigeot * Ben Widawsky <ben@bwidawsk.net> 251b13d190SFrançois Tigeot * Michel Thierry <michel.thierry@intel.com> 261b13d190SFrançois Tigeot * Thomas Daniel <thomas.daniel@intel.com> 271b13d190SFrançois Tigeot * Oscar Mateo <oscar.mateo@intel.com> 281b13d190SFrançois Tigeot * 291b13d190SFrançois Tigeot */ 301b13d190SFrançois Tigeot 311b13d190SFrançois Tigeot /** 321b13d190SFrançois Tigeot * DOC: Logical Rings, Logical Ring Contexts and Execlists 331b13d190SFrançois Tigeot * 341b13d190SFrançois Tigeot * Motivation: 351b13d190SFrançois Tigeot * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts". 361b13d190SFrançois Tigeot * These expanded contexts enable a number of new abilities, especially 371b13d190SFrançois Tigeot * "Execlists" (also implemented in this file). 381b13d190SFrançois Tigeot * 391b13d190SFrançois Tigeot * One of the main differences with the legacy HW contexts is that logical 401b13d190SFrançois Tigeot * ring contexts incorporate many more things to the context's state, like 411b13d190SFrançois Tigeot * PDPs or ringbuffer control registers: 421b13d190SFrançois Tigeot * 431b13d190SFrançois Tigeot * The reason why PDPs are included in the context is straightforward: as 441b13d190SFrançois Tigeot * PPGTTs (per-process GTTs) are actually per-context, having the PDPs 451b13d190SFrançois Tigeot * contained there mean you don't need to do a ppgtt->switch_mm yourself, 461b13d190SFrançois Tigeot * instead, the GPU will do it for you on the context switch. 471b13d190SFrançois Tigeot * 481b13d190SFrançois Tigeot * But, what about the ringbuffer control registers (head, tail, etc..)? 491b13d190SFrançois Tigeot * shouldn't we just need a set of those per engine command streamer? This is 501b13d190SFrançois Tigeot * where the name "Logical Rings" starts to make sense: by virtualizing the 511b13d190SFrançois Tigeot * rings, the engine cs shifts to a new "ring buffer" with every context 521b13d190SFrançois Tigeot * switch. When you want to submit a workload to the GPU you: A) choose your 531b13d190SFrançois Tigeot * context, B) find its appropriate virtualized ring, C) write commands to it 541b13d190SFrançois Tigeot * and then, finally, D) tell the GPU to switch to that context. 551b13d190SFrançois Tigeot * 561b13d190SFrançois Tigeot * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch 571b13d190SFrançois Tigeot * to a contexts is via a context execution list, ergo "Execlists". 581b13d190SFrançois Tigeot * 591b13d190SFrançois Tigeot * LRC implementation: 601b13d190SFrançois Tigeot * Regarding the creation of contexts, we have: 611b13d190SFrançois Tigeot * 621b13d190SFrançois Tigeot * - One global default context. 631b13d190SFrançois Tigeot * - One local default context for each opened fd. 641b13d190SFrançois Tigeot * - One local extra context for each context create ioctl call. 651b13d190SFrançois Tigeot * 661b13d190SFrançois Tigeot * Now that ringbuffers belong per-context (and not per-engine, like before) 671b13d190SFrançois Tigeot * and that contexts are uniquely tied to a given engine (and not reusable, 681b13d190SFrançois Tigeot * like before) we need: 691b13d190SFrançois Tigeot * 701b13d190SFrançois Tigeot * - One ringbuffer per-engine inside each context. 711b13d190SFrançois Tigeot * - One backing object per-engine inside each context. 721b13d190SFrançois Tigeot * 731b13d190SFrançois Tigeot * The global default context starts its life with these new objects fully 741b13d190SFrançois Tigeot * allocated and populated. The local default context for each opened fd is 751b13d190SFrançois Tigeot * more complex, because we don't know at creation time which engine is going 761b13d190SFrançois Tigeot * to use them. To handle this, we have implemented a deferred creation of LR 771b13d190SFrançois Tigeot * contexts: 781b13d190SFrançois Tigeot * 791b13d190SFrançois Tigeot * The local context starts its life as a hollow or blank holder, that only 801b13d190SFrançois Tigeot * gets populated for a given engine once we receive an execbuffer. If later 811b13d190SFrançois Tigeot * on we receive another execbuffer ioctl for the same context but a different 821b13d190SFrançois Tigeot * engine, we allocate/populate a new ringbuffer and context backing object and 831b13d190SFrançois Tigeot * so on. 841b13d190SFrançois Tigeot * 851b13d190SFrançois Tigeot * Finally, regarding local contexts created using the ioctl call: as they are 861b13d190SFrançois Tigeot * only allowed with the render ring, we can allocate & populate them right 871b13d190SFrançois Tigeot * away (no need to defer anything, at least for now). 881b13d190SFrançois Tigeot * 891b13d190SFrançois Tigeot * Execlists implementation: 901b13d190SFrançois Tigeot * Execlists are the new method by which, on gen8+ hardware, workloads are 911b13d190SFrançois Tigeot * submitted for execution (as opposed to the legacy, ringbuffer-based, method). 921b13d190SFrançois Tigeot * This method works as follows: 931b13d190SFrançois Tigeot * 941b13d190SFrançois Tigeot * When a request is committed, its commands (the BB start and any leading or 951b13d190SFrançois Tigeot * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer 961b13d190SFrançois Tigeot * for the appropriate context. The tail pointer in the hardware context is not 971b13d190SFrançois Tigeot * updated at this time, but instead, kept by the driver in the ringbuffer 981b13d190SFrançois Tigeot * structure. A structure representing this request is added to a request queue 991b13d190SFrançois Tigeot * for the appropriate engine: this structure contains a copy of the context's 1001b13d190SFrançois Tigeot * tail after the request was written to the ring buffer and a pointer to the 1011b13d190SFrançois Tigeot * context itself. 1021b13d190SFrançois Tigeot * 1031b13d190SFrançois Tigeot * If the engine's request queue was empty before the request was added, the 1041b13d190SFrançois Tigeot * queue is processed immediately. Otherwise the queue will be processed during 1051b13d190SFrançois Tigeot * a context switch interrupt. In any case, elements on the queue will get sent 1061b13d190SFrançois Tigeot * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a 1071b13d190SFrançois Tigeot * globally unique 20-bits submission ID. 1081b13d190SFrançois Tigeot * 1091b13d190SFrançois Tigeot * When execution of a request completes, the GPU updates the context status 1101b13d190SFrançois Tigeot * buffer with a context complete event and generates a context switch interrupt. 1111b13d190SFrançois Tigeot * During the interrupt handling, the driver examines the events in the buffer: 1121b13d190SFrançois Tigeot * for each context complete event, if the announced ID matches that on the head 1131b13d190SFrançois Tigeot * of the request queue, then that request is retired and removed from the queue. 1141b13d190SFrançois Tigeot * 1151b13d190SFrançois Tigeot * After processing, if any requests were retired and the queue is not empty 1161b13d190SFrançois Tigeot * then a new execution list can be submitted. The two requests at the front of 1171b13d190SFrançois Tigeot * the queue are next to be submitted but since a context may not occur twice in 1181b13d190SFrançois Tigeot * an execution list, if subsequent requests have the same ID as the first then 1191b13d190SFrançois Tigeot * the two requests must be combined. This is done simply by discarding requests 1201b13d190SFrançois Tigeot * at the head of the queue until either only one requests is left (in which case 1211b13d190SFrançois Tigeot * we use a NULL second context) or the first two requests have unique IDs. 1221b13d190SFrançois Tigeot * 1231b13d190SFrançois Tigeot * By always executing the first two requests in the queue the driver ensures 1241b13d190SFrançois Tigeot * that the GPU is kept as busy as possible. In the case where a single context 1251b13d190SFrançois Tigeot * completes but a second context is still executing, the request for this second 1261b13d190SFrançois Tigeot * context will be at the head of the queue when we remove the first one. This 1271b13d190SFrançois Tigeot * request will then be resubmitted along with a new request for a different context, 1281b13d190SFrançois Tigeot * which will cause the hardware to continue executing the second request and queue 1291b13d190SFrançois Tigeot * the new request (the GPU detects the condition of a context getting preempted 1301b13d190SFrançois Tigeot * with the same context and optimizes the context switch flow by not doing 1311b13d190SFrançois Tigeot * preemption, but just sampling the new tail pointer). 1321b13d190SFrançois Tigeot * 1331b13d190SFrançois Tigeot */ 1341b13d190SFrançois Tigeot 1351b13d190SFrançois Tigeot #include <drm/drmP.h> 1361b13d190SFrançois Tigeot #include <drm/i915_drm.h> 1371b13d190SFrançois Tigeot #include "i915_drv.h" 1381b13d190SFrançois Tigeot #include "intel_drv.h" 139a05eeebfSFrançois Tigeot #include "intel_mocs.h" 1401b13d190SFrançois Tigeot 1412c9916cdSFrançois Tigeot #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE) 1421b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE) 1431b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_OTHER_SIZE (2 * PAGE_SIZE) 1441b13d190SFrançois Tigeot 1451b13d190SFrançois Tigeot #define RING_EXECLIST_QFULL (1 << 0x2) 1461b13d190SFrançois Tigeot #define RING_EXECLIST1_VALID (1 << 0x3) 1471b13d190SFrançois Tigeot #define RING_EXECLIST0_VALID (1 << 0x4) 1481b13d190SFrançois Tigeot #define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE) 1491b13d190SFrançois Tigeot #define RING_EXECLIST1_ACTIVE (1 << 0x11) 1501b13d190SFrançois Tigeot #define RING_EXECLIST0_ACTIVE (1 << 0x12) 1511b13d190SFrançois Tigeot 1521b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0) 1531b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_PREEMPTED (1 << 1) 1541b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2) 1551b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3) 1561b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_COMPLETE (1 << 4) 1571b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15) 1581b13d190SFrançois Tigeot 1591b13d190SFrançois Tigeot #define CTX_LRI_HEADER_0 0x01 1601b13d190SFrançois Tigeot #define CTX_CONTEXT_CONTROL 0x02 1611b13d190SFrançois Tigeot #define CTX_RING_HEAD 0x04 1621b13d190SFrançois Tigeot #define CTX_RING_TAIL 0x06 1631b13d190SFrançois Tigeot #define CTX_RING_BUFFER_START 0x08 1641b13d190SFrançois Tigeot #define CTX_RING_BUFFER_CONTROL 0x0a 1651b13d190SFrançois Tigeot #define CTX_BB_HEAD_U 0x0c 1661b13d190SFrançois Tigeot #define CTX_BB_HEAD_L 0x0e 1671b13d190SFrançois Tigeot #define CTX_BB_STATE 0x10 1681b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_U 0x12 1691b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_L 0x14 1701b13d190SFrançois Tigeot #define CTX_SECOND_BB_STATE 0x16 1711b13d190SFrançois Tigeot #define CTX_BB_PER_CTX_PTR 0x18 1721b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX 0x1a 1731b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX_OFFSET 0x1c 1741b13d190SFrançois Tigeot #define CTX_LRI_HEADER_1 0x21 1751b13d190SFrançois Tigeot #define CTX_CTX_TIMESTAMP 0x22 1761b13d190SFrançois Tigeot #define CTX_PDP3_UDW 0x24 1771b13d190SFrançois Tigeot #define CTX_PDP3_LDW 0x26 1781b13d190SFrançois Tigeot #define CTX_PDP2_UDW 0x28 1791b13d190SFrançois Tigeot #define CTX_PDP2_LDW 0x2a 1801b13d190SFrançois Tigeot #define CTX_PDP1_UDW 0x2c 1811b13d190SFrançois Tigeot #define CTX_PDP1_LDW 0x2e 1821b13d190SFrançois Tigeot #define CTX_PDP0_UDW 0x30 1831b13d190SFrançois Tigeot #define CTX_PDP0_LDW 0x32 1841b13d190SFrançois Tigeot #define CTX_LRI_HEADER_2 0x41 1851b13d190SFrançois Tigeot #define CTX_R_PWR_CLK_STATE 0x42 1861b13d190SFrançois Tigeot #define CTX_GPGPU_CSR_BASE_ADDRESS 0x44 1871b13d190SFrançois Tigeot 1881b13d190SFrançois Tigeot #define GEN8_CTX_VALID (1<<0) 1891b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_PD_RESTORE (1<<1) 1901b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_RESTORE (1<<2) 1911b13d190SFrançois Tigeot #define GEN8_CTX_L3LLC_COHERENT (1<<5) 1921b13d190SFrançois Tigeot #define GEN8_CTX_PRIVILEGE (1<<8) 19319c468b4SFrançois Tigeot 194*aee94f86SFrançois Tigeot #define ASSIGN_CTX_REG(reg_state, pos, reg, val) do { \ 195*aee94f86SFrançois Tigeot (reg_state)[(pos)+0] = i915_mmio_reg_offset(reg); \ 196*aee94f86SFrançois Tigeot (reg_state)[(pos)+1] = (val); \ 197*aee94f86SFrançois Tigeot } while (0) 198*aee94f86SFrançois Tigeot 199*aee94f86SFrançois Tigeot #define ASSIGN_CTX_PDP(ppgtt, reg_state, n) do { \ 200a05eeebfSFrançois Tigeot const u64 _addr = i915_page_dir_dma_addr((ppgtt), (n)); \ 20119c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _UDW+1] = upper_32_bits(_addr); \ 20219c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _LDW+1] = lower_32_bits(_addr); \ 203*aee94f86SFrançois Tigeot } while (0) 20419c468b4SFrançois Tigeot 205*aee94f86SFrançois Tigeot #define ASSIGN_CTX_PML4(ppgtt, reg_state) do { \ 206352ff8bdSFrançois Tigeot reg_state[CTX_PDP0_UDW + 1] = upper_32_bits(px_dma(&ppgtt->pml4)); \ 207352ff8bdSFrançois Tigeot reg_state[CTX_PDP0_LDW + 1] = lower_32_bits(px_dma(&ppgtt->pml4)); \ 208*aee94f86SFrançois Tigeot } while (0) 209352ff8bdSFrançois Tigeot 2101b13d190SFrançois Tigeot enum { 2111b13d190SFrançois Tigeot ADVANCED_CONTEXT = 0, 212352ff8bdSFrançois Tigeot LEGACY_32B_CONTEXT, 2131b13d190SFrançois Tigeot ADVANCED_AD_CONTEXT, 2141b13d190SFrançois Tigeot LEGACY_64B_CONTEXT 2151b13d190SFrançois Tigeot }; 216352ff8bdSFrançois Tigeot #define GEN8_CTX_ADDRESSING_MODE_SHIFT 3 217352ff8bdSFrançois Tigeot #define GEN8_CTX_ADDRESSING_MODE(dev) (USES_FULL_48BIT_PPGTT(dev) ?\ 218352ff8bdSFrançois Tigeot LEGACY_64B_CONTEXT :\ 219352ff8bdSFrançois Tigeot LEGACY_32B_CONTEXT) 2201b13d190SFrançois Tigeot enum { 2211b13d190SFrançois Tigeot FAULT_AND_HANG = 0, 2221b13d190SFrançois Tigeot FAULT_AND_HALT, /* Debug only */ 2231b13d190SFrançois Tigeot FAULT_AND_STREAM, 2241b13d190SFrançois Tigeot FAULT_AND_CONTINUE /* Unsupported */ 2251b13d190SFrançois Tigeot }; 2261b13d190SFrançois Tigeot #define GEN8_CTX_ID_SHIFT 32 227a05eeebfSFrançois Tigeot #define CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT 0x17 2281b13d190SFrançois Tigeot 229a05eeebfSFrançois Tigeot static int intel_lr_context_pin(struct drm_i915_gem_request *rq); 230352ff8bdSFrançois Tigeot static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring, 231352ff8bdSFrançois Tigeot struct drm_i915_gem_object *default_ctx_obj); 232352ff8bdSFrançois Tigeot 2332c9916cdSFrançois Tigeot 2341b13d190SFrançois Tigeot /** 2351b13d190SFrançois Tigeot * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists 2361b13d190SFrançois Tigeot * @dev: DRM device. 2371b13d190SFrançois Tigeot * @enable_execlists: value of i915.enable_execlists module parameter. 2381b13d190SFrançois Tigeot * 2391b13d190SFrançois Tigeot * Only certain platforms support Execlists (the prerequisites being 2402c9916cdSFrançois Tigeot * support for Logical Ring Contexts and Aliasing PPGTT or better). 2411b13d190SFrançois Tigeot * 2421b13d190SFrançois Tigeot * Return: 1 if Execlists is supported and has to be enabled. 2431b13d190SFrançois Tigeot */ 2441b13d190SFrançois Tigeot int intel_sanitize_enable_execlists(struct drm_device *dev, int enable_execlists) 2451b13d190SFrançois Tigeot { 2461b13d190SFrançois Tigeot WARN_ON(i915.enable_ppgtt == -1); 2471b13d190SFrançois Tigeot 248352ff8bdSFrançois Tigeot /* On platforms with execlist available, vGPU will only 249352ff8bdSFrançois Tigeot * support execlist mode, no ring buffer mode. 250352ff8bdSFrançois Tigeot */ 251352ff8bdSFrançois Tigeot if (HAS_LOGICAL_RING_CONTEXTS(dev) && intel_vgpu_active(dev)) 252352ff8bdSFrançois Tigeot return 1; 253352ff8bdSFrançois Tigeot 2542c9916cdSFrançois Tigeot if (INTEL_INFO(dev)->gen >= 9) 2552c9916cdSFrançois Tigeot return 1; 2562c9916cdSFrançois Tigeot 2571b13d190SFrançois Tigeot if (enable_execlists == 0) 2581b13d190SFrançois Tigeot return 0; 2591b13d190SFrançois Tigeot 2601b13d190SFrançois Tigeot if (HAS_LOGICAL_RING_CONTEXTS(dev) && USES_PPGTT(dev) && 2611b13d190SFrançois Tigeot i915.use_mmio_flip >= 0) 2621b13d190SFrançois Tigeot return 1; 2631b13d190SFrançois Tigeot 2641b13d190SFrançois Tigeot return 0; 2651b13d190SFrançois Tigeot } 2661b13d190SFrançois Tigeot 2671b13d190SFrançois Tigeot /** 2681b13d190SFrançois Tigeot * intel_execlists_ctx_id() - get the Execlists Context ID 2691b13d190SFrançois Tigeot * @ctx_obj: Logical Ring Context backing object. 2701b13d190SFrançois Tigeot * 2711b13d190SFrançois Tigeot * Do not confuse with ctx->id! Unfortunately we have a name overload 2721b13d190SFrançois Tigeot * here: the old context ID we pass to userspace as a handler so that 2731b13d190SFrançois Tigeot * they can refer to a context, and the new context ID we pass to the 2741b13d190SFrançois Tigeot * ELSP so that the GPU can inform us of the context status via 2751b13d190SFrançois Tigeot * interrupts. 2761b13d190SFrançois Tigeot * 2771b13d190SFrançois Tigeot * Return: 20-bits globally unique context ID. 2781b13d190SFrançois Tigeot */ 2791b13d190SFrançois Tigeot u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj) 2801b13d190SFrançois Tigeot { 281352ff8bdSFrançois Tigeot u32 lrca = i915_gem_obj_ggtt_offset(ctx_obj) + 282352ff8bdSFrançois Tigeot LRC_PPHWSP_PN * PAGE_SIZE; 2831b13d190SFrançois Tigeot 2841b13d190SFrançois Tigeot /* LRCA is required to be 4K aligned so the more significant 20 bits 2851b13d190SFrançois Tigeot * are globally unique */ 2861b13d190SFrançois Tigeot return lrca >> 12; 2871b13d190SFrançois Tigeot } 2881b13d190SFrançois Tigeot 289352ff8bdSFrançois Tigeot static bool disable_lite_restore_wa(struct intel_engine_cs *ring) 2901b13d190SFrançois Tigeot { 291477eb7f9SFrançois Tigeot struct drm_device *dev = ring->dev; 292352ff8bdSFrançois Tigeot 293*aee94f86SFrançois Tigeot return (IS_SKL_REVID(dev, 0, SKL_REVID_B0) || 294*aee94f86SFrançois Tigeot IS_BXT_REVID(dev, 0, BXT_REVID_A1)) && 295352ff8bdSFrançois Tigeot (ring->id == VCS || ring->id == VCS2); 296352ff8bdSFrançois Tigeot } 297352ff8bdSFrançois Tigeot 298352ff8bdSFrançois Tigeot uint64_t intel_lr_context_descriptor(struct intel_context *ctx, 299352ff8bdSFrançois Tigeot struct intel_engine_cs *ring) 300352ff8bdSFrançois Tigeot { 301352ff8bdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ctx->engine[ring->id].state; 3021b13d190SFrançois Tigeot uint64_t desc; 303352ff8bdSFrançois Tigeot uint64_t lrca = i915_gem_obj_ggtt_offset(ctx_obj) + 304352ff8bdSFrançois Tigeot LRC_PPHWSP_PN * PAGE_SIZE; 3051b13d190SFrançois Tigeot 3061b13d190SFrançois Tigeot WARN_ON(lrca & 0xFFFFFFFF00000FFFULL); 3071b13d190SFrançois Tigeot 3081b13d190SFrançois Tigeot desc = GEN8_CTX_VALID; 309352ff8bdSFrançois Tigeot desc |= GEN8_CTX_ADDRESSING_MODE(dev) << GEN8_CTX_ADDRESSING_MODE_SHIFT; 31019c468b4SFrançois Tigeot if (IS_GEN8(ctx_obj->base.dev)) 3111b13d190SFrançois Tigeot desc |= GEN8_CTX_L3LLC_COHERENT; 3121b13d190SFrançois Tigeot desc |= GEN8_CTX_PRIVILEGE; 3131b13d190SFrançois Tigeot desc |= lrca; 3141b13d190SFrançois Tigeot desc |= (u64)intel_execlists_ctx_id(ctx_obj) << GEN8_CTX_ID_SHIFT; 3151b13d190SFrançois Tigeot 3161b13d190SFrançois Tigeot /* TODO: WaDisableLiteRestore when we start using semaphore 3171b13d190SFrançois Tigeot * signalling between Command Streamers */ 3181b13d190SFrançois Tigeot /* desc |= GEN8_CTX_FORCE_RESTORE; */ 3191b13d190SFrançois Tigeot 320477eb7f9SFrançois Tigeot /* WaEnableForceRestoreInCtxtDescForVCS:skl */ 321352ff8bdSFrançois Tigeot /* WaEnableForceRestoreInCtxtDescForVCS:bxt */ 322352ff8bdSFrançois Tigeot if (disable_lite_restore_wa(ring)) 323477eb7f9SFrançois Tigeot desc |= GEN8_CTX_FORCE_RESTORE; 324477eb7f9SFrançois Tigeot 3251b13d190SFrançois Tigeot return desc; 3261b13d190SFrançois Tigeot } 3271b13d190SFrançois Tigeot 328a05eeebfSFrançois Tigeot static void execlists_elsp_write(struct drm_i915_gem_request *rq0, 329a05eeebfSFrançois Tigeot struct drm_i915_gem_request *rq1) 3301b13d190SFrançois Tigeot { 331a05eeebfSFrançois Tigeot 332a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = rq0->ring; 3332c9916cdSFrançois Tigeot struct drm_device *dev = ring->dev; 3342c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 335a05eeebfSFrançois Tigeot uint64_t desc[2]; 3361b13d190SFrançois Tigeot 337a05eeebfSFrançois Tigeot if (rq1) { 338352ff8bdSFrançois Tigeot desc[1] = intel_lr_context_descriptor(rq1->ctx, rq1->ring); 339a05eeebfSFrançois Tigeot rq1->elsp_submitted++; 340a05eeebfSFrançois Tigeot } else { 341a05eeebfSFrançois Tigeot desc[1] = 0; 342a05eeebfSFrançois Tigeot } 3431b13d190SFrançois Tigeot 344352ff8bdSFrançois Tigeot desc[0] = intel_lr_context_descriptor(rq0->ctx, rq0->ring); 345a05eeebfSFrançois Tigeot rq0->elsp_submitted++; 3461b13d190SFrançois Tigeot 347a05eeebfSFrançois Tigeot /* You must always write both descriptors in the order below. */ 34819c468b4SFrançois Tigeot lockmgr(&dev_priv->uncore.lock, LK_EXCLUSIVE); 34919c468b4SFrançois Tigeot intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL); 350a05eeebfSFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), upper_32_bits(desc[1])); 351a05eeebfSFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), lower_32_bits(desc[1])); 3522c9916cdSFrançois Tigeot 353a05eeebfSFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), upper_32_bits(desc[0])); 3541b13d190SFrançois Tigeot /* The context is automatically loaded after the following */ 355a05eeebfSFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), lower_32_bits(desc[0])); 3561b13d190SFrançois Tigeot 357a05eeebfSFrançois Tigeot /* ELSP is a wo register, use another nearby reg for posting */ 358352ff8bdSFrançois Tigeot POSTING_READ_FW(RING_EXECLIST_STATUS_LO(ring)); 35919c468b4SFrançois Tigeot intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL); 36019c468b4SFrançois Tigeot lockmgr(&dev_priv->uncore.lock, LK_RELEASE); 3611b13d190SFrançois Tigeot } 3621b13d190SFrançois Tigeot 363a05eeebfSFrançois Tigeot static int execlists_update_context(struct drm_i915_gem_request *rq) 3641b13d190SFrançois Tigeot { 365a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = rq->ring; 366a05eeebfSFrançois Tigeot struct i915_hw_ppgtt *ppgtt = rq->ctx->ppgtt; 367a05eeebfSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = rq->ctx->engine[ring->id].state; 368a05eeebfSFrançois Tigeot struct drm_i915_gem_object *rb_obj = rq->ringbuf->obj; 3691b13d190SFrançois Tigeot struct vm_page *page; 3701b13d190SFrançois Tigeot uint32_t *reg_state; 3711b13d190SFrançois Tigeot 372a05eeebfSFrançois Tigeot BUG_ON(!ctx_obj); 373a05eeebfSFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(ctx_obj)); 374a05eeebfSFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(rb_obj)); 375a05eeebfSFrançois Tigeot 376*aee94f86SFrançois Tigeot page = i915_gem_object_get_dirty_page(ctx_obj, LRC_STATE_PN); 3771b13d190SFrançois Tigeot reg_state = kmap_atomic(page); 3781b13d190SFrançois Tigeot 379a05eeebfSFrançois Tigeot reg_state[CTX_RING_TAIL+1] = rq->tail; 380a05eeebfSFrançois Tigeot reg_state[CTX_RING_BUFFER_START+1] = i915_gem_obj_ggtt_offset(rb_obj); 3811b13d190SFrançois Tigeot 382352ff8bdSFrançois Tigeot if (ppgtt && !USES_FULL_48BIT_PPGTT(ppgtt->base.dev)) { 383352ff8bdSFrançois Tigeot /* True 32b PPGTT with dynamic page allocation: update PDP 384352ff8bdSFrançois Tigeot * registers and point the unallocated PDPs to scratch page. 385352ff8bdSFrançois Tigeot * PML4 is allocated during ppgtt init, so this is not needed 386352ff8bdSFrançois Tigeot * in 48-bit mode. 38719c468b4SFrançois Tigeot */ 38819c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 3); 38919c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 2); 39019c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 1); 39119c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 0); 39219c468b4SFrançois Tigeot } 39319c468b4SFrançois Tigeot 3941b13d190SFrançois Tigeot kunmap_atomic(reg_state); 3951b13d190SFrançois Tigeot 3961b13d190SFrançois Tigeot return 0; 3971b13d190SFrançois Tigeot } 3981b13d190SFrançois Tigeot 399a05eeebfSFrançois Tigeot static void execlists_submit_requests(struct drm_i915_gem_request *rq0, 400a05eeebfSFrançois Tigeot struct drm_i915_gem_request *rq1) 4011b13d190SFrançois Tigeot { 402a05eeebfSFrançois Tigeot execlists_update_context(rq0); 4031b13d190SFrançois Tigeot 404a05eeebfSFrançois Tigeot if (rq1) 405a05eeebfSFrançois Tigeot execlists_update_context(rq1); 4061b13d190SFrançois Tigeot 407a05eeebfSFrançois Tigeot execlists_elsp_write(rq0, rq1); 4081b13d190SFrançois Tigeot } 4091b13d190SFrançois Tigeot 4101b13d190SFrançois Tigeot static void execlists_context_unqueue(struct intel_engine_cs *ring) 4111b13d190SFrançois Tigeot { 4122c9916cdSFrançois Tigeot struct drm_i915_gem_request *req0 = NULL, *req1 = NULL; 4132c9916cdSFrançois Tigeot struct drm_i915_gem_request *cursor = NULL, *tmp = NULL; 4141b13d190SFrançois Tigeot 4151b13d190SFrançois Tigeot assert_spin_locked(&ring->execlist_lock); 4161b13d190SFrançois Tigeot 41719c468b4SFrançois Tigeot /* 41819c468b4SFrançois Tigeot * If irqs are not active generate a warning as batches that finish 41919c468b4SFrançois Tigeot * without the irqs may get lost and a GPU Hang may occur. 42019c468b4SFrançois Tigeot */ 42119c468b4SFrançois Tigeot WARN_ON(!intel_irqs_enabled(ring->dev->dev_private)); 42219c468b4SFrançois Tigeot 4231b13d190SFrançois Tigeot if (list_empty(&ring->execlist_queue)) 4241b13d190SFrançois Tigeot return; 4251b13d190SFrançois Tigeot 4261b13d190SFrançois Tigeot /* Try to read in pairs */ 4271b13d190SFrançois Tigeot list_for_each_entry_safe(cursor, tmp, &ring->execlist_queue, 4281b13d190SFrançois Tigeot execlist_link) { 4291b13d190SFrançois Tigeot if (!req0) { 4301b13d190SFrançois Tigeot req0 = cursor; 4311b13d190SFrançois Tigeot } else if (req0->ctx == cursor->ctx) { 4321b13d190SFrançois Tigeot /* Same ctx: ignore first request, as second request 4331b13d190SFrançois Tigeot * will update tail past first request's workload */ 4341b13d190SFrançois Tigeot cursor->elsp_submitted = req0->elsp_submitted; 4351b13d190SFrançois Tigeot list_del(&req0->execlist_link); 4362c9916cdSFrançois Tigeot list_add_tail(&req0->execlist_link, 4372c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 4381b13d190SFrançois Tigeot req0 = cursor; 4391b13d190SFrançois Tigeot } else { 4401b13d190SFrançois Tigeot req1 = cursor; 4411b13d190SFrançois Tigeot break; 4421b13d190SFrançois Tigeot } 4431b13d190SFrançois Tigeot } 4441b13d190SFrançois Tigeot 445477eb7f9SFrançois Tigeot if (IS_GEN8(ring->dev) || IS_GEN9(ring->dev)) { 446477eb7f9SFrançois Tigeot /* 447477eb7f9SFrançois Tigeot * WaIdleLiteRestore: make sure we never cause a lite 448477eb7f9SFrançois Tigeot * restore with HEAD==TAIL 449477eb7f9SFrançois Tigeot */ 45019c468b4SFrançois Tigeot if (req0->elsp_submitted) { 451477eb7f9SFrançois Tigeot /* 452477eb7f9SFrançois Tigeot * Apply the wa NOOPS to prevent ring:HEAD == req:TAIL 453477eb7f9SFrançois Tigeot * as we resubmit the request. See gen8_emit_request() 454477eb7f9SFrançois Tigeot * for where we prepare the padding after the end of the 455477eb7f9SFrançois Tigeot * request. 456477eb7f9SFrançois Tigeot */ 457477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf; 458477eb7f9SFrançois Tigeot 459477eb7f9SFrançois Tigeot ringbuf = req0->ctx->engine[ring->id].ringbuf; 460477eb7f9SFrançois Tigeot req0->tail += 8; 461477eb7f9SFrançois Tigeot req0->tail &= ringbuf->size - 1; 462477eb7f9SFrançois Tigeot } 463477eb7f9SFrançois Tigeot } 464477eb7f9SFrançois Tigeot 4651b13d190SFrançois Tigeot WARN_ON(req1 && req1->elsp_submitted); 4661b13d190SFrançois Tigeot 467a05eeebfSFrançois Tigeot execlists_submit_requests(req0, req1); 4681b13d190SFrançois Tigeot } 4691b13d190SFrançois Tigeot 4701b13d190SFrançois Tigeot static bool execlists_check_remove_request(struct intel_engine_cs *ring, 4711b13d190SFrançois Tigeot u32 request_id) 4721b13d190SFrançois Tigeot { 4732c9916cdSFrançois Tigeot struct drm_i915_gem_request *head_req; 4741b13d190SFrançois Tigeot 4751b13d190SFrançois Tigeot assert_spin_locked(&ring->execlist_lock); 4761b13d190SFrançois Tigeot 4771b13d190SFrançois Tigeot head_req = list_first_entry_or_null(&ring->execlist_queue, 4782c9916cdSFrançois Tigeot struct drm_i915_gem_request, 4791b13d190SFrançois Tigeot execlist_link); 4801b13d190SFrançois Tigeot 4811b13d190SFrançois Tigeot if (head_req != NULL) { 4821b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 4831b13d190SFrançois Tigeot head_req->ctx->engine[ring->id].state; 4841b13d190SFrançois Tigeot if (intel_execlists_ctx_id(ctx_obj) == request_id) { 4851b13d190SFrançois Tigeot WARN(head_req->elsp_submitted == 0, 4861b13d190SFrançois Tigeot "Never submitted head request\n"); 4871b13d190SFrançois Tigeot 4881b13d190SFrançois Tigeot if (--head_req->elsp_submitted <= 0) { 4891b13d190SFrançois Tigeot list_del(&head_req->execlist_link); 4902c9916cdSFrançois Tigeot list_add_tail(&head_req->execlist_link, 4912c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 4921b13d190SFrançois Tigeot return true; 4931b13d190SFrançois Tigeot } 4941b13d190SFrançois Tigeot } 4951b13d190SFrançois Tigeot } 4961b13d190SFrançois Tigeot 4971b13d190SFrançois Tigeot return false; 4981b13d190SFrançois Tigeot } 4991b13d190SFrançois Tigeot 5001b13d190SFrançois Tigeot /** 5012c9916cdSFrançois Tigeot * intel_lrc_irq_handler() - handle Context Switch interrupts 5021b13d190SFrançois Tigeot * @ring: Engine Command Streamer to handle. 5031b13d190SFrançois Tigeot * 5041b13d190SFrançois Tigeot * Check the unread Context Status Buffers and manage the submission of new 5051b13d190SFrançois Tigeot * contexts to the ELSP accordingly. 5061b13d190SFrançois Tigeot */ 5072c9916cdSFrançois Tigeot void intel_lrc_irq_handler(struct intel_engine_cs *ring) 5081b13d190SFrançois Tigeot { 5091b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 5101b13d190SFrançois Tigeot u32 status_pointer; 5111b13d190SFrançois Tigeot u8 read_pointer; 5121b13d190SFrançois Tigeot u8 write_pointer; 513352ff8bdSFrançois Tigeot u32 status = 0; 5141b13d190SFrançois Tigeot u32 status_id; 5151b13d190SFrançois Tigeot u32 submit_contexts = 0; 5161b13d190SFrançois Tigeot 5171b13d190SFrançois Tigeot status_pointer = I915_READ(RING_CONTEXT_STATUS_PTR(ring)); 5181b13d190SFrançois Tigeot 5191b13d190SFrançois Tigeot read_pointer = ring->next_context_status_buffer; 520a05eeebfSFrançois Tigeot write_pointer = status_pointer & GEN8_CSB_PTR_MASK; 5211b13d190SFrançois Tigeot if (read_pointer > write_pointer) 522a05eeebfSFrançois Tigeot write_pointer += GEN8_CSB_ENTRIES; 5231b13d190SFrançois Tigeot 5241b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_EXCLUSIVE); 5251b13d190SFrançois Tigeot 5261b13d190SFrançois Tigeot while (read_pointer < write_pointer) { 5271b13d190SFrançois Tigeot read_pointer++; 528352ff8bdSFrançois Tigeot status = I915_READ(RING_CONTEXT_STATUS_BUF_LO(ring, read_pointer % GEN8_CSB_ENTRIES)); 529352ff8bdSFrançois Tigeot status_id = I915_READ(RING_CONTEXT_STATUS_BUF_HI(ring, read_pointer % GEN8_CSB_ENTRIES)); 530a05eeebfSFrançois Tigeot 531a05eeebfSFrançois Tigeot if (status & GEN8_CTX_STATUS_IDLE_ACTIVE) 532a05eeebfSFrançois Tigeot continue; 5331b13d190SFrançois Tigeot 5341b13d190SFrançois Tigeot if (status & GEN8_CTX_STATUS_PREEMPTED) { 5351b13d190SFrançois Tigeot if (status & GEN8_CTX_STATUS_LITE_RESTORE) { 5361b13d190SFrançois Tigeot if (execlists_check_remove_request(ring, status_id)) 5371b13d190SFrançois Tigeot WARN(1, "Lite Restored request removed from queue\n"); 5381b13d190SFrançois Tigeot } else 5391b13d190SFrançois Tigeot WARN(1, "Preemption without Lite Restore\n"); 5401b13d190SFrançois Tigeot } 5411b13d190SFrançois Tigeot 5421b13d190SFrançois Tigeot if ((status & GEN8_CTX_STATUS_ACTIVE_IDLE) || 5431b13d190SFrançois Tigeot (status & GEN8_CTX_STATUS_ELEMENT_SWITCH)) { 5441b13d190SFrançois Tigeot if (execlists_check_remove_request(ring, status_id)) 5451b13d190SFrançois Tigeot submit_contexts++; 5461b13d190SFrançois Tigeot } 5471b13d190SFrançois Tigeot } 5481b13d190SFrançois Tigeot 549352ff8bdSFrançois Tigeot if (disable_lite_restore_wa(ring)) { 550352ff8bdSFrançois Tigeot /* Prevent a ctx to preempt itself */ 551352ff8bdSFrançois Tigeot if ((status & GEN8_CTX_STATUS_ACTIVE_IDLE) && 552352ff8bdSFrançois Tigeot (submit_contexts != 0)) 5531b13d190SFrançois Tigeot execlists_context_unqueue(ring); 554352ff8bdSFrançois Tigeot } else if (submit_contexts != 0) { 555352ff8bdSFrançois Tigeot execlists_context_unqueue(ring); 556352ff8bdSFrançois Tigeot } 5571b13d190SFrançois Tigeot 5581b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_RELEASE); 5591b13d190SFrançois Tigeot 5601b13d190SFrançois Tigeot WARN(submit_contexts > 2, "More than two context complete events?\n"); 561a05eeebfSFrançois Tigeot ring->next_context_status_buffer = write_pointer % GEN8_CSB_ENTRIES; 5621b13d190SFrançois Tigeot 5631b13d190SFrançois Tigeot I915_WRITE(RING_CONTEXT_STATUS_PTR(ring), 564a05eeebfSFrançois Tigeot _MASKED_FIELD(GEN8_CSB_PTR_MASK << 8, 565a05eeebfSFrançois Tigeot ((u32)ring->next_context_status_buffer & 566a05eeebfSFrançois Tigeot GEN8_CSB_PTR_MASK) << 8)); 5671b13d190SFrançois Tigeot } 5681b13d190SFrançois Tigeot 569a05eeebfSFrançois Tigeot static int execlists_context_queue(struct drm_i915_gem_request *request) 5701b13d190SFrançois Tigeot { 571a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = request->ring; 5722c9916cdSFrançois Tigeot struct drm_i915_gem_request *cursor; 5731b13d190SFrançois Tigeot int num_elements = 0; 5741b13d190SFrançois Tigeot 575a05eeebfSFrançois Tigeot if (request->ctx != ring->default_context) 576a05eeebfSFrançois Tigeot intel_lr_context_pin(request); 5772c9916cdSFrançois Tigeot 5782c9916cdSFrançois Tigeot i915_gem_request_reference(request); 579a05eeebfSFrançois Tigeot 5805e269720SFrançois Tigeot spin_lock_irq(&ring->execlist_lock); 5811b13d190SFrançois Tigeot 5821b13d190SFrançois Tigeot list_for_each_entry(cursor, &ring->execlist_queue, execlist_link) 5831b13d190SFrançois Tigeot if (++num_elements > 2) 5841b13d190SFrançois Tigeot break; 5851b13d190SFrançois Tigeot 5861b13d190SFrançois Tigeot if (num_elements > 2) { 5872c9916cdSFrançois Tigeot struct drm_i915_gem_request *tail_req; 5881b13d190SFrançois Tigeot 5891b13d190SFrançois Tigeot tail_req = list_last_entry(&ring->execlist_queue, 5902c9916cdSFrançois Tigeot struct drm_i915_gem_request, 5911b13d190SFrançois Tigeot execlist_link); 5921b13d190SFrançois Tigeot 593a05eeebfSFrançois Tigeot if (request->ctx == tail_req->ctx) { 5941b13d190SFrançois Tigeot WARN(tail_req->elsp_submitted != 0, 5951b13d190SFrançois Tigeot "More than 2 already-submitted reqs queued\n"); 5961b13d190SFrançois Tigeot list_del(&tail_req->execlist_link); 5972c9916cdSFrançois Tigeot list_add_tail(&tail_req->execlist_link, 5982c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 5991b13d190SFrançois Tigeot } 6001b13d190SFrançois Tigeot } 6011b13d190SFrançois Tigeot 6022c9916cdSFrançois Tigeot list_add_tail(&request->execlist_link, &ring->execlist_queue); 6031b13d190SFrançois Tigeot if (num_elements == 0) 6041b13d190SFrançois Tigeot execlists_context_unqueue(ring); 6051b13d190SFrançois Tigeot 6065e269720SFrançois Tigeot spin_unlock_irq(&ring->execlist_lock); 6071b13d190SFrançois Tigeot 6081b13d190SFrançois Tigeot return 0; 6091b13d190SFrançois Tigeot } 6101b13d190SFrançois Tigeot 611a05eeebfSFrançois Tigeot static int logical_ring_invalidate_all_caches(struct drm_i915_gem_request *req) 6121b13d190SFrançois Tigeot { 613a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = req->ring; 6141b13d190SFrançois Tigeot uint32_t flush_domains; 6151b13d190SFrançois Tigeot int ret; 6161b13d190SFrançois Tigeot 6171b13d190SFrançois Tigeot flush_domains = 0; 6181b13d190SFrançois Tigeot if (ring->gpu_caches_dirty) 6191b13d190SFrançois Tigeot flush_domains = I915_GEM_GPU_DOMAINS; 6201b13d190SFrançois Tigeot 621a05eeebfSFrançois Tigeot ret = ring->emit_flush(req, I915_GEM_GPU_DOMAINS, flush_domains); 6221b13d190SFrançois Tigeot if (ret) 6231b13d190SFrançois Tigeot return ret; 6241b13d190SFrançois Tigeot 6251b13d190SFrançois Tigeot ring->gpu_caches_dirty = false; 6261b13d190SFrançois Tigeot return 0; 6271b13d190SFrançois Tigeot } 6281b13d190SFrançois Tigeot 629a05eeebfSFrançois Tigeot static int execlists_move_to_gpu(struct drm_i915_gem_request *req, 6301b13d190SFrançois Tigeot struct list_head *vmas) 6311b13d190SFrançois Tigeot { 632a05eeebfSFrançois Tigeot const unsigned other_rings = ~intel_ring_flag(req->ring); 6331b13d190SFrançois Tigeot struct i915_vma *vma; 6341b13d190SFrançois Tigeot uint32_t flush_domains = 0; 6351b13d190SFrançois Tigeot bool flush_chipset = false; 6361b13d190SFrançois Tigeot int ret; 6371b13d190SFrançois Tigeot 6381b13d190SFrançois Tigeot list_for_each_entry(vma, vmas, exec_list) { 6391b13d190SFrançois Tigeot struct drm_i915_gem_object *obj = vma->obj; 6401b13d190SFrançois Tigeot 64119c468b4SFrançois Tigeot if (obj->active & other_rings) { 642a05eeebfSFrançois Tigeot ret = i915_gem_object_sync(obj, req->ring, &req); 6431b13d190SFrançois Tigeot if (ret) 6441b13d190SFrançois Tigeot return ret; 64519c468b4SFrançois Tigeot } 6461b13d190SFrançois Tigeot 6471b13d190SFrançois Tigeot if (obj->base.write_domain & I915_GEM_DOMAIN_CPU) 6481b13d190SFrançois Tigeot flush_chipset |= i915_gem_clflush_object(obj, false); 6491b13d190SFrançois Tigeot 6501b13d190SFrançois Tigeot flush_domains |= obj->base.write_domain; 6511b13d190SFrançois Tigeot } 6521b13d190SFrançois Tigeot 6531b13d190SFrançois Tigeot if (flush_domains & I915_GEM_DOMAIN_GTT) 6541b13d190SFrançois Tigeot wmb(); 6551b13d190SFrançois Tigeot 6561b13d190SFrançois Tigeot /* Unconditionally invalidate gpu caches and ensure that we do flush 6571b13d190SFrançois Tigeot * any residual writes from the previous batch. 6581b13d190SFrançois Tigeot */ 659a05eeebfSFrançois Tigeot return logical_ring_invalidate_all_caches(req); 6601b13d190SFrançois Tigeot } 6611b13d190SFrançois Tigeot 662a05eeebfSFrançois Tigeot int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request) 66319c468b4SFrançois Tigeot { 66419c468b4SFrançois Tigeot int ret; 66519c468b4SFrançois Tigeot 666a05eeebfSFrançois Tigeot request->ringbuf = request->ctx->engine[request->ring->id].ringbuf; 667a05eeebfSFrançois Tigeot 668a05eeebfSFrançois Tigeot if (request->ctx != request->ring->default_context) { 669a05eeebfSFrançois Tigeot ret = intel_lr_context_pin(request); 67019c468b4SFrançois Tigeot if (ret) 67119c468b4SFrançois Tigeot return ret; 67219c468b4SFrançois Tigeot } 67319c468b4SFrançois Tigeot 67419c468b4SFrançois Tigeot return 0; 67519c468b4SFrançois Tigeot } 67619c468b4SFrançois Tigeot 677a05eeebfSFrançois Tigeot static int logical_ring_wait_for_space(struct drm_i915_gem_request *req, 67819c468b4SFrançois Tigeot int bytes) 67919c468b4SFrançois Tigeot { 680a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 681a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = req->ring; 682a05eeebfSFrançois Tigeot struct drm_i915_gem_request *target; 68319c468b4SFrançois Tigeot unsigned space; 68419c468b4SFrançois Tigeot int ret; 68519c468b4SFrançois Tigeot 68619c468b4SFrançois Tigeot if (intel_ring_space(ringbuf) >= bytes) 68719c468b4SFrançois Tigeot return 0; 68819c468b4SFrançois Tigeot 689a05eeebfSFrançois Tigeot /* The whole point of reserving space is to not wait! */ 690a05eeebfSFrançois Tigeot WARN_ON(ringbuf->reserved_in_use); 691a05eeebfSFrançois Tigeot 692a05eeebfSFrançois Tigeot list_for_each_entry(target, &ring->request_list, list) { 69319c468b4SFrançois Tigeot /* 69419c468b4SFrançois Tigeot * The request queue is per-engine, so can contain requests 69519c468b4SFrançois Tigeot * from multiple ringbuffers. Here, we must ignore any that 69619c468b4SFrançois Tigeot * aren't from the ringbuffer we're considering. 69719c468b4SFrançois Tigeot */ 698a05eeebfSFrançois Tigeot if (target->ringbuf != ringbuf) 69919c468b4SFrançois Tigeot continue; 70019c468b4SFrançois Tigeot 70119c468b4SFrançois Tigeot /* Would completion of this request free enough space? */ 702a05eeebfSFrançois Tigeot space = __intel_ring_space(target->postfix, ringbuf->tail, 70319c468b4SFrançois Tigeot ringbuf->size); 70419c468b4SFrançois Tigeot if (space >= bytes) 70519c468b4SFrançois Tigeot break; 70619c468b4SFrançois Tigeot } 70719c468b4SFrançois Tigeot 708a05eeebfSFrançois Tigeot if (WARN_ON(&target->list == &ring->request_list)) 70919c468b4SFrançois Tigeot return -ENOSPC; 71019c468b4SFrançois Tigeot 711a05eeebfSFrançois Tigeot ret = i915_wait_request(target); 71219c468b4SFrançois Tigeot if (ret) 71319c468b4SFrançois Tigeot return ret; 71419c468b4SFrançois Tigeot 71519c468b4SFrançois Tigeot ringbuf->space = space; 71619c468b4SFrançois Tigeot return 0; 71719c468b4SFrançois Tigeot } 71819c468b4SFrançois Tigeot 71919c468b4SFrançois Tigeot /* 72019c468b4SFrançois Tigeot * intel_logical_ring_advance_and_submit() - advance the tail and submit the workload 721a05eeebfSFrançois Tigeot * @request: Request to advance the logical ringbuffer of. 72219c468b4SFrançois Tigeot * 72319c468b4SFrançois Tigeot * The tail is updated in our logical ringbuffer struct, not in the actual context. What 72419c468b4SFrançois Tigeot * really happens during submission is that the context and current tail will be placed 72519c468b4SFrançois Tigeot * on a queue waiting for the ELSP to be ready to accept a new context submission. At that 72619c468b4SFrançois Tigeot * point, the tail *inside* the context is updated and the ELSP written to. 72719c468b4SFrançois Tigeot */ 72819c468b4SFrançois Tigeot static void 729a05eeebfSFrançois Tigeot intel_logical_ring_advance_and_submit(struct drm_i915_gem_request *request) 73019c468b4SFrançois Tigeot { 731a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = request->ring; 7328cd49a21SFrançois Tigeot struct drm_i915_private *dev_priv = request->i915; 73319c468b4SFrançois Tigeot 734a05eeebfSFrançois Tigeot intel_logical_ring_advance(request->ringbuf); 73519c468b4SFrançois Tigeot 736352ff8bdSFrançois Tigeot request->tail = request->ringbuf->tail; 737352ff8bdSFrançois Tigeot 73819c468b4SFrançois Tigeot if (intel_ring_stopped(ring)) 73919c468b4SFrançois Tigeot return; 74019c468b4SFrançois Tigeot 741352ff8bdSFrançois Tigeot if (dev_priv->guc.execbuf_client) 742352ff8bdSFrançois Tigeot i915_guc_submit(dev_priv->guc.execbuf_client, request); 743352ff8bdSFrançois Tigeot else 744a05eeebfSFrançois Tigeot execlists_context_queue(request); 74519c468b4SFrançois Tigeot } 74619c468b4SFrançois Tigeot 747a05eeebfSFrançois Tigeot static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf) 74819c468b4SFrançois Tigeot { 74919c468b4SFrançois Tigeot uint32_t __iomem *virt; 75019c468b4SFrançois Tigeot int rem = ringbuf->size - ringbuf->tail; 75119c468b4SFrançois Tigeot 75219c468b4SFrançois Tigeot virt = (uint32_t *)(ringbuf->virtual_start + ringbuf->tail); 75319c468b4SFrançois Tigeot rem /= 4; 75419c468b4SFrançois Tigeot while (rem--) 75519c468b4SFrançois Tigeot iowrite32(MI_NOOP, virt++); 75619c468b4SFrançois Tigeot 75719c468b4SFrançois Tigeot ringbuf->tail = 0; 75819c468b4SFrançois Tigeot intel_ring_update_space(ringbuf); 75919c468b4SFrançois Tigeot } 76019c468b4SFrançois Tigeot 761a05eeebfSFrançois Tigeot static int logical_ring_prepare(struct drm_i915_gem_request *req, int bytes) 76219c468b4SFrançois Tigeot { 763a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 764a05eeebfSFrançois Tigeot int remain_usable = ringbuf->effective_size - ringbuf->tail; 765a05eeebfSFrançois Tigeot int remain_actual = ringbuf->size - ringbuf->tail; 766a05eeebfSFrançois Tigeot int ret, total_bytes, wait_bytes = 0; 767a05eeebfSFrançois Tigeot bool need_wrap = false; 76819c468b4SFrançois Tigeot 769a05eeebfSFrançois Tigeot if (ringbuf->reserved_in_use) 770a05eeebfSFrançois Tigeot total_bytes = bytes; 771a05eeebfSFrançois Tigeot else 772a05eeebfSFrançois Tigeot total_bytes = bytes + ringbuf->reserved_size; 773a05eeebfSFrançois Tigeot 774a05eeebfSFrançois Tigeot if (unlikely(bytes > remain_usable)) { 775a05eeebfSFrançois Tigeot /* 776a05eeebfSFrançois Tigeot * Not enough space for the basic request. So need to flush 777a05eeebfSFrançois Tigeot * out the remainder and then wait for base + reserved. 778a05eeebfSFrançois Tigeot */ 779a05eeebfSFrançois Tigeot wait_bytes = remain_actual + total_bytes; 780a05eeebfSFrançois Tigeot need_wrap = true; 781a05eeebfSFrançois Tigeot } else { 782a05eeebfSFrançois Tigeot if (unlikely(total_bytes > remain_usable)) { 783a05eeebfSFrançois Tigeot /* 784a05eeebfSFrançois Tigeot * The base request will fit but the reserved space 785a05eeebfSFrançois Tigeot * falls off the end. So only need to to wait for the 786a05eeebfSFrançois Tigeot * reserved size after flushing out the remainder. 787a05eeebfSFrançois Tigeot */ 788a05eeebfSFrançois Tigeot wait_bytes = remain_actual + ringbuf->reserved_size; 789a05eeebfSFrançois Tigeot need_wrap = true; 790a05eeebfSFrançois Tigeot } else if (total_bytes > ringbuf->space) { 791a05eeebfSFrançois Tigeot /* No wrapping required, just waiting. */ 792a05eeebfSFrançois Tigeot wait_bytes = total_bytes; 793a05eeebfSFrançois Tigeot } 79419c468b4SFrançois Tigeot } 79519c468b4SFrançois Tigeot 796a05eeebfSFrançois Tigeot if (wait_bytes) { 797a05eeebfSFrançois Tigeot ret = logical_ring_wait_for_space(req, wait_bytes); 79819c468b4SFrançois Tigeot if (unlikely(ret)) 79919c468b4SFrançois Tigeot return ret; 800a05eeebfSFrançois Tigeot 801a05eeebfSFrançois Tigeot if (need_wrap) 802a05eeebfSFrançois Tigeot __wrap_ring_buffer(ringbuf); 80319c468b4SFrançois Tigeot } 80419c468b4SFrançois Tigeot 80519c468b4SFrançois Tigeot return 0; 80619c468b4SFrançois Tigeot } 80719c468b4SFrançois Tigeot 80819c468b4SFrançois Tigeot /** 80919c468b4SFrançois Tigeot * intel_logical_ring_begin() - prepare the logical ringbuffer to accept some commands 81019c468b4SFrançois Tigeot * 811352ff8bdSFrançois Tigeot * @req: The request to start some new work for 81219c468b4SFrançois Tigeot * @num_dwords: number of DWORDs that we plan to write to the ringbuffer. 81319c468b4SFrançois Tigeot * 81419c468b4SFrançois Tigeot * The ringbuffer might not be ready to accept the commands right away (maybe it needs to 81519c468b4SFrançois Tigeot * be wrapped, or wait a bit for the tail to be updated). This function takes care of that 81619c468b4SFrançois Tigeot * and also preallocates a request (every workload submission is still mediated through 81719c468b4SFrançois Tigeot * requests, same as it did with legacy ringbuffer submission). 81819c468b4SFrançois Tigeot * 81919c468b4SFrançois Tigeot * Return: non-zero if the ringbuffer is not ready to be written to. 82019c468b4SFrançois Tigeot */ 821a05eeebfSFrançois Tigeot int intel_logical_ring_begin(struct drm_i915_gem_request *req, int num_dwords) 82219c468b4SFrançois Tigeot { 823a05eeebfSFrançois Tigeot struct drm_i915_private *dev_priv; 82419c468b4SFrançois Tigeot int ret; 82519c468b4SFrançois Tigeot 826a05eeebfSFrançois Tigeot WARN_ON(req == NULL); 827a05eeebfSFrançois Tigeot dev_priv = req->ring->dev->dev_private; 828a05eeebfSFrançois Tigeot 82919c468b4SFrançois Tigeot ret = i915_gem_check_wedge(&dev_priv->gpu_error, 83019c468b4SFrançois Tigeot dev_priv->mm.interruptible); 83119c468b4SFrançois Tigeot if (ret) 83219c468b4SFrançois Tigeot return ret; 83319c468b4SFrançois Tigeot 834a05eeebfSFrançois Tigeot ret = logical_ring_prepare(req, num_dwords * sizeof(uint32_t)); 83519c468b4SFrançois Tigeot if (ret) 83619c468b4SFrançois Tigeot return ret; 83719c468b4SFrançois Tigeot 838a05eeebfSFrançois Tigeot req->ringbuf->space -= num_dwords * sizeof(uint32_t); 83919c468b4SFrançois Tigeot return 0; 84019c468b4SFrançois Tigeot } 84119c468b4SFrançois Tigeot 842a05eeebfSFrançois Tigeot int intel_logical_ring_reserve_space(struct drm_i915_gem_request *request) 843a05eeebfSFrançois Tigeot { 844a05eeebfSFrançois Tigeot /* 845a05eeebfSFrançois Tigeot * The first call merely notes the reserve request and is common for 846a05eeebfSFrançois Tigeot * all back ends. The subsequent localised _begin() call actually 847a05eeebfSFrançois Tigeot * ensures that the reservation is available. Without the begin, if 848a05eeebfSFrançois Tigeot * the request creator immediately submitted the request without 849a05eeebfSFrançois Tigeot * adding any commands to it then there might not actually be 850a05eeebfSFrançois Tigeot * sufficient room for the submission commands. 851a05eeebfSFrançois Tigeot */ 852a05eeebfSFrançois Tigeot intel_ring_reserved_space_reserve(request->ringbuf, MIN_SPACE_FOR_ADD_REQUEST); 853a05eeebfSFrançois Tigeot 854a05eeebfSFrançois Tigeot return intel_logical_ring_begin(request, 0); 855a05eeebfSFrançois Tigeot } 856a05eeebfSFrançois Tigeot 8571b13d190SFrançois Tigeot /** 8581b13d190SFrançois Tigeot * execlists_submission() - submit a batchbuffer for execution, Execlists style 8591b13d190SFrançois Tigeot * @dev: DRM device. 8601b13d190SFrançois Tigeot * @file: DRM file. 8611b13d190SFrançois Tigeot * @ring: Engine Command Streamer to submit to. 8621b13d190SFrançois Tigeot * @ctx: Context to employ for this submission. 8631b13d190SFrançois Tigeot * @args: execbuffer call arguments. 8641b13d190SFrançois Tigeot * @vmas: list of vmas. 8651b13d190SFrançois Tigeot * @batch_obj: the batchbuffer to submit. 8661b13d190SFrançois Tigeot * @exec_start: batchbuffer start virtual address pointer. 867477eb7f9SFrançois Tigeot * @dispatch_flags: translated execbuffer call flags. 8681b13d190SFrançois Tigeot * 8691b13d190SFrançois Tigeot * This is the evil twin version of i915_gem_ringbuffer_submission. It abstracts 8701b13d190SFrançois Tigeot * away the submission details of the execbuffer ioctl call. 8711b13d190SFrançois Tigeot * 8721b13d190SFrançois Tigeot * Return: non-zero if the submission fails. 8731b13d190SFrançois Tigeot */ 874a05eeebfSFrançois Tigeot int intel_execlists_submission(struct i915_execbuffer_params *params, 8751b13d190SFrançois Tigeot struct drm_i915_gem_execbuffer2 *args, 876a05eeebfSFrançois Tigeot struct list_head *vmas) 8771b13d190SFrançois Tigeot { 878a05eeebfSFrançois Tigeot struct drm_device *dev = params->dev; 879a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = params->ring; 8801b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 881a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf; 882a05eeebfSFrançois Tigeot u64 exec_start; 8831b13d190SFrançois Tigeot int instp_mode; 8841b13d190SFrançois Tigeot u32 instp_mask; 8851b13d190SFrançois Tigeot int ret; 8861b13d190SFrançois Tigeot 8871b13d190SFrançois Tigeot instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK; 8881b13d190SFrançois Tigeot instp_mask = I915_EXEC_CONSTANTS_MASK; 8891b13d190SFrançois Tigeot switch (instp_mode) { 8901b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_GENERAL: 8911b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_ABSOLUTE: 8921b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_SURFACE: 8931b13d190SFrançois Tigeot if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) { 8941b13d190SFrançois Tigeot DRM_DEBUG("non-0 rel constants mode on non-RCS\n"); 8951b13d190SFrançois Tigeot return -EINVAL; 8961b13d190SFrançois Tigeot } 8971b13d190SFrançois Tigeot 8981b13d190SFrançois Tigeot if (instp_mode != dev_priv->relative_constants_mode) { 8991b13d190SFrançois Tigeot if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) { 9001b13d190SFrançois Tigeot DRM_DEBUG("rel surface constants mode invalid on gen5+\n"); 9011b13d190SFrançois Tigeot return -EINVAL; 9021b13d190SFrançois Tigeot } 9031b13d190SFrançois Tigeot 9041b13d190SFrançois Tigeot /* The HW changed the meaning on this bit on gen6 */ 9051b13d190SFrançois Tigeot instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE; 9061b13d190SFrançois Tigeot } 9071b13d190SFrançois Tigeot break; 9081b13d190SFrançois Tigeot default: 9091b13d190SFrançois Tigeot DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode); 9101b13d190SFrançois Tigeot return -EINVAL; 9111b13d190SFrançois Tigeot } 9121b13d190SFrançois Tigeot 9131b13d190SFrançois Tigeot if (args->flags & I915_EXEC_GEN7_SOL_RESET) { 9141b13d190SFrançois Tigeot DRM_DEBUG("sol reset is gen7 only\n"); 9151b13d190SFrançois Tigeot return -EINVAL; 9161b13d190SFrançois Tigeot } 9171b13d190SFrançois Tigeot 918a05eeebfSFrançois Tigeot ret = execlists_move_to_gpu(params->request, vmas); 9191b13d190SFrançois Tigeot if (ret) 9201b13d190SFrançois Tigeot return ret; 9211b13d190SFrançois Tigeot 9221b13d190SFrançois Tigeot if (ring == &dev_priv->ring[RCS] && 9231b13d190SFrançois Tigeot instp_mode != dev_priv->relative_constants_mode) { 924a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(params->request, 4); 9251b13d190SFrançois Tigeot if (ret) 9261b13d190SFrançois Tigeot return ret; 9271b13d190SFrançois Tigeot 9281b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 9291b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1)); 930*aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, INSTPM); 9311b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode); 9321b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 9331b13d190SFrançois Tigeot 9341b13d190SFrançois Tigeot dev_priv->relative_constants_mode = instp_mode; 9351b13d190SFrançois Tigeot } 9361b13d190SFrançois Tigeot 937a05eeebfSFrançois Tigeot exec_start = params->batch_obj_vm_offset + 938a05eeebfSFrançois Tigeot args->batch_start_offset; 939a05eeebfSFrançois Tigeot 940a05eeebfSFrançois Tigeot ret = ring->emit_bb_start(params->request, exec_start, params->dispatch_flags); 9411b13d190SFrançois Tigeot if (ret) 9421b13d190SFrançois Tigeot return ret; 9431b13d190SFrançois Tigeot 944a05eeebfSFrançois Tigeot trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags); 945477eb7f9SFrançois Tigeot 946a05eeebfSFrançois Tigeot i915_gem_execbuffer_move_to_active(vmas, params->request); 947a05eeebfSFrançois Tigeot i915_gem_execbuffer_retire_commands(params); 9481b13d190SFrançois Tigeot 9491b13d190SFrançois Tigeot return 0; 9501b13d190SFrançois Tigeot } 9511b13d190SFrançois Tigeot 9522c9916cdSFrançois Tigeot void intel_execlists_retire_requests(struct intel_engine_cs *ring) 9532c9916cdSFrançois Tigeot { 9542c9916cdSFrançois Tigeot struct drm_i915_gem_request *req, *tmp; 9552c9916cdSFrançois Tigeot struct list_head retired_list; 9562c9916cdSFrançois Tigeot 9572c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 9582c9916cdSFrançois Tigeot if (list_empty(&ring->execlist_retired_req_list)) 9592c9916cdSFrançois Tigeot return; 9602c9916cdSFrançois Tigeot 9612c9916cdSFrançois Tigeot INIT_LIST_HEAD(&retired_list); 9625e269720SFrançois Tigeot spin_lock_irq(&ring->execlist_lock); 9632c9916cdSFrançois Tigeot list_replace_init(&ring->execlist_retired_req_list, &retired_list); 9645e269720SFrançois Tigeot spin_unlock_irq(&ring->execlist_lock); 9652c9916cdSFrançois Tigeot 9662c9916cdSFrançois Tigeot list_for_each_entry_safe(req, tmp, &retired_list, execlist_link) { 9672c9916cdSFrançois Tigeot struct intel_context *ctx = req->ctx; 9682c9916cdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 9692c9916cdSFrançois Tigeot ctx->engine[ring->id].state; 9702c9916cdSFrançois Tigeot 9712c9916cdSFrançois Tigeot if (ctx_obj && (ctx != ring->default_context)) 972a05eeebfSFrançois Tigeot intel_lr_context_unpin(req); 9732c9916cdSFrançois Tigeot list_del(&req->execlist_link); 9742c9916cdSFrançois Tigeot i915_gem_request_unreference(req); 9752c9916cdSFrançois Tigeot } 9762c9916cdSFrançois Tigeot } 9772c9916cdSFrançois Tigeot 9781b13d190SFrançois Tigeot void intel_logical_ring_stop(struct intel_engine_cs *ring) 9791b13d190SFrançois Tigeot { 9801b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 9811b13d190SFrançois Tigeot int ret; 9821b13d190SFrançois Tigeot 9831b13d190SFrançois Tigeot if (!intel_ring_initialized(ring)) 9841b13d190SFrançois Tigeot return; 9851b13d190SFrançois Tigeot 9861b13d190SFrançois Tigeot ret = intel_ring_idle(ring); 9871b13d190SFrançois Tigeot if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error)) 9881b13d190SFrançois Tigeot DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n", 9891b13d190SFrançois Tigeot ring->name, ret); 9901b13d190SFrançois Tigeot 9911b13d190SFrançois Tigeot /* TODO: Is this correct with Execlists enabled? */ 9921b13d190SFrançois Tigeot I915_WRITE_MODE(ring, _MASKED_BIT_ENABLE(STOP_RING)); 9931b13d190SFrançois Tigeot if (wait_for_atomic((I915_READ_MODE(ring) & MODE_IDLE) != 0, 1000)) { 9941b13d190SFrançois Tigeot DRM_ERROR("%s :timed out trying to stop ring\n", ring->name); 9951b13d190SFrançois Tigeot return; 9961b13d190SFrançois Tigeot } 9971b13d190SFrançois Tigeot I915_WRITE_MODE(ring, _MASKED_BIT_DISABLE(STOP_RING)); 9981b13d190SFrançois Tigeot } 9991b13d190SFrançois Tigeot 1000a05eeebfSFrançois Tigeot int logical_ring_flush_all_caches(struct drm_i915_gem_request *req) 10011b13d190SFrançois Tigeot { 1002a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = req->ring; 10031b13d190SFrançois Tigeot int ret; 10041b13d190SFrançois Tigeot 10051b13d190SFrançois Tigeot if (!ring->gpu_caches_dirty) 10061b13d190SFrançois Tigeot return 0; 10071b13d190SFrançois Tigeot 1008a05eeebfSFrançois Tigeot ret = ring->emit_flush(req, 0, I915_GEM_GPU_DOMAINS); 10091b13d190SFrançois Tigeot if (ret) 10101b13d190SFrançois Tigeot return ret; 10111b13d190SFrançois Tigeot 10121b13d190SFrançois Tigeot ring->gpu_caches_dirty = false; 10131b13d190SFrançois Tigeot return 0; 10141b13d190SFrançois Tigeot } 10151b13d190SFrançois Tigeot 1016352ff8bdSFrançois Tigeot static int intel_lr_context_do_pin(struct intel_engine_cs *ring, 1017352ff8bdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj, 1018352ff8bdSFrançois Tigeot struct intel_ringbuffer *ringbuf) 10191b13d190SFrançois Tigeot { 1020352ff8bdSFrançois Tigeot struct drm_device *dev = ring->dev; 1021352ff8bdSFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 10222c9916cdSFrançois Tigeot int ret = 0; 10232c9916cdSFrançois Tigeot 10242c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 1025352ff8bdSFrançois Tigeot ret = i915_gem_obj_ggtt_pin(ctx_obj, GEN8_LR_CONTEXT_ALIGN, 1026352ff8bdSFrançois Tigeot PIN_OFFSET_BIAS | GUC_WOPCM_TOP); 10272c9916cdSFrançois Tigeot if (ret) 1028352ff8bdSFrançois Tigeot return ret; 10292c9916cdSFrançois Tigeot 10302c9916cdSFrançois Tigeot ret = intel_pin_and_map_ringbuffer_obj(ring->dev, ringbuf); 10312c9916cdSFrançois Tigeot if (ret) 10322c9916cdSFrançois Tigeot goto unpin_ctx_obj; 103319c468b4SFrançois Tigeot 103419c468b4SFrançois Tigeot ctx_obj->dirty = true; 1035352ff8bdSFrançois Tigeot 1036352ff8bdSFrançois Tigeot /* Invalidate GuC TLB. */ 1037352ff8bdSFrançois Tigeot if (i915.enable_guc_submission) 1038352ff8bdSFrançois Tigeot I915_WRITE(GEN8_GTCR, GEN8_GTCR_INVALIDATE); 10392c9916cdSFrançois Tigeot 10402c9916cdSFrançois Tigeot return ret; 10412c9916cdSFrançois Tigeot 10422c9916cdSFrançois Tigeot unpin_ctx_obj: 10432c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 1044352ff8bdSFrançois Tigeot 1045352ff8bdSFrançois Tigeot return ret; 1046352ff8bdSFrançois Tigeot } 1047352ff8bdSFrançois Tigeot 1048352ff8bdSFrançois Tigeot static int intel_lr_context_pin(struct drm_i915_gem_request *rq) 1049352ff8bdSFrançois Tigeot { 1050352ff8bdSFrançois Tigeot int ret = 0; 1051352ff8bdSFrançois Tigeot struct intel_engine_cs *ring = rq->ring; 1052352ff8bdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = rq->ctx->engine[ring->id].state; 1053352ff8bdSFrançois Tigeot struct intel_ringbuffer *ringbuf = rq->ringbuf; 1054352ff8bdSFrançois Tigeot 1055352ff8bdSFrançois Tigeot if (rq->ctx->engine[ring->id].pin_count++ == 0) { 1056352ff8bdSFrançois Tigeot ret = intel_lr_context_do_pin(ring, ctx_obj, ringbuf); 1057352ff8bdSFrançois Tigeot if (ret) 1058352ff8bdSFrançois Tigeot goto reset_pin_count; 1059352ff8bdSFrançois Tigeot } 1060352ff8bdSFrançois Tigeot return ret; 1061352ff8bdSFrançois Tigeot 10622c9916cdSFrançois Tigeot reset_pin_count: 1063a05eeebfSFrançois Tigeot rq->ctx->engine[ring->id].pin_count = 0; 10642c9916cdSFrançois Tigeot return ret; 10652c9916cdSFrançois Tigeot } 10662c9916cdSFrançois Tigeot 1067a05eeebfSFrançois Tigeot void intel_lr_context_unpin(struct drm_i915_gem_request *rq) 10682c9916cdSFrançois Tigeot { 1069a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = rq->ring; 1070a05eeebfSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = rq->ctx->engine[ring->id].state; 1071a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = rq->ringbuf; 10722c9916cdSFrançois Tigeot 10732c9916cdSFrançois Tigeot if (ctx_obj) { 10742c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 1075a05eeebfSFrançois Tigeot if (--rq->ctx->engine[ring->id].pin_count == 0) { 10762c9916cdSFrançois Tigeot intel_unpin_ringbuffer_obj(ringbuf); 10772c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 10782c9916cdSFrançois Tigeot } 10792c9916cdSFrançois Tigeot } 10802c9916cdSFrançois Tigeot } 10812c9916cdSFrançois Tigeot 1082a05eeebfSFrançois Tigeot static int intel_logical_ring_workarounds_emit(struct drm_i915_gem_request *req) 10832c9916cdSFrançois Tigeot { 10842c9916cdSFrançois Tigeot int ret, i; 1085a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = req->ring; 1086a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 10872c9916cdSFrançois Tigeot struct drm_device *dev = ring->dev; 10882c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 10892c9916cdSFrançois Tigeot struct i915_workarounds *w = &dev_priv->workarounds; 10902c9916cdSFrançois Tigeot 10912c9916cdSFrançois Tigeot if (WARN_ON_ONCE(w->count == 0)) 10922c9916cdSFrançois Tigeot return 0; 10932c9916cdSFrançois Tigeot 10942c9916cdSFrançois Tigeot ring->gpu_caches_dirty = true; 1095a05eeebfSFrançois Tigeot ret = logical_ring_flush_all_caches(req); 10962c9916cdSFrançois Tigeot if (ret) 10972c9916cdSFrançois Tigeot return ret; 10982c9916cdSFrançois Tigeot 1099a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(req, w->count * 2 + 2); 11002c9916cdSFrançois Tigeot if (ret) 11012c9916cdSFrançois Tigeot return ret; 11022c9916cdSFrançois Tigeot 11032c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(w->count)); 11042c9916cdSFrançois Tigeot for (i = 0; i < w->count; i++) { 1105*aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, w->reg[i].addr); 11062c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, w->reg[i].value); 11072c9916cdSFrançois Tigeot } 11082c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 11092c9916cdSFrançois Tigeot 11102c9916cdSFrançois Tigeot intel_logical_ring_advance(ringbuf); 11112c9916cdSFrançois Tigeot 11122c9916cdSFrançois Tigeot ring->gpu_caches_dirty = true; 1113a05eeebfSFrançois Tigeot ret = logical_ring_flush_all_caches(req); 11142c9916cdSFrançois Tigeot if (ret) 11152c9916cdSFrançois Tigeot return ret; 11162c9916cdSFrançois Tigeot 11172c9916cdSFrançois Tigeot return 0; 11182c9916cdSFrançois Tigeot } 11192c9916cdSFrançois Tigeot 1120a05eeebfSFrançois Tigeot #define wa_ctx_emit(batch, index, cmd) \ 1121a05eeebfSFrançois Tigeot do { \ 1122a05eeebfSFrançois Tigeot int __index = (index)++; \ 1123a05eeebfSFrançois Tigeot if (WARN_ON(__index >= (PAGE_SIZE / sizeof(uint32_t)))) { \ 1124a05eeebfSFrançois Tigeot return -ENOSPC; \ 1125a05eeebfSFrançois Tigeot } \ 1126a05eeebfSFrançois Tigeot batch[__index] = (cmd); \ 1127a05eeebfSFrançois Tigeot } while (0) 1128a05eeebfSFrançois Tigeot 1129*aee94f86SFrançois Tigeot #define wa_ctx_emit_reg(batch, index, reg) \ 1130*aee94f86SFrançois Tigeot wa_ctx_emit((batch), (index), i915_mmio_reg_offset(reg)) 1131a05eeebfSFrançois Tigeot 1132a05eeebfSFrançois Tigeot /* 1133a05eeebfSFrançois Tigeot * In this WA we need to set GEN8_L3SQCREG4[21:21] and reset it after 1134a05eeebfSFrançois Tigeot * PIPE_CONTROL instruction. This is required for the flush to happen correctly 1135a05eeebfSFrançois Tigeot * but there is a slight complication as this is applied in WA batch where the 1136a05eeebfSFrançois Tigeot * values are only initialized once so we cannot take register value at the 1137a05eeebfSFrançois Tigeot * beginning and reuse it further; hence we save its value to memory, upload a 1138a05eeebfSFrançois Tigeot * constant value with bit21 set and then we restore it back with the saved value. 1139a05eeebfSFrançois Tigeot * To simplify the WA, a constant value is formed by using the default value 1140a05eeebfSFrançois Tigeot * of this register. This shouldn't be a problem because we are only modifying 1141a05eeebfSFrançois Tigeot * it for a short period and this batch in non-premptible. We can ofcourse 1142a05eeebfSFrançois Tigeot * use additional instructions that read the actual value of the register 1143a05eeebfSFrançois Tigeot * at that time and set our bit of interest but it makes the WA complicated. 1144a05eeebfSFrançois Tigeot * 1145a05eeebfSFrançois Tigeot * This WA is also required for Gen9 so extracting as a function avoids 1146a05eeebfSFrançois Tigeot * code duplication. 1147a05eeebfSFrançois Tigeot */ 1148a05eeebfSFrançois Tigeot static inline int gen8_emit_flush_coherentl3_wa(struct intel_engine_cs *ring, 1149a05eeebfSFrançois Tigeot uint32_t *const batch, 1150a05eeebfSFrançois Tigeot uint32_t index) 1151a05eeebfSFrançois Tigeot { 1152a05eeebfSFrançois Tigeot uint32_t l3sqc4_flush = (0x40400000 | GEN8_LQSC_FLUSH_COHERENT_LINES); 1153a05eeebfSFrançois Tigeot 1154a05eeebfSFrançois Tigeot /* 1155a05eeebfSFrançois Tigeot * WaDisableLSQCROPERFforOCL:skl 1156a05eeebfSFrançois Tigeot * This WA is implemented in skl_init_clock_gating() but since 1157a05eeebfSFrançois Tigeot * this batch updates GEN8_L3SQCREG4 with default value we need to 1158a05eeebfSFrançois Tigeot * set this bit here to retain the WA during flush. 1159a05eeebfSFrançois Tigeot */ 1160*aee94f86SFrançois Tigeot if (IS_SKL_REVID(ring->dev, 0, SKL_REVID_E0)) 1161a05eeebfSFrançois Tigeot l3sqc4_flush |= GEN8_LQSC_RO_PERF_DIS; 1162a05eeebfSFrançois Tigeot 1163352ff8bdSFrançois Tigeot wa_ctx_emit(batch, index, (MI_STORE_REGISTER_MEM_GEN8 | 1164a05eeebfSFrançois Tigeot MI_SRM_LRM_GLOBAL_GTT)); 1165*aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 1166a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, ring->scratch.gtt_offset + 256); 1167a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1168a05eeebfSFrançois Tigeot 1169a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_LOAD_REGISTER_IMM(1)); 1170*aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 1171a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, l3sqc4_flush); 1172a05eeebfSFrançois Tigeot 1173a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, GFX_OP_PIPE_CONTROL(6)); 1174a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, (PIPE_CONTROL_CS_STALL | 1175a05eeebfSFrançois Tigeot PIPE_CONTROL_DC_FLUSH_ENABLE)); 1176a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1177a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1178a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1179a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1180a05eeebfSFrançois Tigeot 1181352ff8bdSFrançois Tigeot wa_ctx_emit(batch, index, (MI_LOAD_REGISTER_MEM_GEN8 | 1182a05eeebfSFrançois Tigeot MI_SRM_LRM_GLOBAL_GTT)); 1183*aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN8_L3SQCREG4); 1184a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, ring->scratch.gtt_offset + 256); 1185a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1186a05eeebfSFrançois Tigeot 1187a05eeebfSFrançois Tigeot return index; 1188a05eeebfSFrançois Tigeot } 1189a05eeebfSFrançois Tigeot 1190a05eeebfSFrançois Tigeot static inline uint32_t wa_ctx_start(struct i915_wa_ctx_bb *wa_ctx, 1191a05eeebfSFrançois Tigeot uint32_t offset, 1192a05eeebfSFrançois Tigeot uint32_t start_alignment) 1193a05eeebfSFrançois Tigeot { 1194a05eeebfSFrançois Tigeot return wa_ctx->offset = ALIGN(offset, start_alignment); 1195a05eeebfSFrançois Tigeot } 1196a05eeebfSFrançois Tigeot 1197a05eeebfSFrançois Tigeot static inline int wa_ctx_end(struct i915_wa_ctx_bb *wa_ctx, 1198a05eeebfSFrançois Tigeot uint32_t offset, 1199a05eeebfSFrançois Tigeot uint32_t size_alignment) 1200a05eeebfSFrançois Tigeot { 1201a05eeebfSFrançois Tigeot wa_ctx->size = offset - wa_ctx->offset; 1202a05eeebfSFrançois Tigeot 1203a05eeebfSFrançois Tigeot WARN(wa_ctx->size % size_alignment, 1204a05eeebfSFrançois Tigeot "wa_ctx_bb failed sanity checks: size %d is not aligned to %d\n", 1205a05eeebfSFrançois Tigeot wa_ctx->size, size_alignment); 1206a05eeebfSFrançois Tigeot return 0; 1207a05eeebfSFrançois Tigeot } 1208a05eeebfSFrançois Tigeot 1209a05eeebfSFrançois Tigeot /** 1210a05eeebfSFrançois Tigeot * gen8_init_indirectctx_bb() - initialize indirect ctx batch with WA 1211a05eeebfSFrançois Tigeot * 1212a05eeebfSFrançois Tigeot * @ring: only applicable for RCS 1213a05eeebfSFrançois Tigeot * @wa_ctx: structure representing wa_ctx 1214a05eeebfSFrançois Tigeot * offset: specifies start of the batch, should be cache-aligned. This is updated 1215a05eeebfSFrançois Tigeot * with the offset value received as input. 1216a05eeebfSFrançois Tigeot * size: size of the batch in DWORDS but HW expects in terms of cachelines 1217a05eeebfSFrançois Tigeot * @batch: page in which WA are loaded 1218a05eeebfSFrançois Tigeot * @offset: This field specifies the start of the batch, it should be 1219a05eeebfSFrançois Tigeot * cache-aligned otherwise it is adjusted accordingly. 1220a05eeebfSFrançois Tigeot * Typically we only have one indirect_ctx and per_ctx batch buffer which are 1221a05eeebfSFrançois Tigeot * initialized at the beginning and shared across all contexts but this field 1222a05eeebfSFrançois Tigeot * helps us to have multiple batches at different offsets and select them based 1223a05eeebfSFrançois Tigeot * on a criteria. At the moment this batch always start at the beginning of the page 1224a05eeebfSFrançois Tigeot * and at this point we don't have multiple wa_ctx batch buffers. 1225a05eeebfSFrançois Tigeot * 1226a05eeebfSFrançois Tigeot * The number of WA applied are not known at the beginning; we use this field 1227a05eeebfSFrançois Tigeot * to return the no of DWORDS written. 1228a05eeebfSFrançois Tigeot * 1229a05eeebfSFrançois Tigeot * It is to be noted that this batch does not contain MI_BATCH_BUFFER_END 1230a05eeebfSFrançois Tigeot * so it adds NOOPs as padding to make it cacheline aligned. 1231a05eeebfSFrançois Tigeot * MI_BATCH_BUFFER_END will be added to perctx batch and both of them together 1232a05eeebfSFrançois Tigeot * makes a complete batch buffer. 1233a05eeebfSFrançois Tigeot * 1234a05eeebfSFrançois Tigeot * Return: non-zero if we exceed the PAGE_SIZE limit. 1235a05eeebfSFrançois Tigeot */ 1236a05eeebfSFrançois Tigeot 1237a05eeebfSFrançois Tigeot static int gen8_init_indirectctx_bb(struct intel_engine_cs *ring, 1238a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1239a05eeebfSFrançois Tigeot uint32_t *const batch, 1240a05eeebfSFrançois Tigeot uint32_t *offset) 1241a05eeebfSFrançois Tigeot { 1242a05eeebfSFrançois Tigeot uint32_t scratch_addr; 1243a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1244a05eeebfSFrançois Tigeot 1245a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:bdw,chv */ 1246a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_DISABLE); 1247a05eeebfSFrançois Tigeot 1248a05eeebfSFrançois Tigeot /* WaFlushCoherentL3CacheLinesAtContextSwitch:bdw */ 1249a05eeebfSFrançois Tigeot if (IS_BROADWELL(ring->dev)) { 1250352ff8bdSFrançois Tigeot int rc = gen8_emit_flush_coherentl3_wa(ring, batch, index); 1251352ff8bdSFrançois Tigeot if (rc < 0) 1252352ff8bdSFrançois Tigeot return rc; 1253352ff8bdSFrançois Tigeot index = rc; 1254a05eeebfSFrançois Tigeot } 1255a05eeebfSFrançois Tigeot 1256a05eeebfSFrançois Tigeot /* WaClearSlmSpaceAtContextSwitch:bdw,chv */ 1257a05eeebfSFrançois Tigeot /* Actual scratch location is at 128 bytes offset */ 1258a05eeebfSFrançois Tigeot scratch_addr = ring->scratch.gtt_offset + 2*CACHELINE_BYTES; 1259a05eeebfSFrançois Tigeot 1260a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, GFX_OP_PIPE_CONTROL(6)); 1261a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, (PIPE_CONTROL_FLUSH_L3 | 1262a05eeebfSFrançois Tigeot PIPE_CONTROL_GLOBAL_GTT_IVB | 1263a05eeebfSFrançois Tigeot PIPE_CONTROL_CS_STALL | 1264a05eeebfSFrançois Tigeot PIPE_CONTROL_QW_WRITE)); 1265a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, scratch_addr); 1266a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1267a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1268a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 0); 1269a05eeebfSFrançois Tigeot 1270a05eeebfSFrançois Tigeot /* Pad to end of cacheline */ 1271a05eeebfSFrançois Tigeot while (index % CACHELINE_DWORDS) 1272a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1273a05eeebfSFrançois Tigeot 1274a05eeebfSFrançois Tigeot /* 1275a05eeebfSFrançois Tigeot * MI_BATCH_BUFFER_END is not required in Indirect ctx BB because 1276a05eeebfSFrançois Tigeot * execution depends on the length specified in terms of cache lines 1277a05eeebfSFrançois Tigeot * in the register CTX_RCS_INDIRECT_CTX 1278a05eeebfSFrançois Tigeot */ 1279a05eeebfSFrançois Tigeot 1280a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, CACHELINE_DWORDS); 1281a05eeebfSFrançois Tigeot } 1282a05eeebfSFrançois Tigeot 1283a05eeebfSFrançois Tigeot /** 1284a05eeebfSFrançois Tigeot * gen8_init_perctx_bb() - initialize per ctx batch with WA 1285a05eeebfSFrançois Tigeot * 1286a05eeebfSFrançois Tigeot * @ring: only applicable for RCS 1287a05eeebfSFrançois Tigeot * @wa_ctx: structure representing wa_ctx 1288a05eeebfSFrançois Tigeot * offset: specifies start of the batch, should be cache-aligned. 1289a05eeebfSFrançois Tigeot * size: size of the batch in DWORDS but HW expects in terms of cachelines 1290a05eeebfSFrançois Tigeot * @batch: page in which WA are loaded 1291a05eeebfSFrançois Tigeot * @offset: This field specifies the start of this batch. 1292a05eeebfSFrançois Tigeot * This batch is started immediately after indirect_ctx batch. Since we ensure 1293a05eeebfSFrançois Tigeot * that indirect_ctx ends on a cacheline this batch is aligned automatically. 1294a05eeebfSFrançois Tigeot * 1295a05eeebfSFrançois Tigeot * The number of DWORDS written are returned using this field. 1296a05eeebfSFrançois Tigeot * 1297a05eeebfSFrançois Tigeot * This batch is terminated with MI_BATCH_BUFFER_END and so we need not add padding 1298a05eeebfSFrançois Tigeot * to align it with cacheline as padding after MI_BATCH_BUFFER_END is redundant. 1299a05eeebfSFrançois Tigeot */ 1300a05eeebfSFrançois Tigeot static int gen8_init_perctx_bb(struct intel_engine_cs *ring, 1301a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1302a05eeebfSFrançois Tigeot uint32_t *const batch, 1303a05eeebfSFrançois Tigeot uint32_t *offset) 1304a05eeebfSFrançois Tigeot { 1305a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1306a05eeebfSFrançois Tigeot 1307a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:bdw,chv */ 1308a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_ENABLE); 1309a05eeebfSFrançois Tigeot 1310a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_BATCH_BUFFER_END); 1311a05eeebfSFrançois Tigeot 1312a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, 1); 1313a05eeebfSFrançois Tigeot } 1314a05eeebfSFrançois Tigeot 1315a05eeebfSFrançois Tigeot static int gen9_init_indirectctx_bb(struct intel_engine_cs *ring, 1316a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1317a05eeebfSFrançois Tigeot uint32_t *const batch, 1318a05eeebfSFrançois Tigeot uint32_t *offset) 1319a05eeebfSFrançois Tigeot { 1320a05eeebfSFrançois Tigeot int ret; 1321a05eeebfSFrançois Tigeot struct drm_device *dev = ring->dev; 1322a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1323a05eeebfSFrançois Tigeot 1324a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:skl,bxt */ 1325*aee94f86SFrançois Tigeot if (IS_SKL_REVID(dev, 0, SKL_REVID_D0) || 1326*aee94f86SFrançois Tigeot IS_BXT_REVID(dev, 0, BXT_REVID_A1)) 1327a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_DISABLE); 1328a05eeebfSFrançois Tigeot 1329a05eeebfSFrançois Tigeot /* WaFlushCoherentL3CacheLinesAtContextSwitch:skl,bxt */ 1330a05eeebfSFrançois Tigeot ret = gen8_emit_flush_coherentl3_wa(ring, batch, index); 1331a05eeebfSFrançois Tigeot if (ret < 0) 1332a05eeebfSFrançois Tigeot return ret; 1333a05eeebfSFrançois Tigeot index = ret; 1334a05eeebfSFrançois Tigeot 1335a05eeebfSFrançois Tigeot /* Pad to end of cacheline */ 1336a05eeebfSFrançois Tigeot while (index % CACHELINE_DWORDS) 1337a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1338a05eeebfSFrançois Tigeot 1339a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, CACHELINE_DWORDS); 1340a05eeebfSFrançois Tigeot } 1341a05eeebfSFrançois Tigeot 1342a05eeebfSFrançois Tigeot static int gen9_init_perctx_bb(struct intel_engine_cs *ring, 1343a05eeebfSFrançois Tigeot struct i915_wa_ctx_bb *wa_ctx, 1344a05eeebfSFrançois Tigeot uint32_t *const batch, 1345a05eeebfSFrançois Tigeot uint32_t *offset) 1346a05eeebfSFrançois Tigeot { 1347a05eeebfSFrançois Tigeot struct drm_device *dev = ring->dev; 1348a05eeebfSFrançois Tigeot uint32_t index = wa_ctx_start(wa_ctx, *offset, CACHELINE_DWORDS); 1349a05eeebfSFrançois Tigeot 1350a05eeebfSFrançois Tigeot /* WaSetDisablePixMaskCammingAndRhwoInCommonSliceChicken:skl,bxt */ 1351*aee94f86SFrançois Tigeot if (IS_SKL_REVID(dev, 0, SKL_REVID_B0) || 1352*aee94f86SFrançois Tigeot IS_BXT_REVID(dev, 0, BXT_REVID_A1)) { 1353a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_LOAD_REGISTER_IMM(1)); 1354*aee94f86SFrançois Tigeot wa_ctx_emit_reg(batch, index, GEN9_SLICE_COMMON_ECO_CHICKEN0); 1355a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, 1356a05eeebfSFrançois Tigeot _MASKED_BIT_ENABLE(DISABLE_PIXEL_MASK_CAMMING)); 1357a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_NOOP); 1358a05eeebfSFrançois Tigeot } 1359a05eeebfSFrançois Tigeot 1360a05eeebfSFrançois Tigeot /* WaDisableCtxRestoreArbitration:skl,bxt */ 1361*aee94f86SFrançois Tigeot if (IS_SKL_REVID(dev, 0, SKL_REVID_D0) || 1362*aee94f86SFrançois Tigeot IS_BXT_REVID(dev, 0, BXT_REVID_A1)) 1363a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_ARB_ON_OFF | MI_ARB_ENABLE); 1364a05eeebfSFrançois Tigeot 1365a05eeebfSFrançois Tigeot wa_ctx_emit(batch, index, MI_BATCH_BUFFER_END); 1366a05eeebfSFrançois Tigeot 1367a05eeebfSFrançois Tigeot return wa_ctx_end(wa_ctx, *offset = index, 1); 1368a05eeebfSFrançois Tigeot } 1369a05eeebfSFrançois Tigeot 1370a05eeebfSFrançois Tigeot static int lrc_setup_wa_ctx_obj(struct intel_engine_cs *ring, u32 size) 1371a05eeebfSFrançois Tigeot { 1372a05eeebfSFrançois Tigeot int ret; 1373a05eeebfSFrançois Tigeot 1374a05eeebfSFrançois Tigeot ring->wa_ctx.obj = i915_gem_alloc_object(ring->dev, PAGE_ALIGN(size)); 1375a05eeebfSFrançois Tigeot if (!ring->wa_ctx.obj) { 1376a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("alloc LRC WA ctx backing obj failed.\n"); 1377a05eeebfSFrançois Tigeot return -ENOMEM; 1378a05eeebfSFrançois Tigeot } 1379a05eeebfSFrançois Tigeot 1380a05eeebfSFrançois Tigeot ret = i915_gem_obj_ggtt_pin(ring->wa_ctx.obj, PAGE_SIZE, 0); 1381a05eeebfSFrançois Tigeot if (ret) { 1382a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("pin LRC WA ctx backing obj failed: %d\n", 1383a05eeebfSFrançois Tigeot ret); 1384a05eeebfSFrançois Tigeot drm_gem_object_unreference(&ring->wa_ctx.obj->base); 1385a05eeebfSFrançois Tigeot return ret; 1386a05eeebfSFrançois Tigeot } 1387a05eeebfSFrançois Tigeot 1388a05eeebfSFrançois Tigeot return 0; 1389a05eeebfSFrançois Tigeot } 1390a05eeebfSFrançois Tigeot 1391a05eeebfSFrançois Tigeot static void lrc_destroy_wa_ctx_obj(struct intel_engine_cs *ring) 1392a05eeebfSFrançois Tigeot { 1393a05eeebfSFrançois Tigeot if (ring->wa_ctx.obj) { 1394a05eeebfSFrançois Tigeot i915_gem_object_ggtt_unpin(ring->wa_ctx.obj); 1395a05eeebfSFrançois Tigeot drm_gem_object_unreference(&ring->wa_ctx.obj->base); 1396a05eeebfSFrançois Tigeot ring->wa_ctx.obj = NULL; 1397a05eeebfSFrançois Tigeot } 1398a05eeebfSFrançois Tigeot } 1399a05eeebfSFrançois Tigeot 1400a05eeebfSFrançois Tigeot static int intel_init_workaround_bb(struct intel_engine_cs *ring) 1401a05eeebfSFrançois Tigeot { 1402a05eeebfSFrançois Tigeot int ret; 1403a05eeebfSFrançois Tigeot uint32_t *batch; 1404a05eeebfSFrançois Tigeot uint32_t offset; 1405a05eeebfSFrançois Tigeot struct vm_page *page; 1406a05eeebfSFrançois Tigeot struct i915_ctx_workarounds *wa_ctx = &ring->wa_ctx; 1407a05eeebfSFrançois Tigeot 1408a05eeebfSFrançois Tigeot WARN_ON(ring->id != RCS); 1409a05eeebfSFrançois Tigeot 1410a05eeebfSFrançois Tigeot /* update this when WA for higher Gen are added */ 1411a05eeebfSFrançois Tigeot if (INTEL_INFO(ring->dev)->gen > 9) { 1412a05eeebfSFrançois Tigeot DRM_ERROR("WA batch buffer is not initialized for Gen%d\n", 1413a05eeebfSFrançois Tigeot INTEL_INFO(ring->dev)->gen); 1414a05eeebfSFrançois Tigeot return 0; 1415a05eeebfSFrançois Tigeot } 1416a05eeebfSFrançois Tigeot 1417a05eeebfSFrançois Tigeot /* some WA perform writes to scratch page, ensure it is valid */ 1418a05eeebfSFrançois Tigeot if (ring->scratch.obj == NULL) { 1419a05eeebfSFrançois Tigeot DRM_ERROR("scratch page not allocated for %s\n", ring->name); 1420a05eeebfSFrançois Tigeot return -EINVAL; 1421a05eeebfSFrançois Tigeot } 1422a05eeebfSFrançois Tigeot 1423a05eeebfSFrançois Tigeot ret = lrc_setup_wa_ctx_obj(ring, PAGE_SIZE); 1424a05eeebfSFrançois Tigeot if (ret) { 1425a05eeebfSFrançois Tigeot DRM_DEBUG_DRIVER("Failed to setup context WA page: %d\n", ret); 1426a05eeebfSFrançois Tigeot return ret; 1427a05eeebfSFrançois Tigeot } 1428a05eeebfSFrançois Tigeot 1429*aee94f86SFrançois Tigeot page = i915_gem_object_get_dirty_page(wa_ctx->obj, 0); 1430a05eeebfSFrançois Tigeot batch = kmap_atomic(page); 1431a05eeebfSFrançois Tigeot offset = 0; 1432a05eeebfSFrançois Tigeot 1433a05eeebfSFrançois Tigeot if (INTEL_INFO(ring->dev)->gen == 8) { 1434a05eeebfSFrançois Tigeot ret = gen8_init_indirectctx_bb(ring, 1435a05eeebfSFrançois Tigeot &wa_ctx->indirect_ctx, 1436a05eeebfSFrançois Tigeot batch, 1437a05eeebfSFrançois Tigeot &offset); 1438a05eeebfSFrançois Tigeot if (ret) 1439a05eeebfSFrançois Tigeot goto out; 1440a05eeebfSFrançois Tigeot 1441a05eeebfSFrançois Tigeot ret = gen8_init_perctx_bb(ring, 1442a05eeebfSFrançois Tigeot &wa_ctx->per_ctx, 1443a05eeebfSFrançois Tigeot batch, 1444a05eeebfSFrançois Tigeot &offset); 1445a05eeebfSFrançois Tigeot if (ret) 1446a05eeebfSFrançois Tigeot goto out; 1447a05eeebfSFrançois Tigeot } else if (INTEL_INFO(ring->dev)->gen == 9) { 1448a05eeebfSFrançois Tigeot ret = gen9_init_indirectctx_bb(ring, 1449a05eeebfSFrançois Tigeot &wa_ctx->indirect_ctx, 1450a05eeebfSFrançois Tigeot batch, 1451a05eeebfSFrançois Tigeot &offset); 1452a05eeebfSFrançois Tigeot if (ret) 1453a05eeebfSFrançois Tigeot goto out; 1454a05eeebfSFrançois Tigeot 1455a05eeebfSFrançois Tigeot ret = gen9_init_perctx_bb(ring, 1456a05eeebfSFrançois Tigeot &wa_ctx->per_ctx, 1457a05eeebfSFrançois Tigeot batch, 1458a05eeebfSFrançois Tigeot &offset); 1459a05eeebfSFrançois Tigeot if (ret) 1460a05eeebfSFrançois Tigeot goto out; 1461a05eeebfSFrançois Tigeot } 1462a05eeebfSFrançois Tigeot 1463a05eeebfSFrançois Tigeot out: 1464a05eeebfSFrançois Tigeot kunmap_atomic(batch); 1465a05eeebfSFrançois Tigeot if (ret) 1466a05eeebfSFrançois Tigeot lrc_destroy_wa_ctx_obj(ring); 1467a05eeebfSFrançois Tigeot 1468a05eeebfSFrançois Tigeot return ret; 1469a05eeebfSFrançois Tigeot } 1470a05eeebfSFrançois Tigeot 14711b13d190SFrançois Tigeot static int gen8_init_common_ring(struct intel_engine_cs *ring) 14721b13d190SFrançois Tigeot { 14731b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 14741b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 1475a05eeebfSFrançois Tigeot u8 next_context_status_buffer_hw; 14761b13d190SFrançois Tigeot 1477352ff8bdSFrançois Tigeot lrc_setup_hardware_status_page(ring, 1478352ff8bdSFrançois Tigeot ring->default_context->engine[ring->id].state); 1479352ff8bdSFrançois Tigeot 14801b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~(ring->irq_enable_mask | ring->irq_keep_mask)); 14811b13d190SFrançois Tigeot I915_WRITE(RING_HWSTAM(ring->mmio_base), 0xffffffff); 14821b13d190SFrançois Tigeot 14831b13d190SFrançois Tigeot I915_WRITE(RING_MODE_GEN7(ring), 14841b13d190SFrançois Tigeot _MASKED_BIT_DISABLE(GFX_REPLAY_MODE) | 14851b13d190SFrançois Tigeot _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE)); 14861b13d190SFrançois Tigeot POSTING_READ(RING_MODE_GEN7(ring)); 1487a05eeebfSFrançois Tigeot 1488a05eeebfSFrançois Tigeot /* 1489a05eeebfSFrançois Tigeot * Instead of resetting the Context Status Buffer (CSB) read pointer to 1490a05eeebfSFrançois Tigeot * zero, we need to read the write pointer from hardware and use its 1491a05eeebfSFrançois Tigeot * value because "this register is power context save restored". 1492a05eeebfSFrançois Tigeot * Effectively, these states have been observed: 1493a05eeebfSFrançois Tigeot * 1494a05eeebfSFrançois Tigeot * | Suspend-to-idle (freeze) | Suspend-to-RAM (mem) | 1495a05eeebfSFrançois Tigeot * BDW | CSB regs not reset | CSB regs reset | 1496a05eeebfSFrançois Tigeot * CHT | CSB regs not reset | CSB regs not reset | 1497a05eeebfSFrançois Tigeot */ 1498a05eeebfSFrançois Tigeot next_context_status_buffer_hw = (I915_READ(RING_CONTEXT_STATUS_PTR(ring)) 1499a05eeebfSFrançois Tigeot & GEN8_CSB_PTR_MASK); 1500a05eeebfSFrançois Tigeot 1501a05eeebfSFrançois Tigeot /* 1502a05eeebfSFrançois Tigeot * When the CSB registers are reset (also after power-up / gpu reset), 1503a05eeebfSFrançois Tigeot * CSB write pointer is set to all 1's, which is not valid, use '5' in 1504a05eeebfSFrançois Tigeot * this special case, so the first element read is CSB[0]. 1505a05eeebfSFrançois Tigeot */ 1506a05eeebfSFrançois Tigeot if (next_context_status_buffer_hw == GEN8_CSB_PTR_MASK) 1507a05eeebfSFrançois Tigeot next_context_status_buffer_hw = (GEN8_CSB_ENTRIES - 1); 1508a05eeebfSFrançois Tigeot 1509a05eeebfSFrançois Tigeot ring->next_context_status_buffer = next_context_status_buffer_hw; 15101b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Execlists enabled for %s\n", ring->name); 15111b13d190SFrançois Tigeot 15121b13d190SFrançois Tigeot memset(&ring->hangcheck, 0, sizeof(ring->hangcheck)); 15131b13d190SFrançois Tigeot 15141b13d190SFrançois Tigeot return 0; 15151b13d190SFrançois Tigeot } 15161b13d190SFrançois Tigeot 15171b13d190SFrançois Tigeot static int gen8_init_render_ring(struct intel_engine_cs *ring) 15181b13d190SFrançois Tigeot { 15191b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 15201b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 15211b13d190SFrançois Tigeot int ret; 15221b13d190SFrançois Tigeot 15231b13d190SFrançois Tigeot ret = gen8_init_common_ring(ring); 15241b13d190SFrançois Tigeot if (ret) 15251b13d190SFrançois Tigeot return ret; 15261b13d190SFrançois Tigeot 15271b13d190SFrançois Tigeot /* We need to disable the AsyncFlip performance optimisations in order 15281b13d190SFrançois Tigeot * to use MI_WAIT_FOR_EVENT within the CS. It should already be 15291b13d190SFrançois Tigeot * programmed to '1' on all products. 15301b13d190SFrançois Tigeot * 15311b13d190SFrançois Tigeot * WaDisableAsyncFlipPerfMode:snb,ivb,hsw,vlv,bdw,chv 15321b13d190SFrançois Tigeot */ 15331b13d190SFrançois Tigeot I915_WRITE(MI_MODE, _MASKED_BIT_ENABLE(ASYNC_FLIP_PERF_DISABLE)); 15341b13d190SFrançois Tigeot 15351b13d190SFrançois Tigeot I915_WRITE(INSTPM, _MASKED_BIT_ENABLE(INSTPM_FORCE_ORDERING)); 15361b13d190SFrançois Tigeot 15372c9916cdSFrançois Tigeot return init_workarounds_ring(ring); 15381b13d190SFrançois Tigeot } 15391b13d190SFrançois Tigeot 1540477eb7f9SFrançois Tigeot static int gen9_init_render_ring(struct intel_engine_cs *ring) 1541477eb7f9SFrançois Tigeot { 1542477eb7f9SFrançois Tigeot int ret; 1543477eb7f9SFrançois Tigeot 1544477eb7f9SFrançois Tigeot ret = gen8_init_common_ring(ring); 1545477eb7f9SFrançois Tigeot if (ret) 1546477eb7f9SFrançois Tigeot return ret; 1547477eb7f9SFrançois Tigeot 1548477eb7f9SFrançois Tigeot return init_workarounds_ring(ring); 1549477eb7f9SFrançois Tigeot } 1550477eb7f9SFrançois Tigeot 1551a05eeebfSFrançois Tigeot static int intel_logical_ring_emit_pdps(struct drm_i915_gem_request *req) 1552a05eeebfSFrançois Tigeot { 1553a05eeebfSFrançois Tigeot struct i915_hw_ppgtt *ppgtt = req->ctx->ppgtt; 1554a05eeebfSFrançois Tigeot struct intel_engine_cs *ring = req->ring; 1555a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 1556a05eeebfSFrançois Tigeot const int num_lri_cmds = GEN8_LEGACY_PDPES * 2; 1557a05eeebfSFrançois Tigeot int i, ret; 1558a05eeebfSFrançois Tigeot 1559a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(req, num_lri_cmds * 2 + 2); 1560a05eeebfSFrançois Tigeot if (ret) 1561a05eeebfSFrançois Tigeot return ret; 1562a05eeebfSFrançois Tigeot 1563a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(num_lri_cmds)); 1564a05eeebfSFrançois Tigeot for (i = GEN8_LEGACY_PDPES - 1; i >= 0; i--) { 1565a05eeebfSFrançois Tigeot const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i); 1566a05eeebfSFrançois Tigeot 1567*aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, GEN8_RING_PDP_UDW(ring, i)); 1568a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, upper_32_bits(pd_daddr)); 1569*aee94f86SFrançois Tigeot intel_logical_ring_emit_reg(ringbuf, GEN8_RING_PDP_LDW(ring, i)); 1570a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, lower_32_bits(pd_daddr)); 1571a05eeebfSFrançois Tigeot } 1572a05eeebfSFrançois Tigeot 1573a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1574a05eeebfSFrançois Tigeot intel_logical_ring_advance(ringbuf); 1575a05eeebfSFrançois Tigeot 1576a05eeebfSFrançois Tigeot return 0; 1577a05eeebfSFrançois Tigeot } 1578a05eeebfSFrançois Tigeot 1579a05eeebfSFrançois Tigeot static int gen8_emit_bb_start(struct drm_i915_gem_request *req, 1580477eb7f9SFrançois Tigeot u64 offset, unsigned dispatch_flags) 15811b13d190SFrançois Tigeot { 1582a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = req->ringbuf; 1583477eb7f9SFrançois Tigeot bool ppgtt = !(dispatch_flags & I915_DISPATCH_SECURE); 15841b13d190SFrançois Tigeot int ret; 15851b13d190SFrançois Tigeot 1586a05eeebfSFrançois Tigeot /* Don't rely in hw updating PDPs, specially in lite-restore. 1587a05eeebfSFrançois Tigeot * Ideally, we should set Force PD Restore in ctx descriptor, 1588a05eeebfSFrançois Tigeot * but we can't. Force Restore would be a second option, but 1589a05eeebfSFrançois Tigeot * it is unsafe in case of lite-restore (because the ctx is 1590352ff8bdSFrançois Tigeot * not idle). PML4 is allocated during ppgtt init so this is 1591352ff8bdSFrançois Tigeot * not needed in 48-bit.*/ 1592a05eeebfSFrançois Tigeot if (req->ctx->ppgtt && 1593a05eeebfSFrançois Tigeot (intel_ring_flag(req->ring) & req->ctx->ppgtt->pd_dirty_rings)) { 1594352ff8bdSFrançois Tigeot if (!USES_FULL_48BIT_PPGTT(req->i915) && 1595352ff8bdSFrançois Tigeot !intel_vgpu_active(req->i915->dev)) { 1596a05eeebfSFrançois Tigeot ret = intel_logical_ring_emit_pdps(req); 1597a05eeebfSFrançois Tigeot if (ret) 1598a05eeebfSFrançois Tigeot return ret; 1599352ff8bdSFrançois Tigeot } 1600a05eeebfSFrançois Tigeot 1601a05eeebfSFrançois Tigeot req->ctx->ppgtt->pd_dirty_rings &= ~intel_ring_flag(req->ring); 1602a05eeebfSFrançois Tigeot } 1603a05eeebfSFrançois Tigeot 1604a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(req, 4); 16051b13d190SFrançois Tigeot if (ret) 16061b13d190SFrançois Tigeot return ret; 16071b13d190SFrançois Tigeot 16081b13d190SFrançois Tigeot /* FIXME(BDW): Address space and security selectors. */ 1609a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_BATCH_BUFFER_START_GEN8 | 1610a05eeebfSFrançois Tigeot (ppgtt<<8) | 1611a05eeebfSFrançois Tigeot (dispatch_flags & I915_DISPATCH_RS ? 1612a05eeebfSFrançois Tigeot MI_BATCH_RESOURCE_STREAMER : 0)); 16131b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, lower_32_bits(offset)); 16141b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, upper_32_bits(offset)); 16151b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 16161b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 16171b13d190SFrançois Tigeot 16181b13d190SFrançois Tigeot return 0; 16191b13d190SFrançois Tigeot } 16201b13d190SFrançois Tigeot 16211b13d190SFrançois Tigeot static bool gen8_logical_ring_get_irq(struct intel_engine_cs *ring) 16221b13d190SFrançois Tigeot { 16231b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 16241b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 16255e269720SFrançois Tigeot unsigned long flags; 16261b13d190SFrançois Tigeot 16272c9916cdSFrançois Tigeot if (WARN_ON(!intel_irqs_enabled(dev_priv))) 16281b13d190SFrançois Tigeot return false; 16291b13d190SFrançois Tigeot 16305e269720SFrançois Tigeot spin_lock_irqsave(&dev_priv->irq_lock, flags); 16311b13d190SFrançois Tigeot if (ring->irq_refcount++ == 0) { 16321b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~(ring->irq_enable_mask | ring->irq_keep_mask)); 16331b13d190SFrançois Tigeot POSTING_READ(RING_IMR(ring->mmio_base)); 16341b13d190SFrançois Tigeot } 16355e269720SFrançois Tigeot spin_unlock_irqrestore(&dev_priv->irq_lock, flags); 16361b13d190SFrançois Tigeot 16371b13d190SFrançois Tigeot return true; 16381b13d190SFrançois Tigeot } 16391b13d190SFrançois Tigeot 16401b13d190SFrançois Tigeot static void gen8_logical_ring_put_irq(struct intel_engine_cs *ring) 16411b13d190SFrançois Tigeot { 16421b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 16431b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 16445e269720SFrançois Tigeot unsigned long flags; 16451b13d190SFrançois Tigeot 16465e269720SFrançois Tigeot spin_lock_irqsave(&dev_priv->irq_lock, flags); 16471b13d190SFrançois Tigeot if (--ring->irq_refcount == 0) { 16481b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~ring->irq_keep_mask); 16491b13d190SFrançois Tigeot POSTING_READ(RING_IMR(ring->mmio_base)); 16501b13d190SFrançois Tigeot } 16515e269720SFrançois Tigeot spin_unlock_irqrestore(&dev_priv->irq_lock, flags); 16521b13d190SFrançois Tigeot } 16531b13d190SFrançois Tigeot 1654a05eeebfSFrançois Tigeot static int gen8_emit_flush(struct drm_i915_gem_request *request, 16551b13d190SFrançois Tigeot u32 invalidate_domains, 16561b13d190SFrançois Tigeot u32 unused) 16571b13d190SFrançois Tigeot { 1658a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 16591b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 16601b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 16611b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 16621b13d190SFrançois Tigeot uint32_t cmd; 16631b13d190SFrançois Tigeot int ret; 16641b13d190SFrançois Tigeot 1665a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(request, 4); 16661b13d190SFrançois Tigeot if (ret) 16671b13d190SFrançois Tigeot return ret; 16681b13d190SFrançois Tigeot 16691b13d190SFrançois Tigeot cmd = MI_FLUSH_DW + 1; 16701b13d190SFrançois Tigeot 16712c9916cdSFrançois Tigeot /* We always require a command barrier so that subsequent 16722c9916cdSFrançois Tigeot * commands, such as breadcrumb interrupts, are strictly ordered 16732c9916cdSFrançois Tigeot * wrt the contents of the write cache being flushed to memory 16742c9916cdSFrançois Tigeot * (and thus being coherent from the CPU). 16752c9916cdSFrançois Tigeot */ 16762c9916cdSFrançois Tigeot cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW; 16772c9916cdSFrançois Tigeot 16782c9916cdSFrançois Tigeot if (invalidate_domains & I915_GEM_GPU_DOMAINS) { 16792c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_TLB; 16802c9916cdSFrançois Tigeot if (ring == &dev_priv->ring[VCS]) 16812c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_BSD; 16821b13d190SFrançois Tigeot } 16831b13d190SFrançois Tigeot 16841b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, cmd); 16851b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 16861b13d190SFrançois Tigeot I915_GEM_HWS_SCRATCH_ADDR | 16871b13d190SFrançois Tigeot MI_FLUSH_DW_USE_GTT); 16881b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* upper addr */ 16891b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* value */ 16901b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 16911b13d190SFrançois Tigeot 16921b13d190SFrançois Tigeot return 0; 16931b13d190SFrançois Tigeot } 16941b13d190SFrançois Tigeot 1695a05eeebfSFrançois Tigeot static int gen8_emit_flush_render(struct drm_i915_gem_request *request, 16961b13d190SFrançois Tigeot u32 invalidate_domains, 16971b13d190SFrançois Tigeot u32 flush_domains) 16981b13d190SFrançois Tigeot { 1699a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 17001b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 17011b13d190SFrançois Tigeot u32 scratch_addr = ring->scratch.gtt_offset + 2 * CACHELINE_BYTES; 170219c468b4SFrançois Tigeot bool vf_flush_wa; 17031b13d190SFrançois Tigeot u32 flags = 0; 17041b13d190SFrançois Tigeot int ret; 17051b13d190SFrançois Tigeot 17061b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CS_STALL; 17071b13d190SFrançois Tigeot 17081b13d190SFrançois Tigeot if (flush_domains) { 17091b13d190SFrançois Tigeot flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH; 17101b13d190SFrançois Tigeot flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH; 1711*aee94f86SFrançois Tigeot flags |= PIPE_CONTROL_DC_FLUSH_ENABLE; 1712b49c8cf9SFrançois Tigeot flags |= PIPE_CONTROL_FLUSH_ENABLE; 17131b13d190SFrançois Tigeot } 17141b13d190SFrançois Tigeot 17151b13d190SFrançois Tigeot if (invalidate_domains) { 17161b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TLB_INVALIDATE; 17171b13d190SFrançois Tigeot flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE; 17181b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE; 17191b13d190SFrançois Tigeot flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE; 17201b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE; 17211b13d190SFrançois Tigeot flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE; 17221b13d190SFrançois Tigeot flags |= PIPE_CONTROL_QW_WRITE; 17231b13d190SFrançois Tigeot flags |= PIPE_CONTROL_GLOBAL_GTT_IVB; 17241b13d190SFrançois Tigeot } 17251b13d190SFrançois Tigeot 172619c468b4SFrançois Tigeot /* 172719c468b4SFrançois Tigeot * On GEN9+ Before VF_CACHE_INVALIDATE we need to emit a NULL pipe 172819c468b4SFrançois Tigeot * control. 172919c468b4SFrançois Tigeot */ 173019c468b4SFrançois Tigeot vf_flush_wa = INTEL_INFO(ring->dev)->gen >= 9 && 173119c468b4SFrançois Tigeot flags & PIPE_CONTROL_VF_CACHE_INVALIDATE; 173219c468b4SFrançois Tigeot 1733a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(request, vf_flush_wa ? 12 : 6); 17341b13d190SFrançois Tigeot if (ret) 17351b13d190SFrançois Tigeot return ret; 17361b13d190SFrançois Tigeot 173719c468b4SFrançois Tigeot if (vf_flush_wa) { 173819c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 173919c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 174019c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 174119c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 174219c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 174319c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 174419c468b4SFrançois Tigeot } 174519c468b4SFrançois Tigeot 17461b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 17471b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, flags); 17481b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, scratch_addr); 17491b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17501b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17511b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 17521b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 17531b13d190SFrançois Tigeot 17541b13d190SFrançois Tigeot return 0; 17551b13d190SFrançois Tigeot } 17561b13d190SFrançois Tigeot 17571b13d190SFrançois Tigeot static u32 gen8_get_seqno(struct intel_engine_cs *ring, bool lazy_coherency) 17581b13d190SFrançois Tigeot { 17591b13d190SFrançois Tigeot return intel_read_status_page(ring, I915_GEM_HWS_INDEX); 17601b13d190SFrançois Tigeot } 17611b13d190SFrançois Tigeot 17621b13d190SFrançois Tigeot static void gen8_set_seqno(struct intel_engine_cs *ring, u32 seqno) 17631b13d190SFrançois Tigeot { 17641b13d190SFrançois Tigeot intel_write_status_page(ring, I915_GEM_HWS_INDEX, seqno); 17651b13d190SFrançois Tigeot } 17661b13d190SFrançois Tigeot 1767352ff8bdSFrançois Tigeot static u32 bxt_a_get_seqno(struct intel_engine_cs *ring, bool lazy_coherency) 1768352ff8bdSFrançois Tigeot { 1769352ff8bdSFrançois Tigeot 1770352ff8bdSFrançois Tigeot /* 1771352ff8bdSFrançois Tigeot * On BXT A steppings there is a HW coherency issue whereby the 1772352ff8bdSFrançois Tigeot * MI_STORE_DATA_IMM storing the completed request's seqno 1773352ff8bdSFrançois Tigeot * occasionally doesn't invalidate the CPU cache. Work around this by 1774352ff8bdSFrançois Tigeot * clflushing the corresponding cacheline whenever the caller wants 1775352ff8bdSFrançois Tigeot * the coherency to be guaranteed. Note that this cacheline is known 1776352ff8bdSFrançois Tigeot * to be clean at this point, since we only write it in 1777352ff8bdSFrançois Tigeot * bxt_a_set_seqno(), where we also do a clflush after the write. So 1778352ff8bdSFrançois Tigeot * this clflush in practice becomes an invalidate operation. 1779352ff8bdSFrançois Tigeot */ 1780352ff8bdSFrançois Tigeot 1781352ff8bdSFrançois Tigeot if (!lazy_coherency) 1782352ff8bdSFrançois Tigeot intel_flush_status_page(ring, I915_GEM_HWS_INDEX); 1783352ff8bdSFrançois Tigeot 1784352ff8bdSFrançois Tigeot return intel_read_status_page(ring, I915_GEM_HWS_INDEX); 1785352ff8bdSFrançois Tigeot } 1786352ff8bdSFrançois Tigeot 1787352ff8bdSFrançois Tigeot static void bxt_a_set_seqno(struct intel_engine_cs *ring, u32 seqno) 1788352ff8bdSFrançois Tigeot { 1789352ff8bdSFrançois Tigeot intel_write_status_page(ring, I915_GEM_HWS_INDEX, seqno); 1790352ff8bdSFrançois Tigeot 1791352ff8bdSFrançois Tigeot /* See bxt_a_get_seqno() explaining the reason for the clflush. */ 1792352ff8bdSFrançois Tigeot intel_flush_status_page(ring, I915_GEM_HWS_INDEX); 1793352ff8bdSFrançois Tigeot } 1794352ff8bdSFrançois Tigeot 1795a05eeebfSFrançois Tigeot static int gen8_emit_request(struct drm_i915_gem_request *request) 17961b13d190SFrançois Tigeot { 1797a05eeebfSFrançois Tigeot struct intel_ringbuffer *ringbuf = request->ringbuf; 17981b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 17991b13d190SFrançois Tigeot u32 cmd; 18001b13d190SFrançois Tigeot int ret; 18011b13d190SFrançois Tigeot 1802477eb7f9SFrançois Tigeot /* 1803477eb7f9SFrançois Tigeot * Reserve space for 2 NOOPs at the end of each request to be 1804477eb7f9SFrançois Tigeot * used as a workaround for not being allowed to do lite 1805477eb7f9SFrançois Tigeot * restore with HEAD==TAIL (WaIdleLiteRestore). 1806477eb7f9SFrançois Tigeot */ 1807a05eeebfSFrançois Tigeot ret = intel_logical_ring_begin(request, 8); 18081b13d190SFrançois Tigeot if (ret) 18091b13d190SFrançois Tigeot return ret; 18101b13d190SFrançois Tigeot 18112c9916cdSFrançois Tigeot cmd = MI_STORE_DWORD_IMM_GEN4; 18121b13d190SFrançois Tigeot cmd |= MI_GLOBAL_GTT; 18131b13d190SFrançois Tigeot 18141b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, cmd); 18151b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 18161b13d190SFrançois Tigeot (ring->status_page.gfx_addr + 18171b13d190SFrançois Tigeot (I915_GEM_HWS_INDEX << MI_STORE_DWORD_INDEX_SHIFT))); 18181b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1819a05eeebfSFrançois Tigeot intel_logical_ring_emit(ringbuf, i915_gem_request_get_seqno(request)); 18201b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_USER_INTERRUPT); 18211b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1822a05eeebfSFrançois Tigeot intel_logical_ring_advance_and_submit(request); 18231b13d190SFrançois Tigeot 1824477eb7f9SFrançois Tigeot /* 1825477eb7f9SFrançois Tigeot * Here we add two extra NOOPs as padding to avoid 1826477eb7f9SFrançois Tigeot * lite restore of a context with HEAD==TAIL. 1827477eb7f9SFrançois Tigeot */ 1828477eb7f9SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1829477eb7f9SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1830477eb7f9SFrançois Tigeot intel_logical_ring_advance(ringbuf); 1831477eb7f9SFrançois Tigeot 18321b13d190SFrançois Tigeot return 0; 18331b13d190SFrançois Tigeot } 18341b13d190SFrançois Tigeot 1835a05eeebfSFrançois Tigeot static int intel_lr_context_render_state_init(struct drm_i915_gem_request *req) 1836477eb7f9SFrançois Tigeot { 1837477eb7f9SFrançois Tigeot struct render_state so; 1838477eb7f9SFrançois Tigeot int ret; 1839477eb7f9SFrançois Tigeot 1840a05eeebfSFrançois Tigeot ret = i915_gem_render_state_prepare(req->ring, &so); 1841477eb7f9SFrançois Tigeot if (ret) 1842477eb7f9SFrançois Tigeot return ret; 1843477eb7f9SFrançois Tigeot 1844477eb7f9SFrançois Tigeot if (so.rodata == NULL) 1845477eb7f9SFrançois Tigeot return 0; 1846477eb7f9SFrançois Tigeot 1847a05eeebfSFrançois Tigeot ret = req->ring->emit_bb_start(req, so.ggtt_offset, 1848477eb7f9SFrançois Tigeot I915_DISPATCH_SECURE); 1849477eb7f9SFrançois Tigeot if (ret) 1850477eb7f9SFrançois Tigeot goto out; 1851477eb7f9SFrançois Tigeot 1852a05eeebfSFrançois Tigeot ret = req->ring->emit_bb_start(req, 1853a05eeebfSFrançois Tigeot (so.ggtt_offset + so.aux_batch_offset), 1854a05eeebfSFrançois Tigeot I915_DISPATCH_SECURE); 1855a05eeebfSFrançois Tigeot if (ret) 1856a05eeebfSFrançois Tigeot goto out; 1857477eb7f9SFrançois Tigeot 1858a05eeebfSFrançois Tigeot i915_vma_move_to_active(i915_gem_obj_to_ggtt(so.obj), req); 1859a05eeebfSFrançois Tigeot 1860477eb7f9SFrançois Tigeot out: 1861477eb7f9SFrançois Tigeot i915_gem_render_state_fini(&so); 1862477eb7f9SFrançois Tigeot return ret; 1863477eb7f9SFrançois Tigeot } 1864477eb7f9SFrançois Tigeot 1865a05eeebfSFrançois Tigeot static int gen8_init_rcs_context(struct drm_i915_gem_request *req) 18662c9916cdSFrançois Tigeot { 18672c9916cdSFrançois Tigeot int ret; 18682c9916cdSFrançois Tigeot 1869a05eeebfSFrançois Tigeot ret = intel_logical_ring_workarounds_emit(req); 18702c9916cdSFrançois Tigeot if (ret) 18712c9916cdSFrançois Tigeot return ret; 18722c9916cdSFrançois Tigeot 1873a05eeebfSFrançois Tigeot ret = intel_rcs_context_init_mocs(req); 1874a05eeebfSFrançois Tigeot /* 1875a05eeebfSFrançois Tigeot * Failing to program the MOCS is non-fatal.The system will not 1876a05eeebfSFrançois Tigeot * run at peak performance. So generate an error and carry on. 1877a05eeebfSFrançois Tigeot */ 1878a05eeebfSFrançois Tigeot if (ret) 1879a05eeebfSFrançois Tigeot DRM_ERROR("MOCS failed to program: expect performance issues.\n"); 1880a05eeebfSFrançois Tigeot 1881a05eeebfSFrançois Tigeot return intel_lr_context_render_state_init(req); 18822c9916cdSFrançois Tigeot } 18832c9916cdSFrançois Tigeot 18841b13d190SFrançois Tigeot /** 18851b13d190SFrançois Tigeot * intel_logical_ring_cleanup() - deallocate the Engine Command Streamer 18861b13d190SFrançois Tigeot * 18871b13d190SFrançois Tigeot * @ring: Engine Command Streamer. 18881b13d190SFrançois Tigeot * 18891b13d190SFrançois Tigeot */ 18901b13d190SFrançois Tigeot void intel_logical_ring_cleanup(struct intel_engine_cs *ring) 18911b13d190SFrançois Tigeot { 18922c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv; 18931b13d190SFrançois Tigeot 18941b13d190SFrançois Tigeot if (!intel_ring_initialized(ring)) 18951b13d190SFrançois Tigeot return; 18961b13d190SFrançois Tigeot 18972c9916cdSFrançois Tigeot dev_priv = ring->dev->dev_private; 18982c9916cdSFrançois Tigeot 1899*aee94f86SFrançois Tigeot if (ring->buffer) { 19001b13d190SFrançois Tigeot intel_logical_ring_stop(ring); 19011b13d190SFrançois Tigeot WARN_ON((I915_READ_MODE(ring) & MODE_IDLE) == 0); 1902*aee94f86SFrançois Tigeot } 19031b13d190SFrançois Tigeot 19041b13d190SFrançois Tigeot if (ring->cleanup) 19051b13d190SFrançois Tigeot ring->cleanup(ring); 19061b13d190SFrançois Tigeot 19071b13d190SFrançois Tigeot i915_cmd_parser_fini_ring(ring); 190819c468b4SFrançois Tigeot i915_gem_batch_pool_fini(&ring->batch_pool); 19091b13d190SFrançois Tigeot 19101b13d190SFrançois Tigeot if (ring->status_page.obj) { 19117ec9f8e5SFrançois Tigeot kunmap(sg_page(ring->status_page.obj->pages->sgl)); 19121b13d190SFrançois Tigeot ring->status_page.obj = NULL; 19131b13d190SFrançois Tigeot } 1914a05eeebfSFrançois Tigeot 1915a05eeebfSFrançois Tigeot lrc_destroy_wa_ctx_obj(ring); 1916*aee94f86SFrançois Tigeot ring->dev = NULL; 19171b13d190SFrançois Tigeot } 19181b13d190SFrançois Tigeot 19191b13d190SFrançois Tigeot static int logical_ring_init(struct drm_device *dev, struct intel_engine_cs *ring) 19201b13d190SFrançois Tigeot { 19211b13d190SFrançois Tigeot int ret; 19221b13d190SFrançois Tigeot 19231b13d190SFrançois Tigeot /* Intentionally left blank. */ 19241b13d190SFrançois Tigeot ring->buffer = NULL; 19251b13d190SFrançois Tigeot 19261b13d190SFrançois Tigeot ring->dev = dev; 19271b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->active_list); 19281b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->request_list); 192919c468b4SFrançois Tigeot i915_gem_batch_pool_init(dev, &ring->batch_pool); 19301b13d190SFrançois Tigeot init_waitqueue_head(&ring->irq_queue); 19311b13d190SFrançois Tigeot 1932*aee94f86SFrançois Tigeot INIT_LIST_HEAD(&ring->buffers); 19331b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->execlist_queue); 19342c9916cdSFrançois Tigeot INIT_LIST_HEAD(&ring->execlist_retired_req_list); 19351b13d190SFrançois Tigeot lockinit(&ring->execlist_lock, "i915el", 0, LK_CANRECURSE); 19361b13d190SFrançois Tigeot 19371b13d190SFrançois Tigeot ret = i915_cmd_parser_init_ring(ring); 19381b13d190SFrançois Tigeot if (ret) 1939*aee94f86SFrançois Tigeot goto error; 19401b13d190SFrançois Tigeot 1941352ff8bdSFrançois Tigeot ret = intel_lr_context_deferred_alloc(ring->default_context, ring); 1942352ff8bdSFrançois Tigeot if (ret) 1943*aee94f86SFrançois Tigeot goto error; 1944352ff8bdSFrançois Tigeot 1945352ff8bdSFrançois Tigeot /* As this is the default context, always pin it */ 1946352ff8bdSFrançois Tigeot ret = intel_lr_context_do_pin( 1947352ff8bdSFrançois Tigeot ring, 1948352ff8bdSFrançois Tigeot ring->default_context->engine[ring->id].state, 1949352ff8bdSFrançois Tigeot ring->default_context->engine[ring->id].ringbuf); 1950352ff8bdSFrançois Tigeot if (ret) { 1951352ff8bdSFrançois Tigeot DRM_ERROR( 1952352ff8bdSFrançois Tigeot "Failed to pin and map ringbuffer %s: %d\n", 1953352ff8bdSFrançois Tigeot ring->name, ret); 1954*aee94f86SFrançois Tigeot goto error; 1955352ff8bdSFrançois Tigeot } 19561b13d190SFrançois Tigeot 1957*aee94f86SFrançois Tigeot return 0; 1958*aee94f86SFrançois Tigeot 1959*aee94f86SFrançois Tigeot error: 1960*aee94f86SFrançois Tigeot intel_logical_ring_cleanup(ring); 19611b13d190SFrançois Tigeot return ret; 19621b13d190SFrançois Tigeot } 19631b13d190SFrançois Tigeot 19641b13d190SFrançois Tigeot static int logical_render_ring_init(struct drm_device *dev) 19651b13d190SFrançois Tigeot { 19661b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 19671b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[RCS]; 19682c9916cdSFrançois Tigeot int ret; 19691b13d190SFrançois Tigeot 19701b13d190SFrançois Tigeot ring->name = "render ring"; 19711b13d190SFrançois Tigeot ring->id = RCS; 19721b13d190SFrançois Tigeot ring->mmio_base = RENDER_RING_BASE; 19731b13d190SFrançois Tigeot ring->irq_enable_mask = 19741b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_RCS_IRQ_SHIFT; 19751b13d190SFrançois Tigeot ring->irq_keep_mask = 19761b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_RCS_IRQ_SHIFT; 19771b13d190SFrançois Tigeot if (HAS_L3_DPF(dev)) 19781b13d190SFrançois Tigeot ring->irq_keep_mask |= GT_RENDER_L3_PARITY_ERROR_INTERRUPT; 19791b13d190SFrançois Tigeot 1980477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->gen >= 9) 1981477eb7f9SFrançois Tigeot ring->init_hw = gen9_init_render_ring; 1982477eb7f9SFrançois Tigeot else 19832c9916cdSFrançois Tigeot ring->init_hw = gen8_init_render_ring; 19842c9916cdSFrançois Tigeot ring->init_context = gen8_init_rcs_context; 19851b13d190SFrançois Tigeot ring->cleanup = intel_fini_pipe_control; 1986*aee94f86SFrançois Tigeot if (IS_BXT_REVID(dev, 0, BXT_REVID_A1)) { 1987352ff8bdSFrançois Tigeot ring->get_seqno = bxt_a_get_seqno; 1988352ff8bdSFrançois Tigeot ring->set_seqno = bxt_a_set_seqno; 1989352ff8bdSFrançois Tigeot } else { 19901b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 19911b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 1992352ff8bdSFrançois Tigeot } 19931b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 19941b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush_render; 19951b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 19961b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 19971b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 19981b13d190SFrançois Tigeot 19992c9916cdSFrançois Tigeot ring->dev = dev; 2000a05eeebfSFrançois Tigeot 2001a05eeebfSFrançois Tigeot ret = intel_init_pipe_control(ring); 20022c9916cdSFrançois Tigeot if (ret) 20032c9916cdSFrançois Tigeot return ret; 20042c9916cdSFrançois Tigeot 2005a05eeebfSFrançois Tigeot ret = intel_init_workaround_bb(ring); 2006a05eeebfSFrançois Tigeot if (ret) { 2007a05eeebfSFrançois Tigeot /* 2008a05eeebfSFrançois Tigeot * We continue even if we fail to initialize WA batch 2009a05eeebfSFrançois Tigeot * because we only expect rare glitches but nothing 2010a05eeebfSFrançois Tigeot * critical to prevent us from using GPU 2011a05eeebfSFrançois Tigeot */ 2012a05eeebfSFrançois Tigeot DRM_ERROR("WA batch buffer initialization failed: %d\n", 2013a05eeebfSFrançois Tigeot ret); 2014a05eeebfSFrançois Tigeot } 2015a05eeebfSFrançois Tigeot 2016a05eeebfSFrançois Tigeot ret = logical_ring_init(dev, ring); 2017a05eeebfSFrançois Tigeot if (ret) { 2018a05eeebfSFrançois Tigeot lrc_destroy_wa_ctx_obj(ring); 2019a05eeebfSFrançois Tigeot } 2020a05eeebfSFrançois Tigeot 2021a05eeebfSFrançois Tigeot return ret; 20221b13d190SFrançois Tigeot } 20231b13d190SFrançois Tigeot 20241b13d190SFrançois Tigeot static int logical_bsd_ring_init(struct drm_device *dev) 20251b13d190SFrançois Tigeot { 20261b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 20271b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VCS]; 20281b13d190SFrançois Tigeot 20291b13d190SFrançois Tigeot ring->name = "bsd ring"; 20301b13d190SFrançois Tigeot ring->id = VCS; 20311b13d190SFrançois Tigeot ring->mmio_base = GEN6_BSD_RING_BASE; 20321b13d190SFrançois Tigeot ring->irq_enable_mask = 20331b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VCS1_IRQ_SHIFT; 20341b13d190SFrançois Tigeot ring->irq_keep_mask = 20351b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS1_IRQ_SHIFT; 20361b13d190SFrançois Tigeot 20372c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 2038*aee94f86SFrançois Tigeot if (IS_BXT_REVID(dev, 0, BXT_REVID_A1)) { 2039352ff8bdSFrançois Tigeot ring->get_seqno = bxt_a_get_seqno; 2040352ff8bdSFrançois Tigeot ring->set_seqno = bxt_a_set_seqno; 2041352ff8bdSFrançois Tigeot } else { 20421b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 20431b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 2044352ff8bdSFrançois Tigeot } 20451b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 20461b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 20471b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 20481b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 20491b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 20501b13d190SFrançois Tigeot 20511b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 20521b13d190SFrançois Tigeot } 20531b13d190SFrançois Tigeot 20541b13d190SFrançois Tigeot static int logical_bsd2_ring_init(struct drm_device *dev) 20551b13d190SFrançois Tigeot { 20561b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 20571b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VCS2]; 20581b13d190SFrançois Tigeot 20591b13d190SFrançois Tigeot ring->name = "bds2 ring"; 20601b13d190SFrançois Tigeot ring->id = VCS2; 20611b13d190SFrançois Tigeot ring->mmio_base = GEN8_BSD2_RING_BASE; 20621b13d190SFrançois Tigeot ring->irq_enable_mask = 20631b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VCS2_IRQ_SHIFT; 20641b13d190SFrançois Tigeot ring->irq_keep_mask = 20651b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS2_IRQ_SHIFT; 20661b13d190SFrançois Tigeot 20672c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 20681b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 20691b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 20701b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 20711b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 20721b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 20731b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 20741b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 20751b13d190SFrançois Tigeot 20761b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 20771b13d190SFrançois Tigeot } 20781b13d190SFrançois Tigeot 20791b13d190SFrançois Tigeot static int logical_blt_ring_init(struct drm_device *dev) 20801b13d190SFrançois Tigeot { 20811b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 20821b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[BCS]; 20831b13d190SFrançois Tigeot 20841b13d190SFrançois Tigeot ring->name = "blitter ring"; 20851b13d190SFrançois Tigeot ring->id = BCS; 20861b13d190SFrançois Tigeot ring->mmio_base = BLT_RING_BASE; 20871b13d190SFrançois Tigeot ring->irq_enable_mask = 20881b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_BCS_IRQ_SHIFT; 20891b13d190SFrançois Tigeot ring->irq_keep_mask = 20901b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_BCS_IRQ_SHIFT; 20911b13d190SFrançois Tigeot 20922c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 2093*aee94f86SFrançois Tigeot if (IS_BXT_REVID(dev, 0, BXT_REVID_A1)) { 2094352ff8bdSFrançois Tigeot ring->get_seqno = bxt_a_get_seqno; 2095352ff8bdSFrançois Tigeot ring->set_seqno = bxt_a_set_seqno; 2096352ff8bdSFrançois Tigeot } else { 20971b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 20981b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 2099352ff8bdSFrançois Tigeot } 21001b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 21011b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 21021b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 21031b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 21041b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 21051b13d190SFrançois Tigeot 21061b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 21071b13d190SFrançois Tigeot } 21081b13d190SFrançois Tigeot 21091b13d190SFrançois Tigeot static int logical_vebox_ring_init(struct drm_device *dev) 21101b13d190SFrançois Tigeot { 21111b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 21121b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VECS]; 21131b13d190SFrançois Tigeot 21141b13d190SFrançois Tigeot ring->name = "video enhancement ring"; 21151b13d190SFrançois Tigeot ring->id = VECS; 21161b13d190SFrançois Tigeot ring->mmio_base = VEBOX_RING_BASE; 21171b13d190SFrançois Tigeot ring->irq_enable_mask = 21181b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VECS_IRQ_SHIFT; 21191b13d190SFrançois Tigeot ring->irq_keep_mask = 21201b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VECS_IRQ_SHIFT; 21211b13d190SFrançois Tigeot 21222c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 2123*aee94f86SFrançois Tigeot if (IS_BXT_REVID(dev, 0, BXT_REVID_A1)) { 2124352ff8bdSFrançois Tigeot ring->get_seqno = bxt_a_get_seqno; 2125352ff8bdSFrançois Tigeot ring->set_seqno = bxt_a_set_seqno; 2126352ff8bdSFrançois Tigeot } else { 21271b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 21281b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 2129352ff8bdSFrançois Tigeot } 21301b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 21311b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 21321b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 21331b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 21341b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 21351b13d190SFrançois Tigeot 21361b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 21371b13d190SFrançois Tigeot } 21381b13d190SFrançois Tigeot 21391b13d190SFrançois Tigeot /** 21401b13d190SFrançois Tigeot * intel_logical_rings_init() - allocate, populate and init the Engine Command Streamers 21411b13d190SFrançois Tigeot * @dev: DRM device. 21421b13d190SFrançois Tigeot * 21431b13d190SFrançois Tigeot * This function inits the engines for an Execlists submission style (the equivalent in the 21441b13d190SFrançois Tigeot * legacy ringbuffer submission world would be i915_gem_init_rings). It does it only for 21451b13d190SFrançois Tigeot * those engines that are present in the hardware. 21461b13d190SFrançois Tigeot * 21471b13d190SFrançois Tigeot * Return: non-zero if the initialization failed. 21481b13d190SFrançois Tigeot */ 21491b13d190SFrançois Tigeot int intel_logical_rings_init(struct drm_device *dev) 21501b13d190SFrançois Tigeot { 21511b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 21521b13d190SFrançois Tigeot int ret; 21531b13d190SFrançois Tigeot 21541b13d190SFrançois Tigeot ret = logical_render_ring_init(dev); 21551b13d190SFrançois Tigeot if (ret) 21561b13d190SFrançois Tigeot return ret; 21571b13d190SFrançois Tigeot 21581b13d190SFrançois Tigeot if (HAS_BSD(dev)) { 21591b13d190SFrançois Tigeot ret = logical_bsd_ring_init(dev); 21601b13d190SFrançois Tigeot if (ret) 21611b13d190SFrançois Tigeot goto cleanup_render_ring; 21621b13d190SFrançois Tigeot } 21631b13d190SFrançois Tigeot 21641b13d190SFrançois Tigeot if (HAS_BLT(dev)) { 21651b13d190SFrançois Tigeot ret = logical_blt_ring_init(dev); 21661b13d190SFrançois Tigeot if (ret) 21671b13d190SFrançois Tigeot goto cleanup_bsd_ring; 21681b13d190SFrançois Tigeot } 21691b13d190SFrançois Tigeot 21701b13d190SFrançois Tigeot if (HAS_VEBOX(dev)) { 21711b13d190SFrançois Tigeot ret = logical_vebox_ring_init(dev); 21721b13d190SFrançois Tigeot if (ret) 21731b13d190SFrançois Tigeot goto cleanup_blt_ring; 21741b13d190SFrançois Tigeot } 21751b13d190SFrançois Tigeot 21761b13d190SFrançois Tigeot if (HAS_BSD2(dev)) { 21771b13d190SFrançois Tigeot ret = logical_bsd2_ring_init(dev); 21781b13d190SFrançois Tigeot if (ret) 21791b13d190SFrançois Tigeot goto cleanup_vebox_ring; 21801b13d190SFrançois Tigeot } 21811b13d190SFrançois Tigeot 21821b13d190SFrançois Tigeot return 0; 21831b13d190SFrançois Tigeot 21841b13d190SFrançois Tigeot cleanup_vebox_ring: 21851b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[VECS]); 21861b13d190SFrançois Tigeot cleanup_blt_ring: 21871b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[BCS]); 21881b13d190SFrançois Tigeot cleanup_bsd_ring: 21891b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[VCS]); 21901b13d190SFrançois Tigeot cleanup_render_ring: 21911b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[RCS]); 21921b13d190SFrançois Tigeot 21931b13d190SFrançois Tigeot return ret; 21941b13d190SFrançois Tigeot } 21951b13d190SFrançois Tigeot 2196477eb7f9SFrançois Tigeot static u32 2197477eb7f9SFrançois Tigeot make_rpcs(struct drm_device *dev) 21981b13d190SFrançois Tigeot { 2199477eb7f9SFrançois Tigeot u32 rpcs = 0; 22001b13d190SFrançois Tigeot 2201477eb7f9SFrançois Tigeot /* 2202477eb7f9SFrançois Tigeot * No explicit RPCS request is needed to ensure full 2203477eb7f9SFrançois Tigeot * slice/subslice/EU enablement prior to Gen9. 2204477eb7f9SFrançois Tigeot */ 2205477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->gen < 9) 22061b13d190SFrançois Tigeot return 0; 22071b13d190SFrançois Tigeot 2208477eb7f9SFrançois Tigeot /* 2209477eb7f9SFrançois Tigeot * Starting in Gen9, render power gating can leave 2210477eb7f9SFrançois Tigeot * slice/subslice/EU in a partially enabled state. We 2211477eb7f9SFrançois Tigeot * must make an explicit request through RPCS for full 2212477eb7f9SFrançois Tigeot * enablement. 2213477eb7f9SFrançois Tigeot */ 2214477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_slice_pg) { 2215477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_S_CNT_ENABLE; 2216477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->slice_total << 2217477eb7f9SFrançois Tigeot GEN8_RPCS_S_CNT_SHIFT; 2218477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2219477eb7f9SFrançois Tigeot } 22201b13d190SFrançois Tigeot 2221477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_subslice_pg) { 2222477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_SS_CNT_ENABLE; 2223477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->subslice_per_slice << 2224477eb7f9SFrançois Tigeot GEN8_RPCS_SS_CNT_SHIFT; 2225477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2226477eb7f9SFrançois Tigeot } 22271b13d190SFrançois Tigeot 2228477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_eu_pg) { 2229477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->eu_per_subslice << 2230477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MIN_SHIFT; 2231477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->eu_per_subslice << 2232477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MAX_SHIFT; 2233477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 2234477eb7f9SFrançois Tigeot } 2235477eb7f9SFrançois Tigeot 2236477eb7f9SFrançois Tigeot return rpcs; 22371b13d190SFrançois Tigeot } 22381b13d190SFrançois Tigeot 22391b13d190SFrançois Tigeot static int 22401b13d190SFrançois Tigeot populate_lr_context(struct intel_context *ctx, struct drm_i915_gem_object *ctx_obj, 22411b13d190SFrançois Tigeot struct intel_engine_cs *ring, struct intel_ringbuffer *ringbuf) 22421b13d190SFrançois Tigeot { 22431b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 22441b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 22451b13d190SFrançois Tigeot struct i915_hw_ppgtt *ppgtt = ctx->ppgtt; 22461b13d190SFrançois Tigeot struct vm_page *page; 22471b13d190SFrançois Tigeot uint32_t *reg_state; 22481b13d190SFrançois Tigeot int ret; 22491b13d190SFrançois Tigeot 22501b13d190SFrançois Tigeot if (!ppgtt) 22511b13d190SFrançois Tigeot ppgtt = dev_priv->mm.aliasing_ppgtt; 22521b13d190SFrançois Tigeot 22531b13d190SFrançois Tigeot ret = i915_gem_object_set_to_cpu_domain(ctx_obj, true); 22541b13d190SFrançois Tigeot if (ret) { 22551b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Could not set to CPU domain\n"); 22561b13d190SFrançois Tigeot return ret; 22571b13d190SFrançois Tigeot } 22581b13d190SFrançois Tigeot 22591b13d190SFrançois Tigeot ret = i915_gem_object_get_pages(ctx_obj); 22601b13d190SFrançois Tigeot if (ret) { 22611b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Could not get object pages\n"); 22621b13d190SFrançois Tigeot return ret; 22631b13d190SFrançois Tigeot } 22641b13d190SFrançois Tigeot 22651b13d190SFrançois Tigeot i915_gem_object_pin_pages(ctx_obj); 22661b13d190SFrançois Tigeot 22671b13d190SFrançois Tigeot /* The second page of the context object contains some fields which must 22681b13d190SFrançois Tigeot * be set up prior to the first execution. */ 2269*aee94f86SFrançois Tigeot page = i915_gem_object_get_dirty_page(ctx_obj, LRC_STATE_PN); 22701b13d190SFrançois Tigeot reg_state = kmap_atomic(page); 22711b13d190SFrançois Tigeot 22721b13d190SFrançois Tigeot /* A context is actually a big batch buffer with several MI_LOAD_REGISTER_IMM 22731b13d190SFrançois Tigeot * commands followed by (reg, value) pairs. The values we are setting here are 22741b13d190SFrançois Tigeot * only for the first context restore: on a subsequent save, the GPU will 22751b13d190SFrançois Tigeot * recreate this batchbuffer with new values (including all the missing 22761b13d190SFrançois Tigeot * MI_LOAD_REGISTER_IMM commands that we are not initializing here). */ 2277*aee94f86SFrançois Tigeot reg_state[CTX_LRI_HEADER_0] = 2278*aee94f86SFrançois Tigeot MI_LOAD_REGISTER_IMM(ring->id == RCS ? 14 : 11) | MI_LRI_FORCE_POSTED; 2279*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_CONTEXT_CONTROL, RING_CONTEXT_CONTROL(ring), 2280477eb7f9SFrançois Tigeot _MASKED_BIT_ENABLE(CTX_CTRL_INHIBIT_SYN_CTX_SWITCH | 2281a05eeebfSFrançois Tigeot CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT | 2282*aee94f86SFrançois Tigeot CTX_CTRL_RS_CTX_ENABLE)); 2283*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_HEAD, RING_HEAD(ring->mmio_base), 0); 2284*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_TAIL, RING_TAIL(ring->mmio_base), 0); 22852c9916cdSFrançois Tigeot /* Ring buffer start address is not known until the buffer is pinned. 22862c9916cdSFrançois Tigeot * It is written to the context image in execlists_update_context() 22872c9916cdSFrançois Tigeot */ 2288*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_BUFFER_START, RING_START(ring->mmio_base), 0); 2289*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RING_BUFFER_CONTROL, RING_CTL(ring->mmio_base), 2290*aee94f86SFrançois Tigeot ((ringbuf->size - PAGE_SIZE) & RING_NR_PAGES) | RING_VALID); 2291*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_HEAD_U, RING_BBADDR_UDW(ring->mmio_base), 0); 2292*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_HEAD_L, RING_BBADDR(ring->mmio_base), 0); 2293*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_STATE, RING_BBSTATE(ring->mmio_base), 2294*aee94f86SFrançois Tigeot RING_BB_PPGTT); 2295*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_HEAD_U, RING_SBBADDR_UDW(ring->mmio_base), 0); 2296*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_HEAD_L, RING_SBBADDR(ring->mmio_base), 0); 2297*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_SECOND_BB_STATE, RING_SBBSTATE(ring->mmio_base), 0); 22981b13d190SFrançois Tigeot if (ring->id == RCS) { 2299*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_BB_PER_CTX_PTR, RING_BB_PER_CTX_PTR(ring->mmio_base), 0); 2300*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RCS_INDIRECT_CTX, RING_INDIRECT_CTX(ring->mmio_base), 0); 2301*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_RCS_INDIRECT_CTX_OFFSET, RING_INDIRECT_CTX_OFFSET(ring->mmio_base), 0); 2302a05eeebfSFrançois Tigeot if (ring->wa_ctx.obj) { 2303a05eeebfSFrançois Tigeot struct i915_ctx_workarounds *wa_ctx = &ring->wa_ctx; 2304a05eeebfSFrançois Tigeot uint32_t ggtt_offset = i915_gem_obj_ggtt_offset(wa_ctx->obj); 2305a05eeebfSFrançois Tigeot 2306a05eeebfSFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX+1] = 2307a05eeebfSFrançois Tigeot (ggtt_offset + wa_ctx->indirect_ctx.offset * sizeof(uint32_t)) | 2308a05eeebfSFrançois Tigeot (wa_ctx->indirect_ctx.size / CACHELINE_DWORDS); 2309a05eeebfSFrançois Tigeot 2310a05eeebfSFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX_OFFSET+1] = 2311a05eeebfSFrançois Tigeot CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT << 6; 2312a05eeebfSFrançois Tigeot 2313a05eeebfSFrançois Tigeot reg_state[CTX_BB_PER_CTX_PTR+1] = 2314a05eeebfSFrançois Tigeot (ggtt_offset + wa_ctx->per_ctx.offset * sizeof(uint32_t)) | 2315a05eeebfSFrançois Tigeot 0x01; 2316a05eeebfSFrançois Tigeot } 23171b13d190SFrançois Tigeot } 2318*aee94f86SFrançois Tigeot reg_state[CTX_LRI_HEADER_1] = MI_LOAD_REGISTER_IMM(9) | MI_LRI_FORCE_POSTED; 2319*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_CTX_TIMESTAMP, RING_CTX_TIMESTAMP(ring->mmio_base), 0); 2320*aee94f86SFrançois Tigeot /* PDP values well be assigned later if needed */ 2321*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP3_UDW, GEN8_RING_PDP_UDW(ring, 3), 0); 2322*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP3_LDW, GEN8_RING_PDP_LDW(ring, 3), 0); 2323*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP2_UDW, GEN8_RING_PDP_UDW(ring, 2), 0); 2324*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP2_LDW, GEN8_RING_PDP_LDW(ring, 2), 0); 2325*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP1_UDW, GEN8_RING_PDP_UDW(ring, 1), 0); 2326*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP1_LDW, GEN8_RING_PDP_LDW(ring, 1), 0); 2327*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP0_UDW, GEN8_RING_PDP_UDW(ring, 0), 0); 2328*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_PDP0_LDW, GEN8_RING_PDP_LDW(ring, 0), 0); 232919c468b4SFrançois Tigeot 2330352ff8bdSFrançois Tigeot if (USES_FULL_48BIT_PPGTT(ppgtt->base.dev)) { 2331352ff8bdSFrançois Tigeot /* 64b PPGTT (48bit canonical) 2332352ff8bdSFrançois Tigeot * PDP0_DESCRIPTOR contains the base address to PML4 and 2333352ff8bdSFrançois Tigeot * other PDP Descriptors are ignored. 2334352ff8bdSFrançois Tigeot */ 2335352ff8bdSFrançois Tigeot ASSIGN_CTX_PML4(ppgtt, reg_state); 2336352ff8bdSFrançois Tigeot } else { 2337352ff8bdSFrançois Tigeot /* 32b PPGTT 2338352ff8bdSFrançois Tigeot * PDP*_DESCRIPTOR contains the base address of space supported. 2339352ff8bdSFrançois Tigeot * With dynamic page allocation, PDPs may not be allocated at 2340352ff8bdSFrançois Tigeot * this point. Point the unallocated PDPs to the scratch page 234119c468b4SFrançois Tigeot */ 234219c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 3); 234319c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 2); 234419c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 1); 234519c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 0); 2346352ff8bdSFrançois Tigeot } 2347352ff8bdSFrançois Tigeot 23481b13d190SFrançois Tigeot if (ring->id == RCS) { 23491b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1); 2350*aee94f86SFrançois Tigeot ASSIGN_CTX_REG(reg_state, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE, 2351*aee94f86SFrançois Tigeot make_rpcs(dev)); 23521b13d190SFrançois Tigeot } 23531b13d190SFrançois Tigeot 23541b13d190SFrançois Tigeot kunmap_atomic(reg_state); 23551b13d190SFrançois Tigeot i915_gem_object_unpin_pages(ctx_obj); 23561b13d190SFrançois Tigeot 23571b13d190SFrançois Tigeot return 0; 23581b13d190SFrançois Tigeot } 23591b13d190SFrançois Tigeot 23601b13d190SFrançois Tigeot /** 23611b13d190SFrançois Tigeot * intel_lr_context_free() - free the LRC specific bits of a context 23621b13d190SFrançois Tigeot * @ctx: the LR context to free. 23631b13d190SFrançois Tigeot * 23641b13d190SFrançois Tigeot * The real context freeing is done in i915_gem_context_free: this only 23651b13d190SFrançois Tigeot * takes care of the bits that are LRC related: the per-engine backing 23661b13d190SFrançois Tigeot * objects and the logical ringbuffer. 23671b13d190SFrançois Tigeot */ 23681b13d190SFrançois Tigeot void intel_lr_context_free(struct intel_context *ctx) 23691b13d190SFrançois Tigeot { 23701b13d190SFrançois Tigeot int i; 23711b13d190SFrançois Tigeot 23721b13d190SFrançois Tigeot for (i = 0; i < I915_NUM_RINGS; i++) { 23731b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ctx->engine[i].state; 23741b13d190SFrançois Tigeot 23751b13d190SFrançois Tigeot if (ctx_obj) { 23762c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf = 23772c9916cdSFrançois Tigeot ctx->engine[i].ringbuf; 23782c9916cdSFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 23792c9916cdSFrançois Tigeot 23802c9916cdSFrançois Tigeot if (ctx == ring->default_context) { 23812c9916cdSFrançois Tigeot intel_unpin_ringbuffer_obj(ringbuf); 23822c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 23832c9916cdSFrançois Tigeot } 23842c9916cdSFrançois Tigeot WARN_ON(ctx->engine[ring->id].pin_count); 2385352ff8bdSFrançois Tigeot intel_ringbuffer_free(ringbuf); 23861b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 23871b13d190SFrançois Tigeot } 23881b13d190SFrançois Tigeot } 23891b13d190SFrançois Tigeot } 23901b13d190SFrançois Tigeot 23911b13d190SFrançois Tigeot static uint32_t get_lr_context_size(struct intel_engine_cs *ring) 23921b13d190SFrançois Tigeot { 23931b13d190SFrançois Tigeot int ret = 0; 23941b13d190SFrançois Tigeot 23952c9916cdSFrançois Tigeot WARN_ON(INTEL_INFO(ring->dev)->gen < 8); 23961b13d190SFrançois Tigeot 23971b13d190SFrançois Tigeot switch (ring->id) { 23981b13d190SFrançois Tigeot case RCS: 23992c9916cdSFrançois Tigeot if (INTEL_INFO(ring->dev)->gen >= 9) 24002c9916cdSFrançois Tigeot ret = GEN9_LR_CONTEXT_RENDER_SIZE; 24012c9916cdSFrançois Tigeot else 24021b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_RENDER_SIZE; 24031b13d190SFrançois Tigeot break; 24041b13d190SFrançois Tigeot case VCS: 24051b13d190SFrançois Tigeot case BCS: 24061b13d190SFrançois Tigeot case VECS: 24071b13d190SFrançois Tigeot case VCS2: 24081b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_OTHER_SIZE; 24091b13d190SFrançois Tigeot break; 24101b13d190SFrançois Tigeot } 24111b13d190SFrançois Tigeot 24121b13d190SFrançois Tigeot return ret; 24131b13d190SFrançois Tigeot } 24141b13d190SFrançois Tigeot 24152c9916cdSFrançois Tigeot static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring, 24162c9916cdSFrançois Tigeot struct drm_i915_gem_object *default_ctx_obj) 24172c9916cdSFrançois Tigeot { 24182c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 2419352ff8bdSFrançois Tigeot struct vm_page *page; 24202c9916cdSFrançois Tigeot 2421352ff8bdSFrançois Tigeot /* The HWSP is part of the default context object in LRC mode. */ 2422352ff8bdSFrançois Tigeot ring->status_page.gfx_addr = i915_gem_obj_ggtt_offset(default_ctx_obj) 2423352ff8bdSFrançois Tigeot + LRC_PPHWSP_PN * PAGE_SIZE; 2424352ff8bdSFrançois Tigeot page = i915_gem_object_get_page(default_ctx_obj, LRC_PPHWSP_PN); 2425352ff8bdSFrançois Tigeot ring->status_page.page_addr = kmap(page); 24262c9916cdSFrançois Tigeot ring->status_page.obj = default_ctx_obj; 24272c9916cdSFrançois Tigeot 24282c9916cdSFrançois Tigeot I915_WRITE(RING_HWS_PGA(ring->mmio_base), 24292c9916cdSFrançois Tigeot (u32)ring->status_page.gfx_addr); 24302c9916cdSFrançois Tigeot POSTING_READ(RING_HWS_PGA(ring->mmio_base)); 24312c9916cdSFrançois Tigeot } 24322c9916cdSFrançois Tigeot 24331b13d190SFrançois Tigeot /** 2434352ff8bdSFrançois Tigeot * intel_lr_context_deferred_alloc() - create the LRC specific bits of a context 24351b13d190SFrançois Tigeot * @ctx: LR context to create. 24361b13d190SFrançois Tigeot * @ring: engine to be used with the context. 24371b13d190SFrançois Tigeot * 24381b13d190SFrançois Tigeot * This function can be called more than once, with different engines, if we plan 24391b13d190SFrançois Tigeot * to use the context with them. The context backing objects and the ringbuffers 24401b13d190SFrançois Tigeot * (specially the ringbuffer backing objects) suck a lot of memory up, and that's why 24411b13d190SFrançois Tigeot * the creation is a deferred call: it's better to make sure first that we need to use 24421b13d190SFrançois Tigeot * a given ring with the context. 24431b13d190SFrançois Tigeot * 24442c9916cdSFrançois Tigeot * Return: non-zero on error. 24451b13d190SFrançois Tigeot */ 2446352ff8bdSFrançois Tigeot 2447352ff8bdSFrançois Tigeot int intel_lr_context_deferred_alloc(struct intel_context *ctx, 24481b13d190SFrançois Tigeot struct intel_engine_cs *ring) 24491b13d190SFrançois Tigeot { 24501b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 24511b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj; 24521b13d190SFrançois Tigeot uint32_t context_size; 24531b13d190SFrançois Tigeot struct intel_ringbuffer *ringbuf; 24541b13d190SFrançois Tigeot int ret; 24551b13d190SFrançois Tigeot 24561b13d190SFrançois Tigeot WARN_ON(ctx->legacy_hw_ctx.rcs_state != NULL); 24572c9916cdSFrançois Tigeot WARN_ON(ctx->engine[ring->id].state); 24581b13d190SFrançois Tigeot 24591b13d190SFrançois Tigeot context_size = round_up(get_lr_context_size(ring), 4096); 24601b13d190SFrançois Tigeot 2461352ff8bdSFrançois Tigeot /* One extra page as the sharing data between driver and GuC */ 2462352ff8bdSFrançois Tigeot context_size += PAGE_SIZE * LRC_PPHWSP_PN; 2463352ff8bdSFrançois Tigeot 246419c468b4SFrançois Tigeot ctx_obj = i915_gem_alloc_object(dev, context_size); 246519c468b4SFrançois Tigeot if (!ctx_obj) { 246619c468b4SFrançois Tigeot DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n"); 246719c468b4SFrançois Tigeot return -ENOMEM; 24681b13d190SFrançois Tigeot } 24691b13d190SFrançois Tigeot 2470352ff8bdSFrançois Tigeot ringbuf = intel_engine_create_ringbuffer(ring, 4 * PAGE_SIZE); 2471352ff8bdSFrançois Tigeot if (IS_ERR(ringbuf)) { 2472352ff8bdSFrançois Tigeot ret = PTR_ERR(ringbuf); 2473352ff8bdSFrançois Tigeot goto error_deref_obj; 24741b13d190SFrançois Tigeot } 24751b13d190SFrançois Tigeot 24761b13d190SFrançois Tigeot ret = populate_lr_context(ctx, ctx_obj, ring, ringbuf); 24771b13d190SFrançois Tigeot if (ret) { 24781b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Failed to populate LRC: %d\n", ret); 2479352ff8bdSFrançois Tigeot goto error_ringbuf; 24801b13d190SFrançois Tigeot } 24811b13d190SFrançois Tigeot 24821b13d190SFrançois Tigeot ctx->engine[ring->id].ringbuf = ringbuf; 24831b13d190SFrançois Tigeot ctx->engine[ring->id].state = ctx_obj; 24841b13d190SFrançois Tigeot 2485352ff8bdSFrançois Tigeot if (ctx != ring->default_context && ring->init_context) { 2486a05eeebfSFrançois Tigeot struct drm_i915_gem_request *req; 2487a05eeebfSFrançois Tigeot 2488352ff8bdSFrançois Tigeot ret = i915_gem_request_alloc(ring, 2489352ff8bdSFrançois Tigeot ctx, &req); 2490352ff8bdSFrançois Tigeot if (ret) { 2491352ff8bdSFrançois Tigeot DRM_ERROR("ring create req: %d\n", 2492352ff8bdSFrançois Tigeot ret); 2493352ff8bdSFrançois Tigeot goto error_ringbuf; 2494352ff8bdSFrançois Tigeot } 2495a05eeebfSFrançois Tigeot 2496a05eeebfSFrançois Tigeot ret = ring->init_context(req); 24971b13d190SFrançois Tigeot if (ret) { 2498352ff8bdSFrançois Tigeot DRM_ERROR("ring init context: %d\n", 2499352ff8bdSFrançois Tigeot ret); 2500a05eeebfSFrançois Tigeot i915_gem_request_cancel(req); 2501352ff8bdSFrançois Tigeot goto error_ringbuf; 25021b13d190SFrançois Tigeot } 2503a05eeebfSFrançois Tigeot i915_add_request_no_flush(req); 25042c9916cdSFrançois Tigeot } 25051b13d190SFrançois Tigeot return 0; 25061b13d190SFrançois Tigeot 2507352ff8bdSFrançois Tigeot error_ringbuf: 2508352ff8bdSFrançois Tigeot intel_ringbuffer_free(ringbuf); 2509352ff8bdSFrançois Tigeot error_deref_obj: 25101b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 2511352ff8bdSFrançois Tigeot ctx->engine[ring->id].ringbuf = NULL; 2512352ff8bdSFrançois Tigeot ctx->engine[ring->id].state = NULL; 25131b13d190SFrançois Tigeot return ret; 25141b13d190SFrançois Tigeot } 2515477eb7f9SFrançois Tigeot 2516477eb7f9SFrançois Tigeot void intel_lr_context_reset(struct drm_device *dev, 2517477eb7f9SFrançois Tigeot struct intel_context *ctx) 2518477eb7f9SFrançois Tigeot { 2519477eb7f9SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 2520477eb7f9SFrançois Tigeot struct intel_engine_cs *ring; 2521477eb7f9SFrançois Tigeot int i; 2522477eb7f9SFrançois Tigeot 2523477eb7f9SFrançois Tigeot for_each_ring(ring, dev_priv, i) { 2524477eb7f9SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 2525477eb7f9SFrançois Tigeot ctx->engine[ring->id].state; 2526477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf = 2527477eb7f9SFrançois Tigeot ctx->engine[ring->id].ringbuf; 2528477eb7f9SFrançois Tigeot uint32_t *reg_state; 2529477eb7f9SFrançois Tigeot struct vm_page *page; 2530477eb7f9SFrançois Tigeot 2531477eb7f9SFrançois Tigeot if (!ctx_obj) 2532477eb7f9SFrançois Tigeot continue; 2533477eb7f9SFrançois Tigeot 2534477eb7f9SFrançois Tigeot if (i915_gem_object_get_pages(ctx_obj)) { 2535477eb7f9SFrançois Tigeot WARN(1, "Failed get_pages for context obj\n"); 2536477eb7f9SFrançois Tigeot continue; 2537477eb7f9SFrançois Tigeot } 2538*aee94f86SFrançois Tigeot page = i915_gem_object_get_dirty_page(ctx_obj, LRC_STATE_PN); 2539477eb7f9SFrançois Tigeot reg_state = kmap_atomic(page); 2540477eb7f9SFrançois Tigeot 2541477eb7f9SFrançois Tigeot reg_state[CTX_RING_HEAD+1] = 0; 2542477eb7f9SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = 0; 2543477eb7f9SFrançois Tigeot 2544477eb7f9SFrançois Tigeot kunmap_atomic(reg_state); 2545477eb7f9SFrançois Tigeot 2546477eb7f9SFrançois Tigeot ringbuf->head = 0; 2547477eb7f9SFrançois Tigeot ringbuf->tail = 0; 2548477eb7f9SFrançois Tigeot } 2549477eb7f9SFrançois Tigeot } 2550