11b13d190SFrançois Tigeot /* 21b13d190SFrançois Tigeot * Copyright © 2014 Intel Corporation 31b13d190SFrançois Tigeot * 41b13d190SFrançois Tigeot * Permission is hereby granted, free of charge, to any person obtaining a 51b13d190SFrançois Tigeot * copy of this software and associated documentation files (the "Software"), 61b13d190SFrançois Tigeot * to deal in the Software without restriction, including without limitation 71b13d190SFrançois Tigeot * the rights to use, copy, modify, merge, publish, distribute, sublicense, 81b13d190SFrançois Tigeot * and/or sell copies of the Software, and to permit persons to whom the 91b13d190SFrançois Tigeot * Software is furnished to do so, subject to the following conditions: 101b13d190SFrançois Tigeot * 111b13d190SFrançois Tigeot * The above copyright notice and this permission notice (including the next 121b13d190SFrançois Tigeot * paragraph) shall be included in all copies or substantial portions of the 131b13d190SFrançois Tigeot * Software. 141b13d190SFrançois Tigeot * 151b13d190SFrançois Tigeot * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 161b13d190SFrançois Tigeot * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 171b13d190SFrançois Tigeot * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 181b13d190SFrançois Tigeot * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 191b13d190SFrançois Tigeot * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 201b13d190SFrançois Tigeot * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 211b13d190SFrançois Tigeot * IN THE SOFTWARE. 221b13d190SFrançois Tigeot * 231b13d190SFrançois Tigeot * Authors: 241b13d190SFrançois Tigeot * Ben Widawsky <ben@bwidawsk.net> 251b13d190SFrançois Tigeot * Michel Thierry <michel.thierry@intel.com> 261b13d190SFrançois Tigeot * Thomas Daniel <thomas.daniel@intel.com> 271b13d190SFrançois Tigeot * Oscar Mateo <oscar.mateo@intel.com> 281b13d190SFrançois Tigeot * 291b13d190SFrançois Tigeot */ 301b13d190SFrançois Tigeot 311b13d190SFrançois Tigeot /** 321b13d190SFrançois Tigeot * DOC: Logical Rings, Logical Ring Contexts and Execlists 331b13d190SFrançois Tigeot * 341b13d190SFrançois Tigeot * Motivation: 351b13d190SFrançois Tigeot * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts". 361b13d190SFrançois Tigeot * These expanded contexts enable a number of new abilities, especially 371b13d190SFrançois Tigeot * "Execlists" (also implemented in this file). 381b13d190SFrançois Tigeot * 391b13d190SFrançois Tigeot * One of the main differences with the legacy HW contexts is that logical 401b13d190SFrançois Tigeot * ring contexts incorporate many more things to the context's state, like 411b13d190SFrançois Tigeot * PDPs or ringbuffer control registers: 421b13d190SFrançois Tigeot * 431b13d190SFrançois Tigeot * The reason why PDPs are included in the context is straightforward: as 441b13d190SFrançois Tigeot * PPGTTs (per-process GTTs) are actually per-context, having the PDPs 451b13d190SFrançois Tigeot * contained there mean you don't need to do a ppgtt->switch_mm yourself, 461b13d190SFrançois Tigeot * instead, the GPU will do it for you on the context switch. 471b13d190SFrançois Tigeot * 481b13d190SFrançois Tigeot * But, what about the ringbuffer control registers (head, tail, etc..)? 491b13d190SFrançois Tigeot * shouldn't we just need a set of those per engine command streamer? This is 501b13d190SFrançois Tigeot * where the name "Logical Rings" starts to make sense: by virtualizing the 511b13d190SFrançois Tigeot * rings, the engine cs shifts to a new "ring buffer" with every context 521b13d190SFrançois Tigeot * switch. When you want to submit a workload to the GPU you: A) choose your 531b13d190SFrançois Tigeot * context, B) find its appropriate virtualized ring, C) write commands to it 541b13d190SFrançois Tigeot * and then, finally, D) tell the GPU to switch to that context. 551b13d190SFrançois Tigeot * 561b13d190SFrançois Tigeot * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch 571b13d190SFrançois Tigeot * to a contexts is via a context execution list, ergo "Execlists". 581b13d190SFrançois Tigeot * 591b13d190SFrançois Tigeot * LRC implementation: 601b13d190SFrançois Tigeot * Regarding the creation of contexts, we have: 611b13d190SFrançois Tigeot * 621b13d190SFrançois Tigeot * - One global default context. 631b13d190SFrançois Tigeot * - One local default context for each opened fd. 641b13d190SFrançois Tigeot * - One local extra context for each context create ioctl call. 651b13d190SFrançois Tigeot * 661b13d190SFrançois Tigeot * Now that ringbuffers belong per-context (and not per-engine, like before) 671b13d190SFrançois Tigeot * and that contexts are uniquely tied to a given engine (and not reusable, 681b13d190SFrançois Tigeot * like before) we need: 691b13d190SFrançois Tigeot * 701b13d190SFrançois Tigeot * - One ringbuffer per-engine inside each context. 711b13d190SFrançois Tigeot * - One backing object per-engine inside each context. 721b13d190SFrançois Tigeot * 731b13d190SFrançois Tigeot * The global default context starts its life with these new objects fully 741b13d190SFrançois Tigeot * allocated and populated. The local default context for each opened fd is 751b13d190SFrançois Tigeot * more complex, because we don't know at creation time which engine is going 761b13d190SFrançois Tigeot * to use them. To handle this, we have implemented a deferred creation of LR 771b13d190SFrançois Tigeot * contexts: 781b13d190SFrançois Tigeot * 791b13d190SFrançois Tigeot * The local context starts its life as a hollow or blank holder, that only 801b13d190SFrançois Tigeot * gets populated for a given engine once we receive an execbuffer. If later 811b13d190SFrançois Tigeot * on we receive another execbuffer ioctl for the same context but a different 821b13d190SFrançois Tigeot * engine, we allocate/populate a new ringbuffer and context backing object and 831b13d190SFrançois Tigeot * so on. 841b13d190SFrançois Tigeot * 851b13d190SFrançois Tigeot * Finally, regarding local contexts created using the ioctl call: as they are 861b13d190SFrançois Tigeot * only allowed with the render ring, we can allocate & populate them right 871b13d190SFrançois Tigeot * away (no need to defer anything, at least for now). 881b13d190SFrançois Tigeot * 891b13d190SFrançois Tigeot * Execlists implementation: 901b13d190SFrançois Tigeot * Execlists are the new method by which, on gen8+ hardware, workloads are 911b13d190SFrançois Tigeot * submitted for execution (as opposed to the legacy, ringbuffer-based, method). 921b13d190SFrançois Tigeot * This method works as follows: 931b13d190SFrançois Tigeot * 941b13d190SFrançois Tigeot * When a request is committed, its commands (the BB start and any leading or 951b13d190SFrançois Tigeot * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer 961b13d190SFrançois Tigeot * for the appropriate context. The tail pointer in the hardware context is not 971b13d190SFrançois Tigeot * updated at this time, but instead, kept by the driver in the ringbuffer 981b13d190SFrançois Tigeot * structure. A structure representing this request is added to a request queue 991b13d190SFrançois Tigeot * for the appropriate engine: this structure contains a copy of the context's 1001b13d190SFrançois Tigeot * tail after the request was written to the ring buffer and a pointer to the 1011b13d190SFrançois Tigeot * context itself. 1021b13d190SFrançois Tigeot * 1031b13d190SFrançois Tigeot * If the engine's request queue was empty before the request was added, the 1041b13d190SFrançois Tigeot * queue is processed immediately. Otherwise the queue will be processed during 1051b13d190SFrançois Tigeot * a context switch interrupt. In any case, elements on the queue will get sent 1061b13d190SFrançois Tigeot * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a 1071b13d190SFrançois Tigeot * globally unique 20-bits submission ID. 1081b13d190SFrançois Tigeot * 1091b13d190SFrançois Tigeot * When execution of a request completes, the GPU updates the context status 1101b13d190SFrançois Tigeot * buffer with a context complete event and generates a context switch interrupt. 1111b13d190SFrançois Tigeot * During the interrupt handling, the driver examines the events in the buffer: 1121b13d190SFrançois Tigeot * for each context complete event, if the announced ID matches that on the head 1131b13d190SFrançois Tigeot * of the request queue, then that request is retired and removed from the queue. 1141b13d190SFrançois Tigeot * 1151b13d190SFrançois Tigeot * After processing, if any requests were retired and the queue is not empty 1161b13d190SFrançois Tigeot * then a new execution list can be submitted. The two requests at the front of 1171b13d190SFrançois Tigeot * the queue are next to be submitted but since a context may not occur twice in 1181b13d190SFrançois Tigeot * an execution list, if subsequent requests have the same ID as the first then 1191b13d190SFrançois Tigeot * the two requests must be combined. This is done simply by discarding requests 1201b13d190SFrançois Tigeot * at the head of the queue until either only one requests is left (in which case 1211b13d190SFrançois Tigeot * we use a NULL second context) or the first two requests have unique IDs. 1221b13d190SFrançois Tigeot * 1231b13d190SFrançois Tigeot * By always executing the first two requests in the queue the driver ensures 1241b13d190SFrançois Tigeot * that the GPU is kept as busy as possible. In the case where a single context 1251b13d190SFrançois Tigeot * completes but a second context is still executing, the request for this second 1261b13d190SFrançois Tigeot * context will be at the head of the queue when we remove the first one. This 1271b13d190SFrançois Tigeot * request will then be resubmitted along with a new request for a different context, 1281b13d190SFrançois Tigeot * which will cause the hardware to continue executing the second request and queue 1291b13d190SFrançois Tigeot * the new request (the GPU detects the condition of a context getting preempted 1301b13d190SFrançois Tigeot * with the same context and optimizes the context switch flow by not doing 1311b13d190SFrançois Tigeot * preemption, but just sampling the new tail pointer). 1321b13d190SFrançois Tigeot * 1331b13d190SFrançois Tigeot */ 1341b13d190SFrançois Tigeot 1351b13d190SFrançois Tigeot #include <drm/drmP.h> 1361b13d190SFrançois Tigeot #include <drm/i915_drm.h> 1371b13d190SFrançois Tigeot #include "i915_drv.h" 1381b13d190SFrançois Tigeot #include "intel_drv.h" 1391b13d190SFrançois Tigeot 1402c9916cdSFrançois Tigeot #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE) 1411b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE) 1421b13d190SFrançois Tigeot #define GEN8_LR_CONTEXT_OTHER_SIZE (2 * PAGE_SIZE) 1431b13d190SFrançois Tigeot 1441b13d190SFrançois Tigeot #define RING_EXECLIST_QFULL (1 << 0x2) 1451b13d190SFrançois Tigeot #define RING_EXECLIST1_VALID (1 << 0x3) 1461b13d190SFrançois Tigeot #define RING_EXECLIST0_VALID (1 << 0x4) 1471b13d190SFrançois Tigeot #define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE) 1481b13d190SFrançois Tigeot #define RING_EXECLIST1_ACTIVE (1 << 0x11) 1491b13d190SFrançois Tigeot #define RING_EXECLIST0_ACTIVE (1 << 0x12) 1501b13d190SFrançois Tigeot 1511b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0) 1521b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_PREEMPTED (1 << 1) 1531b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2) 1541b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3) 1551b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_COMPLETE (1 << 4) 1561b13d190SFrançois Tigeot #define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15) 1571b13d190SFrançois Tigeot 1581b13d190SFrançois Tigeot #define CTX_LRI_HEADER_0 0x01 1591b13d190SFrançois Tigeot #define CTX_CONTEXT_CONTROL 0x02 1601b13d190SFrançois Tigeot #define CTX_RING_HEAD 0x04 1611b13d190SFrançois Tigeot #define CTX_RING_TAIL 0x06 1621b13d190SFrançois Tigeot #define CTX_RING_BUFFER_START 0x08 1631b13d190SFrançois Tigeot #define CTX_RING_BUFFER_CONTROL 0x0a 1641b13d190SFrançois Tigeot #define CTX_BB_HEAD_U 0x0c 1651b13d190SFrançois Tigeot #define CTX_BB_HEAD_L 0x0e 1661b13d190SFrançois Tigeot #define CTX_BB_STATE 0x10 1671b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_U 0x12 1681b13d190SFrançois Tigeot #define CTX_SECOND_BB_HEAD_L 0x14 1691b13d190SFrançois Tigeot #define CTX_SECOND_BB_STATE 0x16 1701b13d190SFrançois Tigeot #define CTX_BB_PER_CTX_PTR 0x18 1711b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX 0x1a 1721b13d190SFrançois Tigeot #define CTX_RCS_INDIRECT_CTX_OFFSET 0x1c 1731b13d190SFrançois Tigeot #define CTX_LRI_HEADER_1 0x21 1741b13d190SFrançois Tigeot #define CTX_CTX_TIMESTAMP 0x22 1751b13d190SFrançois Tigeot #define CTX_PDP3_UDW 0x24 1761b13d190SFrançois Tigeot #define CTX_PDP3_LDW 0x26 1771b13d190SFrançois Tigeot #define CTX_PDP2_UDW 0x28 1781b13d190SFrançois Tigeot #define CTX_PDP2_LDW 0x2a 1791b13d190SFrançois Tigeot #define CTX_PDP1_UDW 0x2c 1801b13d190SFrançois Tigeot #define CTX_PDP1_LDW 0x2e 1811b13d190SFrançois Tigeot #define CTX_PDP0_UDW 0x30 1821b13d190SFrançois Tigeot #define CTX_PDP0_LDW 0x32 1831b13d190SFrançois Tigeot #define CTX_LRI_HEADER_2 0x41 1841b13d190SFrançois Tigeot #define CTX_R_PWR_CLK_STATE 0x42 1851b13d190SFrançois Tigeot #define CTX_GPGPU_CSR_BASE_ADDRESS 0x44 1861b13d190SFrançois Tigeot 1871b13d190SFrançois Tigeot #define GEN8_CTX_VALID (1<<0) 1881b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_PD_RESTORE (1<<1) 1891b13d190SFrançois Tigeot #define GEN8_CTX_FORCE_RESTORE (1<<2) 1901b13d190SFrançois Tigeot #define GEN8_CTX_L3LLC_COHERENT (1<<5) 1911b13d190SFrançois Tigeot #define GEN8_CTX_PRIVILEGE (1<<8) 192*19c468b4SFrançois Tigeot 193*19c468b4SFrançois Tigeot #define ASSIGN_CTX_PDP(ppgtt, reg_state, n) { \ 194*19c468b4SFrançois Tigeot const u64 _addr = test_bit(n, ppgtt->pdp.used_pdpes) ? \ 195*19c468b4SFrançois Tigeot ppgtt->pdp.page_directory[n]->daddr : \ 196*19c468b4SFrançois Tigeot ppgtt->scratch_pd->daddr; \ 197*19c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _UDW+1] = upper_32_bits(_addr); \ 198*19c468b4SFrançois Tigeot reg_state[CTX_PDP ## n ## _LDW+1] = lower_32_bits(_addr); \ 199*19c468b4SFrançois Tigeot } 200*19c468b4SFrançois Tigeot 2011b13d190SFrançois Tigeot enum { 2021b13d190SFrançois Tigeot ADVANCED_CONTEXT = 0, 2031b13d190SFrançois Tigeot LEGACY_CONTEXT, 2041b13d190SFrançois Tigeot ADVANCED_AD_CONTEXT, 2051b13d190SFrançois Tigeot LEGACY_64B_CONTEXT 2061b13d190SFrançois Tigeot }; 2071b13d190SFrançois Tigeot #define GEN8_CTX_MODE_SHIFT 3 2081b13d190SFrançois Tigeot enum { 2091b13d190SFrançois Tigeot FAULT_AND_HANG = 0, 2101b13d190SFrançois Tigeot FAULT_AND_HALT, /* Debug only */ 2111b13d190SFrançois Tigeot FAULT_AND_STREAM, 2121b13d190SFrançois Tigeot FAULT_AND_CONTINUE /* Unsupported */ 2131b13d190SFrançois Tigeot }; 2141b13d190SFrançois Tigeot #define GEN8_CTX_ID_SHIFT 32 2151b13d190SFrançois Tigeot 2162c9916cdSFrançois Tigeot static int intel_lr_context_pin(struct intel_engine_cs *ring, 2172c9916cdSFrançois Tigeot struct intel_context *ctx); 2182c9916cdSFrançois Tigeot 2191b13d190SFrançois Tigeot /** 2201b13d190SFrançois Tigeot * intel_sanitize_enable_execlists() - sanitize i915.enable_execlists 2211b13d190SFrançois Tigeot * @dev: DRM device. 2221b13d190SFrançois Tigeot * @enable_execlists: value of i915.enable_execlists module parameter. 2231b13d190SFrançois Tigeot * 2241b13d190SFrançois Tigeot * Only certain platforms support Execlists (the prerequisites being 2252c9916cdSFrançois Tigeot * support for Logical Ring Contexts and Aliasing PPGTT or better). 2261b13d190SFrançois Tigeot * 2271b13d190SFrançois Tigeot * Return: 1 if Execlists is supported and has to be enabled. 2281b13d190SFrançois Tigeot */ 2291b13d190SFrançois Tigeot int intel_sanitize_enable_execlists(struct drm_device *dev, int enable_execlists) 2301b13d190SFrançois Tigeot { 2311b13d190SFrançois Tigeot WARN_ON(i915.enable_ppgtt == -1); 2321b13d190SFrançois Tigeot 2332c9916cdSFrançois Tigeot if (INTEL_INFO(dev)->gen >= 9) 2342c9916cdSFrançois Tigeot return 1; 2352c9916cdSFrançois Tigeot 2361b13d190SFrançois Tigeot if (enable_execlists == 0) 2371b13d190SFrançois Tigeot return 0; 2381b13d190SFrançois Tigeot 2391b13d190SFrançois Tigeot if (HAS_LOGICAL_RING_CONTEXTS(dev) && USES_PPGTT(dev) && 2401b13d190SFrançois Tigeot i915.use_mmio_flip >= 0) 2411b13d190SFrançois Tigeot return 1; 2421b13d190SFrançois Tigeot 2431b13d190SFrançois Tigeot return 0; 2441b13d190SFrançois Tigeot } 2451b13d190SFrançois Tigeot 2461b13d190SFrançois Tigeot /** 2471b13d190SFrançois Tigeot * intel_execlists_ctx_id() - get the Execlists Context ID 2481b13d190SFrançois Tigeot * @ctx_obj: Logical Ring Context backing object. 2491b13d190SFrançois Tigeot * 2501b13d190SFrançois Tigeot * Do not confuse with ctx->id! Unfortunately we have a name overload 2511b13d190SFrançois Tigeot * here: the old context ID we pass to userspace as a handler so that 2521b13d190SFrançois Tigeot * they can refer to a context, and the new context ID we pass to the 2531b13d190SFrançois Tigeot * ELSP so that the GPU can inform us of the context status via 2541b13d190SFrançois Tigeot * interrupts. 2551b13d190SFrançois Tigeot * 2561b13d190SFrançois Tigeot * Return: 20-bits globally unique context ID. 2571b13d190SFrançois Tigeot */ 2581b13d190SFrançois Tigeot u32 intel_execlists_ctx_id(struct drm_i915_gem_object *ctx_obj) 2591b13d190SFrançois Tigeot { 2601b13d190SFrançois Tigeot u32 lrca = i915_gem_obj_ggtt_offset(ctx_obj); 2611b13d190SFrançois Tigeot 2621b13d190SFrançois Tigeot /* LRCA is required to be 4K aligned so the more significant 20 bits 2631b13d190SFrançois Tigeot * are globally unique */ 2641b13d190SFrançois Tigeot return lrca >> 12; 2651b13d190SFrançois Tigeot } 2661b13d190SFrançois Tigeot 267477eb7f9SFrançois Tigeot static uint64_t execlists_ctx_descriptor(struct intel_engine_cs *ring, 268477eb7f9SFrançois Tigeot struct drm_i915_gem_object *ctx_obj) 2691b13d190SFrançois Tigeot { 270477eb7f9SFrançois Tigeot struct drm_device *dev = ring->dev; 2711b13d190SFrançois Tigeot uint64_t desc; 2721b13d190SFrançois Tigeot uint64_t lrca = i915_gem_obj_ggtt_offset(ctx_obj); 2731b13d190SFrançois Tigeot 2741b13d190SFrançois Tigeot WARN_ON(lrca & 0xFFFFFFFF00000FFFULL); 2751b13d190SFrançois Tigeot 2761b13d190SFrançois Tigeot desc = GEN8_CTX_VALID; 2771b13d190SFrançois Tigeot desc |= LEGACY_CONTEXT << GEN8_CTX_MODE_SHIFT; 278*19c468b4SFrançois Tigeot if (IS_GEN8(ctx_obj->base.dev)) 2791b13d190SFrançois Tigeot desc |= GEN8_CTX_L3LLC_COHERENT; 2801b13d190SFrançois Tigeot desc |= GEN8_CTX_PRIVILEGE; 2811b13d190SFrançois Tigeot desc |= lrca; 2821b13d190SFrançois Tigeot desc |= (u64)intel_execlists_ctx_id(ctx_obj) << GEN8_CTX_ID_SHIFT; 2831b13d190SFrançois Tigeot 2841b13d190SFrançois Tigeot /* TODO: WaDisableLiteRestore when we start using semaphore 2851b13d190SFrançois Tigeot * signalling between Command Streamers */ 2861b13d190SFrançois Tigeot /* desc |= GEN8_CTX_FORCE_RESTORE; */ 2871b13d190SFrançois Tigeot 288477eb7f9SFrançois Tigeot /* WaEnableForceRestoreInCtxtDescForVCS:skl */ 289477eb7f9SFrançois Tigeot if (IS_GEN9(dev) && 290477eb7f9SFrançois Tigeot INTEL_REVID(dev) <= SKL_REVID_B0 && 291477eb7f9SFrançois Tigeot (ring->id == BCS || ring->id == VCS || 292477eb7f9SFrançois Tigeot ring->id == VECS || ring->id == VCS2)) 293477eb7f9SFrançois Tigeot desc |= GEN8_CTX_FORCE_RESTORE; 294477eb7f9SFrançois Tigeot 2951b13d190SFrançois Tigeot return desc; 2961b13d190SFrançois Tigeot } 2971b13d190SFrançois Tigeot 2981b13d190SFrançois Tigeot static void execlists_elsp_write(struct intel_engine_cs *ring, 2991b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj0, 3001b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj1) 3011b13d190SFrançois Tigeot { 3022c9916cdSFrançois Tigeot struct drm_device *dev = ring->dev; 3032c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 3041b13d190SFrançois Tigeot uint64_t temp = 0; 3051b13d190SFrançois Tigeot uint32_t desc[4]; 3061b13d190SFrançois Tigeot 3071b13d190SFrançois Tigeot /* XXX: You must always write both descriptors in the order below. */ 3081b13d190SFrançois Tigeot if (ctx_obj1) 309477eb7f9SFrançois Tigeot temp = execlists_ctx_descriptor(ring, ctx_obj1); 3101b13d190SFrançois Tigeot else 3111b13d190SFrançois Tigeot temp = 0; 3121b13d190SFrançois Tigeot desc[1] = (u32)(temp >> 32); 3131b13d190SFrançois Tigeot desc[0] = (u32)temp; 3141b13d190SFrançois Tigeot 315477eb7f9SFrançois Tigeot temp = execlists_ctx_descriptor(ring, ctx_obj0); 3161b13d190SFrançois Tigeot desc[3] = (u32)(temp >> 32); 3171b13d190SFrançois Tigeot desc[2] = (u32)temp; 3181b13d190SFrançois Tigeot 319*19c468b4SFrançois Tigeot lockmgr(&dev_priv->uncore.lock, LK_EXCLUSIVE); 320*19c468b4SFrançois Tigeot intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL); 321*19c468b4SFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), desc[1]); 322*19c468b4SFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), desc[0]); 323*19c468b4SFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), desc[3]); 3242c9916cdSFrançois Tigeot 3251b13d190SFrançois Tigeot /* The context is automatically loaded after the following */ 326*19c468b4SFrançois Tigeot I915_WRITE_FW(RING_ELSP(ring), desc[2]); 3271b13d190SFrançois Tigeot 3281b13d190SFrançois Tigeot /* ELSP is a wo register, so use another nearby reg for posting instead */ 329*19c468b4SFrançois Tigeot POSTING_READ_FW(RING_EXECLIST_STATUS(ring)); 330*19c468b4SFrançois Tigeot intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL); 331*19c468b4SFrançois Tigeot lockmgr(&dev_priv->uncore.lock, LK_RELEASE); 3321b13d190SFrançois Tigeot } 3331b13d190SFrançois Tigeot 3342c9916cdSFrançois Tigeot static int execlists_update_context(struct drm_i915_gem_object *ctx_obj, 3352c9916cdSFrançois Tigeot struct drm_i915_gem_object *ring_obj, 336*19c468b4SFrançois Tigeot struct i915_hw_ppgtt *ppgtt, 3372c9916cdSFrançois Tigeot u32 tail) 3381b13d190SFrançois Tigeot { 3391b13d190SFrançois Tigeot struct vm_page *page; 3401b13d190SFrançois Tigeot uint32_t *reg_state; 3411b13d190SFrançois Tigeot 3421b13d190SFrançois Tigeot page = i915_gem_object_get_page(ctx_obj, 1); 3431b13d190SFrançois Tigeot reg_state = kmap_atomic(page); 3441b13d190SFrançois Tigeot 3451b13d190SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = tail; 3462c9916cdSFrançois Tigeot reg_state[CTX_RING_BUFFER_START+1] = i915_gem_obj_ggtt_offset(ring_obj); 3471b13d190SFrançois Tigeot 348*19c468b4SFrançois Tigeot /* True PPGTT with dynamic page allocation: update PDP registers and 349*19c468b4SFrançois Tigeot * point the unallocated PDPs to the scratch page 350*19c468b4SFrançois Tigeot */ 351*19c468b4SFrançois Tigeot if (ppgtt) { 352*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 3); 353*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 2); 354*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 1); 355*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 0); 356*19c468b4SFrançois Tigeot } 357*19c468b4SFrançois Tigeot 3581b13d190SFrançois Tigeot kunmap_atomic(reg_state); 3591b13d190SFrançois Tigeot 3601b13d190SFrançois Tigeot return 0; 3611b13d190SFrançois Tigeot } 3621b13d190SFrançois Tigeot 3632c9916cdSFrançois Tigeot static void execlists_submit_contexts(struct intel_engine_cs *ring, 3641b13d190SFrançois Tigeot struct intel_context *to0, u32 tail0, 3651b13d190SFrançois Tigeot struct intel_context *to1, u32 tail1) 3661b13d190SFrançois Tigeot { 3672c9916cdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj0 = to0->engine[ring->id].state; 3682c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf0 = to0->engine[ring->id].ringbuf; 3691b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj1 = NULL; 3702c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf1 = NULL; 3711b13d190SFrançois Tigeot 3721b13d190SFrançois Tigeot BUG_ON(!ctx_obj0); 3731b13d190SFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(ctx_obj0)); 3742c9916cdSFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(ringbuf0->obj)); 3751b13d190SFrançois Tigeot 376*19c468b4SFrançois Tigeot execlists_update_context(ctx_obj0, ringbuf0->obj, to0->ppgtt, tail0); 3771b13d190SFrançois Tigeot 3781b13d190SFrançois Tigeot if (to1) { 3792c9916cdSFrançois Tigeot ringbuf1 = to1->engine[ring->id].ringbuf; 3801b13d190SFrançois Tigeot ctx_obj1 = to1->engine[ring->id].state; 3811b13d190SFrançois Tigeot BUG_ON(!ctx_obj1); 3821b13d190SFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(ctx_obj1)); 3832c9916cdSFrançois Tigeot WARN_ON(!i915_gem_obj_is_pinned(ringbuf1->obj)); 3841b13d190SFrançois Tigeot 385*19c468b4SFrançois Tigeot execlists_update_context(ctx_obj1, ringbuf1->obj, to1->ppgtt, tail1); 3861b13d190SFrançois Tigeot } 3871b13d190SFrançois Tigeot 3881b13d190SFrançois Tigeot execlists_elsp_write(ring, ctx_obj0, ctx_obj1); 3891b13d190SFrançois Tigeot } 3901b13d190SFrançois Tigeot 3911b13d190SFrançois Tigeot static void execlists_context_unqueue(struct intel_engine_cs *ring) 3921b13d190SFrançois Tigeot { 3932c9916cdSFrançois Tigeot struct drm_i915_gem_request *req0 = NULL, *req1 = NULL; 3942c9916cdSFrançois Tigeot struct drm_i915_gem_request *cursor = NULL, *tmp = NULL; 3951b13d190SFrançois Tigeot 3961b13d190SFrançois Tigeot assert_spin_locked(&ring->execlist_lock); 3971b13d190SFrançois Tigeot 398*19c468b4SFrançois Tigeot /* 399*19c468b4SFrançois Tigeot * If irqs are not active generate a warning as batches that finish 400*19c468b4SFrançois Tigeot * without the irqs may get lost and a GPU Hang may occur. 401*19c468b4SFrançois Tigeot */ 402*19c468b4SFrançois Tigeot WARN_ON(!intel_irqs_enabled(ring->dev->dev_private)); 403*19c468b4SFrançois Tigeot 4041b13d190SFrançois Tigeot if (list_empty(&ring->execlist_queue)) 4051b13d190SFrançois Tigeot return; 4061b13d190SFrançois Tigeot 4071b13d190SFrançois Tigeot /* Try to read in pairs */ 4081b13d190SFrançois Tigeot list_for_each_entry_safe(cursor, tmp, &ring->execlist_queue, 4091b13d190SFrançois Tigeot execlist_link) { 4101b13d190SFrançois Tigeot if (!req0) { 4111b13d190SFrançois Tigeot req0 = cursor; 4121b13d190SFrançois Tigeot } else if (req0->ctx == cursor->ctx) { 4131b13d190SFrançois Tigeot /* Same ctx: ignore first request, as second request 4141b13d190SFrançois Tigeot * will update tail past first request's workload */ 4151b13d190SFrançois Tigeot cursor->elsp_submitted = req0->elsp_submitted; 4161b13d190SFrançois Tigeot list_del(&req0->execlist_link); 4172c9916cdSFrançois Tigeot list_add_tail(&req0->execlist_link, 4182c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 4191b13d190SFrançois Tigeot req0 = cursor; 4201b13d190SFrançois Tigeot } else { 4211b13d190SFrançois Tigeot req1 = cursor; 4221b13d190SFrançois Tigeot break; 4231b13d190SFrançois Tigeot } 4241b13d190SFrançois Tigeot } 4251b13d190SFrançois Tigeot 426477eb7f9SFrançois Tigeot if (IS_GEN8(ring->dev) || IS_GEN9(ring->dev)) { 427477eb7f9SFrançois Tigeot /* 428477eb7f9SFrançois Tigeot * WaIdleLiteRestore: make sure we never cause a lite 429477eb7f9SFrançois Tigeot * restore with HEAD==TAIL 430477eb7f9SFrançois Tigeot */ 431*19c468b4SFrançois Tigeot if (req0->elsp_submitted) { 432477eb7f9SFrançois Tigeot /* 433477eb7f9SFrançois Tigeot * Apply the wa NOOPS to prevent ring:HEAD == req:TAIL 434477eb7f9SFrançois Tigeot * as we resubmit the request. See gen8_emit_request() 435477eb7f9SFrançois Tigeot * for where we prepare the padding after the end of the 436477eb7f9SFrançois Tigeot * request. 437477eb7f9SFrançois Tigeot */ 438477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf; 439477eb7f9SFrançois Tigeot 440477eb7f9SFrançois Tigeot ringbuf = req0->ctx->engine[ring->id].ringbuf; 441477eb7f9SFrançois Tigeot req0->tail += 8; 442477eb7f9SFrançois Tigeot req0->tail &= ringbuf->size - 1; 443477eb7f9SFrançois Tigeot } 444477eb7f9SFrançois Tigeot } 445477eb7f9SFrançois Tigeot 4461b13d190SFrançois Tigeot WARN_ON(req1 && req1->elsp_submitted); 4471b13d190SFrançois Tigeot 4482c9916cdSFrançois Tigeot execlists_submit_contexts(ring, req0->ctx, req0->tail, 4491b13d190SFrançois Tigeot req1 ? req1->ctx : NULL, 4502c9916cdSFrançois Tigeot req1 ? req1->tail : 0); 4511b13d190SFrançois Tigeot 4521b13d190SFrançois Tigeot req0->elsp_submitted++; 4531b13d190SFrançois Tigeot if (req1) 4541b13d190SFrançois Tigeot req1->elsp_submitted++; 4551b13d190SFrançois Tigeot } 4561b13d190SFrançois Tigeot 4571b13d190SFrançois Tigeot static bool execlists_check_remove_request(struct intel_engine_cs *ring, 4581b13d190SFrançois Tigeot u32 request_id) 4591b13d190SFrançois Tigeot { 4602c9916cdSFrançois Tigeot struct drm_i915_gem_request *head_req; 4611b13d190SFrançois Tigeot 4621b13d190SFrançois Tigeot assert_spin_locked(&ring->execlist_lock); 4631b13d190SFrançois Tigeot 4641b13d190SFrançois Tigeot head_req = list_first_entry_or_null(&ring->execlist_queue, 4652c9916cdSFrançois Tigeot struct drm_i915_gem_request, 4661b13d190SFrançois Tigeot execlist_link); 4671b13d190SFrançois Tigeot 4681b13d190SFrançois Tigeot if (head_req != NULL) { 4691b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 4701b13d190SFrançois Tigeot head_req->ctx->engine[ring->id].state; 4711b13d190SFrançois Tigeot if (intel_execlists_ctx_id(ctx_obj) == request_id) { 4721b13d190SFrançois Tigeot WARN(head_req->elsp_submitted == 0, 4731b13d190SFrançois Tigeot "Never submitted head request\n"); 4741b13d190SFrançois Tigeot 4751b13d190SFrançois Tigeot if (--head_req->elsp_submitted <= 0) { 4761b13d190SFrançois Tigeot list_del(&head_req->execlist_link); 4772c9916cdSFrançois Tigeot list_add_tail(&head_req->execlist_link, 4782c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 4791b13d190SFrançois Tigeot return true; 4801b13d190SFrançois Tigeot } 4811b13d190SFrançois Tigeot } 4821b13d190SFrançois Tigeot } 4831b13d190SFrançois Tigeot 4841b13d190SFrançois Tigeot return false; 4851b13d190SFrançois Tigeot } 4861b13d190SFrançois Tigeot 4871b13d190SFrançois Tigeot /** 4882c9916cdSFrançois Tigeot * intel_lrc_irq_handler() - handle Context Switch interrupts 4891b13d190SFrançois Tigeot * @ring: Engine Command Streamer to handle. 4901b13d190SFrançois Tigeot * 4911b13d190SFrançois Tigeot * Check the unread Context Status Buffers and manage the submission of new 4921b13d190SFrançois Tigeot * contexts to the ELSP accordingly. 4931b13d190SFrançois Tigeot */ 4942c9916cdSFrançois Tigeot void intel_lrc_irq_handler(struct intel_engine_cs *ring) 4951b13d190SFrançois Tigeot { 4961b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 4971b13d190SFrançois Tigeot u32 status_pointer; 4981b13d190SFrançois Tigeot u8 read_pointer; 4991b13d190SFrançois Tigeot u8 write_pointer; 5001b13d190SFrançois Tigeot u32 status; 5011b13d190SFrançois Tigeot u32 status_id; 5021b13d190SFrançois Tigeot u32 submit_contexts = 0; 5031b13d190SFrançois Tigeot 5041b13d190SFrançois Tigeot status_pointer = I915_READ(RING_CONTEXT_STATUS_PTR(ring)); 5051b13d190SFrançois Tigeot 5061b13d190SFrançois Tigeot read_pointer = ring->next_context_status_buffer; 5071b13d190SFrançois Tigeot write_pointer = status_pointer & 0x07; 5081b13d190SFrançois Tigeot if (read_pointer > write_pointer) 5091b13d190SFrançois Tigeot write_pointer += 6; 5101b13d190SFrançois Tigeot 5111b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_EXCLUSIVE); 5121b13d190SFrançois Tigeot 5131b13d190SFrançois Tigeot while (read_pointer < write_pointer) { 5141b13d190SFrançois Tigeot read_pointer++; 5151b13d190SFrançois Tigeot status = I915_READ(RING_CONTEXT_STATUS_BUF(ring) + 5161b13d190SFrançois Tigeot (read_pointer % 6) * 8); 5171b13d190SFrançois Tigeot status_id = I915_READ(RING_CONTEXT_STATUS_BUF(ring) + 5181b13d190SFrançois Tigeot (read_pointer % 6) * 8 + 4); 5191b13d190SFrançois Tigeot 5201b13d190SFrançois Tigeot if (status & GEN8_CTX_STATUS_PREEMPTED) { 5211b13d190SFrançois Tigeot if (status & GEN8_CTX_STATUS_LITE_RESTORE) { 5221b13d190SFrançois Tigeot if (execlists_check_remove_request(ring, status_id)) 5231b13d190SFrançois Tigeot WARN(1, "Lite Restored request removed from queue\n"); 5241b13d190SFrançois Tigeot } else 5251b13d190SFrançois Tigeot WARN(1, "Preemption without Lite Restore\n"); 5261b13d190SFrançois Tigeot } 5271b13d190SFrançois Tigeot 5281b13d190SFrançois Tigeot if ((status & GEN8_CTX_STATUS_ACTIVE_IDLE) || 5291b13d190SFrançois Tigeot (status & GEN8_CTX_STATUS_ELEMENT_SWITCH)) { 5301b13d190SFrançois Tigeot if (execlists_check_remove_request(ring, status_id)) 5311b13d190SFrançois Tigeot submit_contexts++; 5321b13d190SFrançois Tigeot } 5331b13d190SFrançois Tigeot } 5341b13d190SFrançois Tigeot 5351b13d190SFrançois Tigeot if (submit_contexts != 0) 5361b13d190SFrançois Tigeot execlists_context_unqueue(ring); 5371b13d190SFrançois Tigeot 5381b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_RELEASE); 5391b13d190SFrançois Tigeot 5401b13d190SFrançois Tigeot WARN(submit_contexts > 2, "More than two context complete events?\n"); 5411b13d190SFrançois Tigeot ring->next_context_status_buffer = write_pointer % 6; 5421b13d190SFrançois Tigeot 5431b13d190SFrançois Tigeot I915_WRITE(RING_CONTEXT_STATUS_PTR(ring), 5441b13d190SFrançois Tigeot ((u32)ring->next_context_status_buffer & 0x07) << 8); 5451b13d190SFrançois Tigeot } 5461b13d190SFrançois Tigeot 5471b13d190SFrançois Tigeot static int execlists_context_queue(struct intel_engine_cs *ring, 5481b13d190SFrançois Tigeot struct intel_context *to, 5492c9916cdSFrançois Tigeot u32 tail, 5502c9916cdSFrançois Tigeot struct drm_i915_gem_request *request) 5511b13d190SFrançois Tigeot { 5522c9916cdSFrançois Tigeot struct drm_i915_gem_request *cursor; 5531b13d190SFrançois Tigeot int num_elements = 0; 5541b13d190SFrançois Tigeot 5552c9916cdSFrançois Tigeot if (to != ring->default_context) 5562c9916cdSFrançois Tigeot intel_lr_context_pin(ring, to); 5572c9916cdSFrançois Tigeot 5582c9916cdSFrançois Tigeot if (!request) { 5592c9916cdSFrançois Tigeot /* 5602c9916cdSFrançois Tigeot * If there isn't a request associated with this submission, 5612c9916cdSFrançois Tigeot * create one as a temporary holder. 5622c9916cdSFrançois Tigeot */ 5632c9916cdSFrançois Tigeot request = kzalloc(sizeof(*request), GFP_KERNEL); 5642c9916cdSFrançois Tigeot if (request == NULL) 5651b13d190SFrançois Tigeot return -ENOMEM; 5662c9916cdSFrançois Tigeot request->ring = ring; 5672c9916cdSFrançois Tigeot request->ctx = to; 5682c9916cdSFrançois Tigeot kref_init(&request->ref); 5692c9916cdSFrançois Tigeot i915_gem_context_reference(request->ctx); 5702c9916cdSFrançois Tigeot } else { 5712c9916cdSFrançois Tigeot i915_gem_request_reference(request); 5722c9916cdSFrançois Tigeot WARN_ON(to != request->ctx); 5732c9916cdSFrançois Tigeot } 5742c9916cdSFrançois Tigeot request->tail = tail; 5751b13d190SFrançois Tigeot 5761b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_EXCLUSIVE); 5771b13d190SFrançois Tigeot 5781b13d190SFrançois Tigeot list_for_each_entry(cursor, &ring->execlist_queue, execlist_link) 5791b13d190SFrançois Tigeot if (++num_elements > 2) 5801b13d190SFrançois Tigeot break; 5811b13d190SFrançois Tigeot 5821b13d190SFrançois Tigeot if (num_elements > 2) { 5832c9916cdSFrançois Tigeot struct drm_i915_gem_request *tail_req; 5841b13d190SFrançois Tigeot 5851b13d190SFrançois Tigeot tail_req = list_last_entry(&ring->execlist_queue, 5862c9916cdSFrançois Tigeot struct drm_i915_gem_request, 5871b13d190SFrançois Tigeot execlist_link); 5881b13d190SFrançois Tigeot 5891b13d190SFrançois Tigeot if (to == tail_req->ctx) { 5901b13d190SFrançois Tigeot WARN(tail_req->elsp_submitted != 0, 5911b13d190SFrançois Tigeot "More than 2 already-submitted reqs queued\n"); 5921b13d190SFrançois Tigeot list_del(&tail_req->execlist_link); 5932c9916cdSFrançois Tigeot list_add_tail(&tail_req->execlist_link, 5942c9916cdSFrançois Tigeot &ring->execlist_retired_req_list); 5951b13d190SFrançois Tigeot } 5961b13d190SFrançois Tigeot } 5971b13d190SFrançois Tigeot 5982c9916cdSFrançois Tigeot list_add_tail(&request->execlist_link, &ring->execlist_queue); 5991b13d190SFrançois Tigeot if (num_elements == 0) 6001b13d190SFrançois Tigeot execlists_context_unqueue(ring); 6011b13d190SFrançois Tigeot 6021b13d190SFrançois Tigeot lockmgr(&ring->execlist_lock, LK_RELEASE); 6031b13d190SFrançois Tigeot 6041b13d190SFrançois Tigeot return 0; 6051b13d190SFrançois Tigeot } 6061b13d190SFrançois Tigeot 6072c9916cdSFrançois Tigeot static int logical_ring_invalidate_all_caches(struct intel_ringbuffer *ringbuf, 6082c9916cdSFrançois Tigeot struct intel_context *ctx) 6091b13d190SFrançois Tigeot { 6101b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 6111b13d190SFrançois Tigeot uint32_t flush_domains; 6121b13d190SFrançois Tigeot int ret; 6131b13d190SFrançois Tigeot 6141b13d190SFrançois Tigeot flush_domains = 0; 6151b13d190SFrançois Tigeot if (ring->gpu_caches_dirty) 6161b13d190SFrançois Tigeot flush_domains = I915_GEM_GPU_DOMAINS; 6171b13d190SFrançois Tigeot 6182c9916cdSFrançois Tigeot ret = ring->emit_flush(ringbuf, ctx, 6192c9916cdSFrançois Tigeot I915_GEM_GPU_DOMAINS, flush_domains); 6201b13d190SFrançois Tigeot if (ret) 6211b13d190SFrançois Tigeot return ret; 6221b13d190SFrançois Tigeot 6231b13d190SFrançois Tigeot ring->gpu_caches_dirty = false; 6241b13d190SFrançois Tigeot return 0; 6251b13d190SFrançois Tigeot } 6261b13d190SFrançois Tigeot 6271b13d190SFrançois Tigeot static int execlists_move_to_gpu(struct intel_ringbuffer *ringbuf, 6282c9916cdSFrançois Tigeot struct intel_context *ctx, 6291b13d190SFrançois Tigeot struct list_head *vmas) 6301b13d190SFrançois Tigeot { 6311b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 632*19c468b4SFrançois Tigeot const unsigned other_rings = ~intel_ring_flag(ring); 6331b13d190SFrançois Tigeot struct i915_vma *vma; 6341b13d190SFrançois Tigeot uint32_t flush_domains = 0; 6351b13d190SFrançois Tigeot bool flush_chipset = false; 6361b13d190SFrançois Tigeot int ret; 6371b13d190SFrançois Tigeot 6381b13d190SFrançois Tigeot list_for_each_entry(vma, vmas, exec_list) { 6391b13d190SFrançois Tigeot struct drm_i915_gem_object *obj = vma->obj; 6401b13d190SFrançois Tigeot 641*19c468b4SFrançois Tigeot if (obj->active & other_rings) { 6421b13d190SFrançois Tigeot ret = i915_gem_object_sync(obj, ring); 6431b13d190SFrançois Tigeot if (ret) 6441b13d190SFrançois Tigeot return ret; 645*19c468b4SFrançois Tigeot } 6461b13d190SFrançois Tigeot 6471b13d190SFrançois Tigeot if (obj->base.write_domain & I915_GEM_DOMAIN_CPU) 6481b13d190SFrançois Tigeot flush_chipset |= i915_gem_clflush_object(obj, false); 6491b13d190SFrançois Tigeot 6501b13d190SFrançois Tigeot flush_domains |= obj->base.write_domain; 6511b13d190SFrançois Tigeot } 6521b13d190SFrançois Tigeot 6531b13d190SFrançois Tigeot if (flush_domains & I915_GEM_DOMAIN_GTT) 6541b13d190SFrançois Tigeot wmb(); 6551b13d190SFrançois Tigeot 6561b13d190SFrançois Tigeot /* Unconditionally invalidate gpu caches and ensure that we do flush 6571b13d190SFrançois Tigeot * any residual writes from the previous batch. 6581b13d190SFrançois Tigeot */ 6592c9916cdSFrançois Tigeot return logical_ring_invalidate_all_caches(ringbuf, ctx); 6601b13d190SFrançois Tigeot } 6611b13d190SFrançois Tigeot 662*19c468b4SFrançois Tigeot int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request, 663*19c468b4SFrançois Tigeot struct intel_context *ctx) 664*19c468b4SFrançois Tigeot { 665*19c468b4SFrançois Tigeot int ret; 666*19c468b4SFrançois Tigeot 667*19c468b4SFrançois Tigeot if (ctx != request->ring->default_context) { 668*19c468b4SFrançois Tigeot ret = intel_lr_context_pin(request->ring, ctx); 669*19c468b4SFrançois Tigeot if (ret) 670*19c468b4SFrançois Tigeot return ret; 671*19c468b4SFrançois Tigeot } 672*19c468b4SFrançois Tigeot 673*19c468b4SFrançois Tigeot request->ringbuf = ctx->engine[request->ring->id].ringbuf; 674*19c468b4SFrançois Tigeot request->ctx = ctx; 675*19c468b4SFrançois Tigeot i915_gem_context_reference(request->ctx); 676*19c468b4SFrançois Tigeot 677*19c468b4SFrançois Tigeot return 0; 678*19c468b4SFrançois Tigeot } 679*19c468b4SFrançois Tigeot 680*19c468b4SFrançois Tigeot static int logical_ring_wait_for_space(struct intel_ringbuffer *ringbuf, 681*19c468b4SFrançois Tigeot struct intel_context *ctx, 682*19c468b4SFrançois Tigeot int bytes) 683*19c468b4SFrançois Tigeot { 684*19c468b4SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 685*19c468b4SFrançois Tigeot struct drm_i915_gem_request *request; 686*19c468b4SFrançois Tigeot unsigned space; 687*19c468b4SFrançois Tigeot int ret; 688*19c468b4SFrançois Tigeot 689*19c468b4SFrançois Tigeot if (intel_ring_space(ringbuf) >= bytes) 690*19c468b4SFrançois Tigeot return 0; 691*19c468b4SFrançois Tigeot 692*19c468b4SFrançois Tigeot list_for_each_entry(request, &ring->request_list, list) { 693*19c468b4SFrançois Tigeot /* 694*19c468b4SFrançois Tigeot * The request queue is per-engine, so can contain requests 695*19c468b4SFrançois Tigeot * from multiple ringbuffers. Here, we must ignore any that 696*19c468b4SFrançois Tigeot * aren't from the ringbuffer we're considering. 697*19c468b4SFrançois Tigeot */ 698*19c468b4SFrançois Tigeot if (request->ringbuf != ringbuf) 699*19c468b4SFrançois Tigeot continue; 700*19c468b4SFrançois Tigeot 701*19c468b4SFrançois Tigeot /* Would completion of this request free enough space? */ 702*19c468b4SFrançois Tigeot space = __intel_ring_space(request->postfix, ringbuf->tail, 703*19c468b4SFrançois Tigeot ringbuf->size); 704*19c468b4SFrançois Tigeot if (space >= bytes) 705*19c468b4SFrançois Tigeot break; 706*19c468b4SFrançois Tigeot } 707*19c468b4SFrançois Tigeot 708*19c468b4SFrançois Tigeot if (WARN_ON(&request->list == &ring->request_list)) 709*19c468b4SFrançois Tigeot return -ENOSPC; 710*19c468b4SFrançois Tigeot 711*19c468b4SFrançois Tigeot ret = i915_wait_request(request); 712*19c468b4SFrançois Tigeot if (ret) 713*19c468b4SFrançois Tigeot return ret; 714*19c468b4SFrançois Tigeot 715*19c468b4SFrançois Tigeot ringbuf->space = space; 716*19c468b4SFrançois Tigeot return 0; 717*19c468b4SFrançois Tigeot } 718*19c468b4SFrançois Tigeot 719*19c468b4SFrançois Tigeot /* 720*19c468b4SFrançois Tigeot * intel_logical_ring_advance_and_submit() - advance the tail and submit the workload 721*19c468b4SFrançois Tigeot * @ringbuf: Logical Ringbuffer to advance. 722*19c468b4SFrançois Tigeot * 723*19c468b4SFrançois Tigeot * The tail is updated in our logical ringbuffer struct, not in the actual context. What 724*19c468b4SFrançois Tigeot * really happens during submission is that the context and current tail will be placed 725*19c468b4SFrançois Tigeot * on a queue waiting for the ELSP to be ready to accept a new context submission. At that 726*19c468b4SFrançois Tigeot * point, the tail *inside* the context is updated and the ELSP written to. 727*19c468b4SFrançois Tigeot */ 728*19c468b4SFrançois Tigeot static void 729*19c468b4SFrançois Tigeot intel_logical_ring_advance_and_submit(struct intel_ringbuffer *ringbuf, 730*19c468b4SFrançois Tigeot struct intel_context *ctx, 731*19c468b4SFrançois Tigeot struct drm_i915_gem_request *request) 732*19c468b4SFrançois Tigeot { 733*19c468b4SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 734*19c468b4SFrançois Tigeot 735*19c468b4SFrançois Tigeot intel_logical_ring_advance(ringbuf); 736*19c468b4SFrançois Tigeot 737*19c468b4SFrançois Tigeot if (intel_ring_stopped(ring)) 738*19c468b4SFrançois Tigeot return; 739*19c468b4SFrançois Tigeot 740*19c468b4SFrançois Tigeot execlists_context_queue(ring, ctx, ringbuf->tail, request); 741*19c468b4SFrançois Tigeot } 742*19c468b4SFrançois Tigeot 743*19c468b4SFrançois Tigeot static int logical_ring_wrap_buffer(struct intel_ringbuffer *ringbuf, 744*19c468b4SFrançois Tigeot struct intel_context *ctx) 745*19c468b4SFrançois Tigeot { 746*19c468b4SFrançois Tigeot uint32_t __iomem *virt; 747*19c468b4SFrançois Tigeot int rem = ringbuf->size - ringbuf->tail; 748*19c468b4SFrançois Tigeot 749*19c468b4SFrançois Tigeot if (ringbuf->space < rem) { 750*19c468b4SFrançois Tigeot int ret = logical_ring_wait_for_space(ringbuf, ctx, rem); 751*19c468b4SFrançois Tigeot 752*19c468b4SFrançois Tigeot if (ret) 753*19c468b4SFrançois Tigeot return ret; 754*19c468b4SFrançois Tigeot } 755*19c468b4SFrançois Tigeot 756*19c468b4SFrançois Tigeot virt = (uint32_t *)(ringbuf->virtual_start + ringbuf->tail); 757*19c468b4SFrançois Tigeot rem /= 4; 758*19c468b4SFrançois Tigeot while (rem--) 759*19c468b4SFrançois Tigeot iowrite32(MI_NOOP, virt++); 760*19c468b4SFrançois Tigeot 761*19c468b4SFrançois Tigeot ringbuf->tail = 0; 762*19c468b4SFrançois Tigeot intel_ring_update_space(ringbuf); 763*19c468b4SFrançois Tigeot 764*19c468b4SFrançois Tigeot return 0; 765*19c468b4SFrançois Tigeot } 766*19c468b4SFrançois Tigeot 767*19c468b4SFrançois Tigeot static int logical_ring_prepare(struct intel_ringbuffer *ringbuf, 768*19c468b4SFrançois Tigeot struct intel_context *ctx, int bytes) 769*19c468b4SFrançois Tigeot { 770*19c468b4SFrançois Tigeot int ret; 771*19c468b4SFrançois Tigeot 772*19c468b4SFrançois Tigeot if (unlikely(ringbuf->tail + bytes > ringbuf->effective_size)) { 773*19c468b4SFrançois Tigeot ret = logical_ring_wrap_buffer(ringbuf, ctx); 774*19c468b4SFrançois Tigeot if (unlikely(ret)) 775*19c468b4SFrançois Tigeot return ret; 776*19c468b4SFrançois Tigeot } 777*19c468b4SFrançois Tigeot 778*19c468b4SFrançois Tigeot if (unlikely(ringbuf->space < bytes)) { 779*19c468b4SFrançois Tigeot ret = logical_ring_wait_for_space(ringbuf, ctx, bytes); 780*19c468b4SFrançois Tigeot if (unlikely(ret)) 781*19c468b4SFrançois Tigeot return ret; 782*19c468b4SFrançois Tigeot } 783*19c468b4SFrançois Tigeot 784*19c468b4SFrançois Tigeot return 0; 785*19c468b4SFrançois Tigeot } 786*19c468b4SFrançois Tigeot 787*19c468b4SFrançois Tigeot /** 788*19c468b4SFrançois Tigeot * intel_logical_ring_begin() - prepare the logical ringbuffer to accept some commands 789*19c468b4SFrançois Tigeot * 790*19c468b4SFrançois Tigeot * @ringbuf: Logical ringbuffer. 791*19c468b4SFrançois Tigeot * @num_dwords: number of DWORDs that we plan to write to the ringbuffer. 792*19c468b4SFrançois Tigeot * 793*19c468b4SFrançois Tigeot * The ringbuffer might not be ready to accept the commands right away (maybe it needs to 794*19c468b4SFrançois Tigeot * be wrapped, or wait a bit for the tail to be updated). This function takes care of that 795*19c468b4SFrançois Tigeot * and also preallocates a request (every workload submission is still mediated through 796*19c468b4SFrançois Tigeot * requests, same as it did with legacy ringbuffer submission). 797*19c468b4SFrançois Tigeot * 798*19c468b4SFrançois Tigeot * Return: non-zero if the ringbuffer is not ready to be written to. 799*19c468b4SFrançois Tigeot */ 800*19c468b4SFrançois Tigeot static int intel_logical_ring_begin(struct intel_ringbuffer *ringbuf, 801*19c468b4SFrançois Tigeot struct intel_context *ctx, int num_dwords) 802*19c468b4SFrançois Tigeot { 803*19c468b4SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 804*19c468b4SFrançois Tigeot struct drm_device *dev = ring->dev; 805*19c468b4SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 806*19c468b4SFrançois Tigeot int ret; 807*19c468b4SFrançois Tigeot 808*19c468b4SFrançois Tigeot ret = i915_gem_check_wedge(&dev_priv->gpu_error, 809*19c468b4SFrançois Tigeot dev_priv->mm.interruptible); 810*19c468b4SFrançois Tigeot if (ret) 811*19c468b4SFrançois Tigeot return ret; 812*19c468b4SFrançois Tigeot 813*19c468b4SFrançois Tigeot ret = logical_ring_prepare(ringbuf, ctx, num_dwords * sizeof(uint32_t)); 814*19c468b4SFrançois Tigeot if (ret) 815*19c468b4SFrançois Tigeot return ret; 816*19c468b4SFrançois Tigeot 817*19c468b4SFrançois Tigeot /* Preallocate the olr before touching the ring */ 818*19c468b4SFrançois Tigeot ret = i915_gem_request_alloc(ring, ctx); 819*19c468b4SFrançois Tigeot if (ret) 820*19c468b4SFrançois Tigeot return ret; 821*19c468b4SFrançois Tigeot 822*19c468b4SFrançois Tigeot ringbuf->space -= num_dwords * sizeof(uint32_t); 823*19c468b4SFrançois Tigeot return 0; 824*19c468b4SFrançois Tigeot } 825*19c468b4SFrançois Tigeot 8261b13d190SFrançois Tigeot /** 8271b13d190SFrançois Tigeot * execlists_submission() - submit a batchbuffer for execution, Execlists style 8281b13d190SFrançois Tigeot * @dev: DRM device. 8291b13d190SFrançois Tigeot * @file: DRM file. 8301b13d190SFrançois Tigeot * @ring: Engine Command Streamer to submit to. 8311b13d190SFrançois Tigeot * @ctx: Context to employ for this submission. 8321b13d190SFrançois Tigeot * @args: execbuffer call arguments. 8331b13d190SFrançois Tigeot * @vmas: list of vmas. 8341b13d190SFrançois Tigeot * @batch_obj: the batchbuffer to submit. 8351b13d190SFrançois Tigeot * @exec_start: batchbuffer start virtual address pointer. 836477eb7f9SFrançois Tigeot * @dispatch_flags: translated execbuffer call flags. 8371b13d190SFrançois Tigeot * 8381b13d190SFrançois Tigeot * This is the evil twin version of i915_gem_ringbuffer_submission. It abstracts 8391b13d190SFrançois Tigeot * away the submission details of the execbuffer ioctl call. 8401b13d190SFrançois Tigeot * 8411b13d190SFrançois Tigeot * Return: non-zero if the submission fails. 8421b13d190SFrançois Tigeot */ 8431b13d190SFrançois Tigeot int intel_execlists_submission(struct drm_device *dev, struct drm_file *file, 8441b13d190SFrançois Tigeot struct intel_engine_cs *ring, 8451b13d190SFrançois Tigeot struct intel_context *ctx, 8461b13d190SFrançois Tigeot struct drm_i915_gem_execbuffer2 *args, 8471b13d190SFrançois Tigeot struct list_head *vmas, 8481b13d190SFrançois Tigeot struct drm_i915_gem_object *batch_obj, 849477eb7f9SFrançois Tigeot u64 exec_start, u32 dispatch_flags) 8501b13d190SFrançois Tigeot { 8511b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 8521b13d190SFrançois Tigeot struct intel_ringbuffer *ringbuf = ctx->engine[ring->id].ringbuf; 8531b13d190SFrançois Tigeot int instp_mode; 8541b13d190SFrançois Tigeot u32 instp_mask; 8551b13d190SFrançois Tigeot int ret; 8561b13d190SFrançois Tigeot 8571b13d190SFrançois Tigeot instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK; 8581b13d190SFrançois Tigeot instp_mask = I915_EXEC_CONSTANTS_MASK; 8591b13d190SFrançois Tigeot switch (instp_mode) { 8601b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_GENERAL: 8611b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_ABSOLUTE: 8621b13d190SFrançois Tigeot case I915_EXEC_CONSTANTS_REL_SURFACE: 8631b13d190SFrançois Tigeot if (instp_mode != 0 && ring != &dev_priv->ring[RCS]) { 8641b13d190SFrançois Tigeot DRM_DEBUG("non-0 rel constants mode on non-RCS\n"); 8651b13d190SFrançois Tigeot return -EINVAL; 8661b13d190SFrançois Tigeot } 8671b13d190SFrançois Tigeot 8681b13d190SFrançois Tigeot if (instp_mode != dev_priv->relative_constants_mode) { 8691b13d190SFrançois Tigeot if (instp_mode == I915_EXEC_CONSTANTS_REL_SURFACE) { 8701b13d190SFrançois Tigeot DRM_DEBUG("rel surface constants mode invalid on gen5+\n"); 8711b13d190SFrançois Tigeot return -EINVAL; 8721b13d190SFrançois Tigeot } 8731b13d190SFrançois Tigeot 8741b13d190SFrançois Tigeot /* The HW changed the meaning on this bit on gen6 */ 8751b13d190SFrançois Tigeot instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE; 8761b13d190SFrançois Tigeot } 8771b13d190SFrançois Tigeot break; 8781b13d190SFrançois Tigeot default: 8791b13d190SFrançois Tigeot DRM_DEBUG("execbuf with unknown constants: %d\n", instp_mode); 8801b13d190SFrançois Tigeot return -EINVAL; 8811b13d190SFrançois Tigeot } 8821b13d190SFrançois Tigeot 8831b13d190SFrançois Tigeot if (args->num_cliprects != 0) { 8841b13d190SFrançois Tigeot DRM_DEBUG("clip rectangles are only valid on pre-gen5\n"); 8851b13d190SFrançois Tigeot return -EINVAL; 8861b13d190SFrançois Tigeot } else { 8871b13d190SFrançois Tigeot if (args->DR4 == 0xffffffff) { 8881b13d190SFrançois Tigeot DRM_DEBUG("UXA submitting garbage DR4, fixing up\n"); 8891b13d190SFrançois Tigeot args->DR4 = 0; 8901b13d190SFrançois Tigeot } 8911b13d190SFrançois Tigeot 8921b13d190SFrançois Tigeot if (args->DR1 || args->DR4 || args->cliprects_ptr) { 8931b13d190SFrançois Tigeot DRM_DEBUG("0 cliprects but dirt in cliprects fields\n"); 8941b13d190SFrançois Tigeot return -EINVAL; 8951b13d190SFrançois Tigeot } 8961b13d190SFrançois Tigeot } 8971b13d190SFrançois Tigeot 8981b13d190SFrançois Tigeot if (args->flags & I915_EXEC_GEN7_SOL_RESET) { 8991b13d190SFrançois Tigeot DRM_DEBUG("sol reset is gen7 only\n"); 9001b13d190SFrançois Tigeot return -EINVAL; 9011b13d190SFrançois Tigeot } 9021b13d190SFrançois Tigeot 9032c9916cdSFrançois Tigeot ret = execlists_move_to_gpu(ringbuf, ctx, vmas); 9041b13d190SFrançois Tigeot if (ret) 9051b13d190SFrançois Tigeot return ret; 9061b13d190SFrançois Tigeot 9071b13d190SFrançois Tigeot if (ring == &dev_priv->ring[RCS] && 9081b13d190SFrançois Tigeot instp_mode != dev_priv->relative_constants_mode) { 9092c9916cdSFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, ctx, 4); 9101b13d190SFrançois Tigeot if (ret) 9111b13d190SFrançois Tigeot return ret; 9121b13d190SFrançois Tigeot 9131b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 9141b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(1)); 9151b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, INSTPM); 9161b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, instp_mask << 16 | instp_mode); 9171b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 9181b13d190SFrançois Tigeot 9191b13d190SFrançois Tigeot dev_priv->relative_constants_mode = instp_mode; 9201b13d190SFrançois Tigeot } 9211b13d190SFrançois Tigeot 922477eb7f9SFrançois Tigeot ret = ring->emit_bb_start(ringbuf, ctx, exec_start, dispatch_flags); 9231b13d190SFrançois Tigeot if (ret) 9241b13d190SFrançois Tigeot return ret; 9251b13d190SFrançois Tigeot 926477eb7f9SFrançois Tigeot trace_i915_gem_ring_dispatch(intel_ring_get_request(ring), dispatch_flags); 927477eb7f9SFrançois Tigeot 9281b13d190SFrançois Tigeot i915_gem_execbuffer_move_to_active(vmas, ring); 9291b13d190SFrançois Tigeot i915_gem_execbuffer_retire_commands(dev, file, ring, batch_obj); 9301b13d190SFrançois Tigeot 9311b13d190SFrançois Tigeot return 0; 9321b13d190SFrançois Tigeot } 9331b13d190SFrançois Tigeot 9342c9916cdSFrançois Tigeot void intel_execlists_retire_requests(struct intel_engine_cs *ring) 9352c9916cdSFrançois Tigeot { 9362c9916cdSFrançois Tigeot struct drm_i915_gem_request *req, *tmp; 9372c9916cdSFrançois Tigeot struct list_head retired_list; 9382c9916cdSFrançois Tigeot 9392c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 9402c9916cdSFrançois Tigeot if (list_empty(&ring->execlist_retired_req_list)) 9412c9916cdSFrançois Tigeot return; 9422c9916cdSFrançois Tigeot 9432c9916cdSFrançois Tigeot INIT_LIST_HEAD(&retired_list); 9442c9916cdSFrançois Tigeot lockmgr(&ring->execlist_lock, LK_EXCLUSIVE); 9452c9916cdSFrançois Tigeot list_replace_init(&ring->execlist_retired_req_list, &retired_list); 9462c9916cdSFrançois Tigeot lockmgr(&ring->execlist_lock, LK_RELEASE); 9472c9916cdSFrançois Tigeot 9482c9916cdSFrançois Tigeot list_for_each_entry_safe(req, tmp, &retired_list, execlist_link) { 9492c9916cdSFrançois Tigeot struct intel_context *ctx = req->ctx; 9502c9916cdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 9512c9916cdSFrançois Tigeot ctx->engine[ring->id].state; 9522c9916cdSFrançois Tigeot 9532c9916cdSFrançois Tigeot if (ctx_obj && (ctx != ring->default_context)) 9542c9916cdSFrançois Tigeot intel_lr_context_unpin(ring, ctx); 9552c9916cdSFrançois Tigeot list_del(&req->execlist_link); 9562c9916cdSFrançois Tigeot i915_gem_request_unreference(req); 9572c9916cdSFrançois Tigeot } 9582c9916cdSFrançois Tigeot } 9592c9916cdSFrançois Tigeot 9601b13d190SFrançois Tigeot void intel_logical_ring_stop(struct intel_engine_cs *ring) 9611b13d190SFrançois Tigeot { 9621b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 9631b13d190SFrançois Tigeot int ret; 9641b13d190SFrançois Tigeot 9651b13d190SFrançois Tigeot if (!intel_ring_initialized(ring)) 9661b13d190SFrançois Tigeot return; 9671b13d190SFrançois Tigeot 9681b13d190SFrançois Tigeot ret = intel_ring_idle(ring); 9691b13d190SFrançois Tigeot if (ret && !i915_reset_in_progress(&to_i915(ring->dev)->gpu_error)) 9701b13d190SFrançois Tigeot DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n", 9711b13d190SFrançois Tigeot ring->name, ret); 9721b13d190SFrançois Tigeot 9731b13d190SFrançois Tigeot /* TODO: Is this correct with Execlists enabled? */ 9741b13d190SFrançois Tigeot I915_WRITE_MODE(ring, _MASKED_BIT_ENABLE(STOP_RING)); 9751b13d190SFrançois Tigeot if (wait_for_atomic((I915_READ_MODE(ring) & MODE_IDLE) != 0, 1000)) { 9761b13d190SFrançois Tigeot DRM_ERROR("%s :timed out trying to stop ring\n", ring->name); 9771b13d190SFrançois Tigeot return; 9781b13d190SFrançois Tigeot } 9791b13d190SFrançois Tigeot I915_WRITE_MODE(ring, _MASKED_BIT_DISABLE(STOP_RING)); 9801b13d190SFrançois Tigeot } 9811b13d190SFrançois Tigeot 9822c9916cdSFrançois Tigeot int logical_ring_flush_all_caches(struct intel_ringbuffer *ringbuf, 9832c9916cdSFrançois Tigeot struct intel_context *ctx) 9841b13d190SFrançois Tigeot { 9851b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 9861b13d190SFrançois Tigeot int ret; 9871b13d190SFrançois Tigeot 9881b13d190SFrançois Tigeot if (!ring->gpu_caches_dirty) 9891b13d190SFrançois Tigeot return 0; 9901b13d190SFrançois Tigeot 9912c9916cdSFrançois Tigeot ret = ring->emit_flush(ringbuf, ctx, 0, I915_GEM_GPU_DOMAINS); 9921b13d190SFrançois Tigeot if (ret) 9931b13d190SFrançois Tigeot return ret; 9941b13d190SFrançois Tigeot 9951b13d190SFrançois Tigeot ring->gpu_caches_dirty = false; 9961b13d190SFrançois Tigeot return 0; 9971b13d190SFrançois Tigeot } 9981b13d190SFrançois Tigeot 9992c9916cdSFrançois Tigeot static int intel_lr_context_pin(struct intel_engine_cs *ring, 10001b13d190SFrançois Tigeot struct intel_context *ctx) 10011b13d190SFrançois Tigeot { 10022c9916cdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ctx->engine[ring->id].state; 10032c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf = ctx->engine[ring->id].ringbuf; 10042c9916cdSFrançois Tigeot int ret = 0; 10052c9916cdSFrançois Tigeot 10062c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 10072c9916cdSFrançois Tigeot if (ctx->engine[ring->id].pin_count++ == 0) { 10082c9916cdSFrançois Tigeot ret = i915_gem_obj_ggtt_pin(ctx_obj, 10092c9916cdSFrançois Tigeot GEN8_LR_CONTEXT_ALIGN, 0); 10102c9916cdSFrançois Tigeot if (ret) 10112c9916cdSFrançois Tigeot goto reset_pin_count; 10122c9916cdSFrançois Tigeot 10132c9916cdSFrançois Tigeot ret = intel_pin_and_map_ringbuffer_obj(ring->dev, ringbuf); 10142c9916cdSFrançois Tigeot if (ret) 10152c9916cdSFrançois Tigeot goto unpin_ctx_obj; 1016*19c468b4SFrançois Tigeot 1017*19c468b4SFrançois Tigeot ctx_obj->dirty = true; 10182c9916cdSFrançois Tigeot } 10192c9916cdSFrançois Tigeot 10202c9916cdSFrançois Tigeot return ret; 10212c9916cdSFrançois Tigeot 10222c9916cdSFrançois Tigeot unpin_ctx_obj: 10232c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 10242c9916cdSFrançois Tigeot reset_pin_count: 10252c9916cdSFrançois Tigeot ctx->engine[ring->id].pin_count = 0; 10262c9916cdSFrançois Tigeot 10272c9916cdSFrançois Tigeot return ret; 10282c9916cdSFrançois Tigeot } 10292c9916cdSFrançois Tigeot 10302c9916cdSFrançois Tigeot void intel_lr_context_unpin(struct intel_engine_cs *ring, 10312c9916cdSFrançois Tigeot struct intel_context *ctx) 10322c9916cdSFrançois Tigeot { 10332c9916cdSFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ctx->engine[ring->id].state; 10342c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf = ctx->engine[ring->id].ringbuf; 10352c9916cdSFrançois Tigeot 10362c9916cdSFrançois Tigeot if (ctx_obj) { 10372c9916cdSFrançois Tigeot WARN_ON(!mutex_is_locked(&ring->dev->struct_mutex)); 10382c9916cdSFrançois Tigeot if (--ctx->engine[ring->id].pin_count == 0) { 10392c9916cdSFrançois Tigeot intel_unpin_ringbuffer_obj(ringbuf); 10402c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 10412c9916cdSFrançois Tigeot } 10422c9916cdSFrançois Tigeot } 10432c9916cdSFrançois Tigeot } 10442c9916cdSFrançois Tigeot 10452c9916cdSFrançois Tigeot static int intel_logical_ring_workarounds_emit(struct intel_engine_cs *ring, 10462c9916cdSFrançois Tigeot struct intel_context *ctx) 10472c9916cdSFrançois Tigeot { 10482c9916cdSFrançois Tigeot int ret, i; 10492c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf = ctx->engine[ring->id].ringbuf; 10502c9916cdSFrançois Tigeot struct drm_device *dev = ring->dev; 10512c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 10522c9916cdSFrançois Tigeot struct i915_workarounds *w = &dev_priv->workarounds; 10532c9916cdSFrançois Tigeot 10542c9916cdSFrançois Tigeot if (WARN_ON_ONCE(w->count == 0)) 10552c9916cdSFrançois Tigeot return 0; 10562c9916cdSFrançois Tigeot 10572c9916cdSFrançois Tigeot ring->gpu_caches_dirty = true; 10582c9916cdSFrançois Tigeot ret = logical_ring_flush_all_caches(ringbuf, ctx); 10592c9916cdSFrançois Tigeot if (ret) 10602c9916cdSFrançois Tigeot return ret; 10612c9916cdSFrançois Tigeot 10622c9916cdSFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, ctx, w->count * 2 + 2); 10632c9916cdSFrançois Tigeot if (ret) 10642c9916cdSFrançois Tigeot return ret; 10652c9916cdSFrançois Tigeot 10662c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_LOAD_REGISTER_IMM(w->count)); 10672c9916cdSFrançois Tigeot for (i = 0; i < w->count; i++) { 10682c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, w->reg[i].addr); 10692c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, w->reg[i].value); 10702c9916cdSFrançois Tigeot } 10712c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 10722c9916cdSFrançois Tigeot 10732c9916cdSFrançois Tigeot intel_logical_ring_advance(ringbuf); 10742c9916cdSFrançois Tigeot 10752c9916cdSFrançois Tigeot ring->gpu_caches_dirty = true; 10762c9916cdSFrançois Tigeot ret = logical_ring_flush_all_caches(ringbuf, ctx); 10772c9916cdSFrançois Tigeot if (ret) 10782c9916cdSFrançois Tigeot return ret; 10792c9916cdSFrançois Tigeot 10802c9916cdSFrançois Tigeot return 0; 10812c9916cdSFrançois Tigeot } 10822c9916cdSFrançois Tigeot 10831b13d190SFrançois Tigeot static int gen8_init_common_ring(struct intel_engine_cs *ring) 10841b13d190SFrançois Tigeot { 10851b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 10861b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 10871b13d190SFrançois Tigeot 10881b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~(ring->irq_enable_mask | ring->irq_keep_mask)); 10891b13d190SFrançois Tigeot I915_WRITE(RING_HWSTAM(ring->mmio_base), 0xffffffff); 10901b13d190SFrançois Tigeot 1091477eb7f9SFrançois Tigeot if (ring->status_page.obj) { 1092477eb7f9SFrançois Tigeot I915_WRITE(RING_HWS_PGA(ring->mmio_base), 1093477eb7f9SFrançois Tigeot (u32)ring->status_page.gfx_addr); 1094477eb7f9SFrançois Tigeot POSTING_READ(RING_HWS_PGA(ring->mmio_base)); 1095477eb7f9SFrançois Tigeot } 1096477eb7f9SFrançois Tigeot 10971b13d190SFrançois Tigeot I915_WRITE(RING_MODE_GEN7(ring), 10981b13d190SFrançois Tigeot _MASKED_BIT_DISABLE(GFX_REPLAY_MODE) | 10991b13d190SFrançois Tigeot _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE)); 11001b13d190SFrançois Tigeot POSTING_READ(RING_MODE_GEN7(ring)); 11012c9916cdSFrançois Tigeot ring->next_context_status_buffer = 0; 11021b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Execlists enabled for %s\n", ring->name); 11031b13d190SFrançois Tigeot 11041b13d190SFrançois Tigeot memset(&ring->hangcheck, 0, sizeof(ring->hangcheck)); 11051b13d190SFrançois Tigeot 11061b13d190SFrançois Tigeot return 0; 11071b13d190SFrançois Tigeot } 11081b13d190SFrançois Tigeot 11091b13d190SFrançois Tigeot static int gen8_init_render_ring(struct intel_engine_cs *ring) 11101b13d190SFrançois Tigeot { 11111b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 11121b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 11131b13d190SFrançois Tigeot int ret; 11141b13d190SFrançois Tigeot 11151b13d190SFrançois Tigeot ret = gen8_init_common_ring(ring); 11161b13d190SFrançois Tigeot if (ret) 11171b13d190SFrançois Tigeot return ret; 11181b13d190SFrançois Tigeot 11191b13d190SFrançois Tigeot /* We need to disable the AsyncFlip performance optimisations in order 11201b13d190SFrançois Tigeot * to use MI_WAIT_FOR_EVENT within the CS. It should already be 11211b13d190SFrançois Tigeot * programmed to '1' on all products. 11221b13d190SFrançois Tigeot * 11231b13d190SFrançois Tigeot * WaDisableAsyncFlipPerfMode:snb,ivb,hsw,vlv,bdw,chv 11241b13d190SFrançois Tigeot */ 11251b13d190SFrançois Tigeot I915_WRITE(MI_MODE, _MASKED_BIT_ENABLE(ASYNC_FLIP_PERF_DISABLE)); 11261b13d190SFrançois Tigeot 11271b13d190SFrançois Tigeot I915_WRITE(INSTPM, _MASKED_BIT_ENABLE(INSTPM_FORCE_ORDERING)); 11281b13d190SFrançois Tigeot 11292c9916cdSFrançois Tigeot return init_workarounds_ring(ring); 11301b13d190SFrançois Tigeot } 11311b13d190SFrançois Tigeot 1132477eb7f9SFrançois Tigeot static int gen9_init_render_ring(struct intel_engine_cs *ring) 1133477eb7f9SFrançois Tigeot { 1134477eb7f9SFrançois Tigeot int ret; 1135477eb7f9SFrançois Tigeot 1136477eb7f9SFrançois Tigeot ret = gen8_init_common_ring(ring); 1137477eb7f9SFrançois Tigeot if (ret) 1138477eb7f9SFrançois Tigeot return ret; 1139477eb7f9SFrançois Tigeot 1140477eb7f9SFrançois Tigeot return init_workarounds_ring(ring); 1141477eb7f9SFrançois Tigeot } 1142477eb7f9SFrançois Tigeot 11431b13d190SFrançois Tigeot static int gen8_emit_bb_start(struct intel_ringbuffer *ringbuf, 11442c9916cdSFrançois Tigeot struct intel_context *ctx, 1145477eb7f9SFrançois Tigeot u64 offset, unsigned dispatch_flags) 11461b13d190SFrançois Tigeot { 1147477eb7f9SFrançois Tigeot bool ppgtt = !(dispatch_flags & I915_DISPATCH_SECURE); 11481b13d190SFrançois Tigeot int ret; 11491b13d190SFrançois Tigeot 11502c9916cdSFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, ctx, 4); 11511b13d190SFrançois Tigeot if (ret) 11521b13d190SFrançois Tigeot return ret; 11531b13d190SFrançois Tigeot 11541b13d190SFrançois Tigeot /* FIXME(BDW): Address space and security selectors. */ 11551b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_BATCH_BUFFER_START_GEN8 | (ppgtt<<8)); 11561b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, lower_32_bits(offset)); 11571b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, upper_32_bits(offset)); 11581b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 11591b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 11601b13d190SFrançois Tigeot 11611b13d190SFrançois Tigeot return 0; 11621b13d190SFrançois Tigeot } 11631b13d190SFrançois Tigeot 11641b13d190SFrançois Tigeot static bool gen8_logical_ring_get_irq(struct intel_engine_cs *ring) 11651b13d190SFrançois Tigeot { 11661b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 11671b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 11681b13d190SFrançois Tigeot 11692c9916cdSFrançois Tigeot if (WARN_ON(!intel_irqs_enabled(dev_priv))) 11701b13d190SFrançois Tigeot return false; 11711b13d190SFrançois Tigeot 11721b13d190SFrançois Tigeot lockmgr(&dev_priv->irq_lock, LK_EXCLUSIVE); 11731b13d190SFrançois Tigeot if (ring->irq_refcount++ == 0) { 11741b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~(ring->irq_enable_mask | ring->irq_keep_mask)); 11751b13d190SFrançois Tigeot POSTING_READ(RING_IMR(ring->mmio_base)); 11761b13d190SFrançois Tigeot } 11771b13d190SFrançois Tigeot lockmgr(&dev_priv->irq_lock, LK_RELEASE); 11781b13d190SFrançois Tigeot 11791b13d190SFrançois Tigeot return true; 11801b13d190SFrançois Tigeot } 11811b13d190SFrançois Tigeot 11821b13d190SFrançois Tigeot static void gen8_logical_ring_put_irq(struct intel_engine_cs *ring) 11831b13d190SFrançois Tigeot { 11841b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 11851b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 11861b13d190SFrançois Tigeot 11871b13d190SFrançois Tigeot lockmgr(&dev_priv->irq_lock, LK_EXCLUSIVE); 11881b13d190SFrançois Tigeot if (--ring->irq_refcount == 0) { 11891b13d190SFrançois Tigeot I915_WRITE_IMR(ring, ~ring->irq_keep_mask); 11901b13d190SFrançois Tigeot POSTING_READ(RING_IMR(ring->mmio_base)); 11911b13d190SFrançois Tigeot } 11921b13d190SFrançois Tigeot lockmgr(&dev_priv->irq_lock, LK_RELEASE); 11931b13d190SFrançois Tigeot } 11941b13d190SFrançois Tigeot 11951b13d190SFrançois Tigeot static int gen8_emit_flush(struct intel_ringbuffer *ringbuf, 11962c9916cdSFrançois Tigeot struct intel_context *ctx, 11971b13d190SFrançois Tigeot u32 invalidate_domains, 11981b13d190SFrançois Tigeot u32 unused) 11991b13d190SFrançois Tigeot { 12001b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 12011b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 12021b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 12031b13d190SFrançois Tigeot uint32_t cmd; 12041b13d190SFrançois Tigeot int ret; 12051b13d190SFrançois Tigeot 12062c9916cdSFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, ctx, 4); 12071b13d190SFrançois Tigeot if (ret) 12081b13d190SFrançois Tigeot return ret; 12091b13d190SFrançois Tigeot 12101b13d190SFrançois Tigeot cmd = MI_FLUSH_DW + 1; 12111b13d190SFrançois Tigeot 12122c9916cdSFrançois Tigeot /* We always require a command barrier so that subsequent 12132c9916cdSFrançois Tigeot * commands, such as breadcrumb interrupts, are strictly ordered 12142c9916cdSFrançois Tigeot * wrt the contents of the write cache being flushed to memory 12152c9916cdSFrançois Tigeot * (and thus being coherent from the CPU). 12162c9916cdSFrançois Tigeot */ 12172c9916cdSFrançois Tigeot cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW; 12182c9916cdSFrançois Tigeot 12192c9916cdSFrançois Tigeot if (invalidate_domains & I915_GEM_GPU_DOMAINS) { 12202c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_TLB; 12212c9916cdSFrançois Tigeot if (ring == &dev_priv->ring[VCS]) 12222c9916cdSFrançois Tigeot cmd |= MI_INVALIDATE_BSD; 12231b13d190SFrançois Tigeot } 12241b13d190SFrançois Tigeot 12251b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, cmd); 12261b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 12271b13d190SFrançois Tigeot I915_GEM_HWS_SCRATCH_ADDR | 12281b13d190SFrançois Tigeot MI_FLUSH_DW_USE_GTT); 12291b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* upper addr */ 12301b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); /* value */ 12311b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 12321b13d190SFrançois Tigeot 12331b13d190SFrançois Tigeot return 0; 12341b13d190SFrançois Tigeot } 12351b13d190SFrançois Tigeot 12361b13d190SFrançois Tigeot static int gen8_emit_flush_render(struct intel_ringbuffer *ringbuf, 12372c9916cdSFrançois Tigeot struct intel_context *ctx, 12381b13d190SFrançois Tigeot u32 invalidate_domains, 12391b13d190SFrançois Tigeot u32 flush_domains) 12401b13d190SFrançois Tigeot { 12411b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 12421b13d190SFrançois Tigeot u32 scratch_addr = ring->scratch.gtt_offset + 2 * CACHELINE_BYTES; 1243*19c468b4SFrançois Tigeot bool vf_flush_wa; 12441b13d190SFrançois Tigeot u32 flags = 0; 12451b13d190SFrançois Tigeot int ret; 12461b13d190SFrançois Tigeot 12471b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CS_STALL; 12481b13d190SFrançois Tigeot 12491b13d190SFrançois Tigeot if (flush_domains) { 12501b13d190SFrançois Tigeot flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH; 12511b13d190SFrançois Tigeot flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH; 12521b13d190SFrançois Tigeot } 12531b13d190SFrançois Tigeot 12541b13d190SFrançois Tigeot if (invalidate_domains) { 12551b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TLB_INVALIDATE; 12561b13d190SFrançois Tigeot flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE; 12571b13d190SFrançois Tigeot flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE; 12581b13d190SFrançois Tigeot flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE; 12591b13d190SFrançois Tigeot flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE; 12601b13d190SFrançois Tigeot flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE; 12611b13d190SFrançois Tigeot flags |= PIPE_CONTROL_QW_WRITE; 12621b13d190SFrançois Tigeot flags |= PIPE_CONTROL_GLOBAL_GTT_IVB; 12631b13d190SFrançois Tigeot } 12641b13d190SFrançois Tigeot 1265*19c468b4SFrançois Tigeot /* 1266*19c468b4SFrançois Tigeot * On GEN9+ Before VF_CACHE_INVALIDATE we need to emit a NULL pipe 1267*19c468b4SFrançois Tigeot * control. 1268*19c468b4SFrançois Tigeot */ 1269*19c468b4SFrançois Tigeot vf_flush_wa = INTEL_INFO(ring->dev)->gen >= 9 && 1270*19c468b4SFrançois Tigeot flags & PIPE_CONTROL_VF_CACHE_INVALIDATE; 1271*19c468b4SFrançois Tigeot 1272*19c468b4SFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, ctx, vf_flush_wa ? 12 : 6); 12731b13d190SFrançois Tigeot if (ret) 12741b13d190SFrançois Tigeot return ret; 12751b13d190SFrançois Tigeot 1276*19c468b4SFrançois Tigeot if (vf_flush_wa) { 1277*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 1278*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1279*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1280*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1281*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1282*19c468b4SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 1283*19c468b4SFrançois Tigeot } 1284*19c468b4SFrançois Tigeot 12851b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, GFX_OP_PIPE_CONTROL(6)); 12861b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, flags); 12871b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, scratch_addr); 12881b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 12891b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 12901b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 12911b13d190SFrançois Tigeot intel_logical_ring_advance(ringbuf); 12921b13d190SFrançois Tigeot 12931b13d190SFrançois Tigeot return 0; 12941b13d190SFrançois Tigeot } 12951b13d190SFrançois Tigeot 12961b13d190SFrançois Tigeot static u32 gen8_get_seqno(struct intel_engine_cs *ring, bool lazy_coherency) 12971b13d190SFrançois Tigeot { 12981b13d190SFrançois Tigeot return intel_read_status_page(ring, I915_GEM_HWS_INDEX); 12991b13d190SFrançois Tigeot } 13001b13d190SFrançois Tigeot 13011b13d190SFrançois Tigeot static void gen8_set_seqno(struct intel_engine_cs *ring, u32 seqno) 13021b13d190SFrançois Tigeot { 13031b13d190SFrançois Tigeot intel_write_status_page(ring, I915_GEM_HWS_INDEX, seqno); 13041b13d190SFrançois Tigeot } 13051b13d190SFrançois Tigeot 13062c9916cdSFrançois Tigeot static int gen8_emit_request(struct intel_ringbuffer *ringbuf, 13072c9916cdSFrançois Tigeot struct drm_i915_gem_request *request) 13081b13d190SFrançois Tigeot { 13091b13d190SFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 13101b13d190SFrançois Tigeot u32 cmd; 13111b13d190SFrançois Tigeot int ret; 13121b13d190SFrançois Tigeot 1313477eb7f9SFrançois Tigeot /* 1314477eb7f9SFrançois Tigeot * Reserve space for 2 NOOPs at the end of each request to be 1315477eb7f9SFrançois Tigeot * used as a workaround for not being allowed to do lite 1316477eb7f9SFrançois Tigeot * restore with HEAD==TAIL (WaIdleLiteRestore). 1317477eb7f9SFrançois Tigeot */ 1318477eb7f9SFrançois Tigeot ret = intel_logical_ring_begin(ringbuf, request->ctx, 8); 13191b13d190SFrançois Tigeot if (ret) 13201b13d190SFrançois Tigeot return ret; 13211b13d190SFrançois Tigeot 13222c9916cdSFrançois Tigeot cmd = MI_STORE_DWORD_IMM_GEN4; 13231b13d190SFrançois Tigeot cmd |= MI_GLOBAL_GTT; 13241b13d190SFrançois Tigeot 13251b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, cmd); 13261b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 13271b13d190SFrançois Tigeot (ring->status_page.gfx_addr + 13281b13d190SFrançois Tigeot (I915_GEM_HWS_INDEX << MI_STORE_DWORD_INDEX_SHIFT))); 13291b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, 0); 13302c9916cdSFrançois Tigeot intel_logical_ring_emit(ringbuf, 13312c9916cdSFrançois Tigeot i915_gem_request_get_seqno(ring->outstanding_lazy_request)); 13321b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_USER_INTERRUPT); 13331b13d190SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 13342c9916cdSFrançois Tigeot intel_logical_ring_advance_and_submit(ringbuf, request->ctx, request); 13351b13d190SFrançois Tigeot 1336477eb7f9SFrançois Tigeot /* 1337477eb7f9SFrançois Tigeot * Here we add two extra NOOPs as padding to avoid 1338477eb7f9SFrançois Tigeot * lite restore of a context with HEAD==TAIL. 1339477eb7f9SFrançois Tigeot */ 1340477eb7f9SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1341477eb7f9SFrançois Tigeot intel_logical_ring_emit(ringbuf, MI_NOOP); 1342477eb7f9SFrançois Tigeot intel_logical_ring_advance(ringbuf); 1343477eb7f9SFrançois Tigeot 13441b13d190SFrançois Tigeot return 0; 13451b13d190SFrançois Tigeot } 13461b13d190SFrançois Tigeot 1347477eb7f9SFrançois Tigeot static int intel_lr_context_render_state_init(struct intel_engine_cs *ring, 1348477eb7f9SFrançois Tigeot struct intel_context *ctx) 1349477eb7f9SFrançois Tigeot { 1350477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf = ctx->engine[ring->id].ringbuf; 1351477eb7f9SFrançois Tigeot struct render_state so; 1352477eb7f9SFrançois Tigeot struct drm_i915_file_private *file_priv = ctx->file_priv; 1353477eb7f9SFrançois Tigeot struct drm_file *file = file_priv ? file_priv->file : NULL; 1354477eb7f9SFrançois Tigeot int ret; 1355477eb7f9SFrançois Tigeot 1356477eb7f9SFrançois Tigeot ret = i915_gem_render_state_prepare(ring, &so); 1357477eb7f9SFrançois Tigeot if (ret) 1358477eb7f9SFrançois Tigeot return ret; 1359477eb7f9SFrançois Tigeot 1360477eb7f9SFrançois Tigeot if (so.rodata == NULL) 1361477eb7f9SFrançois Tigeot return 0; 1362477eb7f9SFrançois Tigeot 1363477eb7f9SFrançois Tigeot ret = ring->emit_bb_start(ringbuf, 1364477eb7f9SFrançois Tigeot ctx, 1365477eb7f9SFrançois Tigeot so.ggtt_offset, 1366477eb7f9SFrançois Tigeot I915_DISPATCH_SECURE); 1367477eb7f9SFrançois Tigeot if (ret) 1368477eb7f9SFrançois Tigeot goto out; 1369477eb7f9SFrançois Tigeot 1370477eb7f9SFrançois Tigeot i915_vma_move_to_active(i915_gem_obj_to_ggtt(so.obj), ring); 1371477eb7f9SFrançois Tigeot 1372477eb7f9SFrançois Tigeot ret = __i915_add_request(ring, file, so.obj); 1373477eb7f9SFrançois Tigeot /* intel_logical_ring_add_request moves object to inactive if it 1374477eb7f9SFrançois Tigeot * fails */ 1375477eb7f9SFrançois Tigeot out: 1376477eb7f9SFrançois Tigeot i915_gem_render_state_fini(&so); 1377477eb7f9SFrançois Tigeot return ret; 1378477eb7f9SFrançois Tigeot } 1379477eb7f9SFrançois Tigeot 13802c9916cdSFrançois Tigeot static int gen8_init_rcs_context(struct intel_engine_cs *ring, 13812c9916cdSFrançois Tigeot struct intel_context *ctx) 13822c9916cdSFrançois Tigeot { 13832c9916cdSFrançois Tigeot int ret; 13842c9916cdSFrançois Tigeot 13852c9916cdSFrançois Tigeot ret = intel_logical_ring_workarounds_emit(ring, ctx); 13862c9916cdSFrançois Tigeot if (ret) 13872c9916cdSFrançois Tigeot return ret; 13882c9916cdSFrançois Tigeot 13892c9916cdSFrançois Tigeot return intel_lr_context_render_state_init(ring, ctx); 13902c9916cdSFrançois Tigeot } 13912c9916cdSFrançois Tigeot 13921b13d190SFrançois Tigeot /** 13931b13d190SFrançois Tigeot * intel_logical_ring_cleanup() - deallocate the Engine Command Streamer 13941b13d190SFrançois Tigeot * 13951b13d190SFrançois Tigeot * @ring: Engine Command Streamer. 13961b13d190SFrançois Tigeot * 13971b13d190SFrançois Tigeot */ 13981b13d190SFrançois Tigeot void intel_logical_ring_cleanup(struct intel_engine_cs *ring) 13991b13d190SFrançois Tigeot { 14002c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv; 14011b13d190SFrançois Tigeot 14021b13d190SFrançois Tigeot if (!intel_ring_initialized(ring)) 14031b13d190SFrançois Tigeot return; 14041b13d190SFrançois Tigeot 14052c9916cdSFrançois Tigeot dev_priv = ring->dev->dev_private; 14062c9916cdSFrançois Tigeot 14071b13d190SFrançois Tigeot intel_logical_ring_stop(ring); 14081b13d190SFrançois Tigeot WARN_ON((I915_READ_MODE(ring) & MODE_IDLE) == 0); 14092c9916cdSFrançois Tigeot i915_gem_request_assign(&ring->outstanding_lazy_request, NULL); 14101b13d190SFrançois Tigeot 14111b13d190SFrançois Tigeot if (ring->cleanup) 14121b13d190SFrançois Tigeot ring->cleanup(ring); 14131b13d190SFrançois Tigeot 14141b13d190SFrançois Tigeot i915_cmd_parser_fini_ring(ring); 1415*19c468b4SFrançois Tigeot i915_gem_batch_pool_fini(&ring->batch_pool); 14161b13d190SFrançois Tigeot 14171b13d190SFrançois Tigeot if (ring->status_page.obj) { 14181b13d190SFrançois Tigeot kunmap(ring->status_page.obj->pages[0]); 14191b13d190SFrançois Tigeot ring->status_page.obj = NULL; 14201b13d190SFrançois Tigeot } 14211b13d190SFrançois Tigeot } 14221b13d190SFrançois Tigeot 14231b13d190SFrançois Tigeot static int logical_ring_init(struct drm_device *dev, struct intel_engine_cs *ring) 14241b13d190SFrançois Tigeot { 14251b13d190SFrançois Tigeot int ret; 14261b13d190SFrançois Tigeot 14271b13d190SFrançois Tigeot /* Intentionally left blank. */ 14281b13d190SFrançois Tigeot ring->buffer = NULL; 14291b13d190SFrançois Tigeot 14301b13d190SFrançois Tigeot ring->dev = dev; 14311b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->active_list); 14321b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->request_list); 1433*19c468b4SFrançois Tigeot i915_gem_batch_pool_init(dev, &ring->batch_pool); 14341b13d190SFrançois Tigeot init_waitqueue_head(&ring->irq_queue); 14351b13d190SFrançois Tigeot 14361b13d190SFrançois Tigeot INIT_LIST_HEAD(&ring->execlist_queue); 14372c9916cdSFrançois Tigeot INIT_LIST_HEAD(&ring->execlist_retired_req_list); 14381b13d190SFrançois Tigeot lockinit(&ring->execlist_lock, "i915el", 0, LK_CANRECURSE); 14391b13d190SFrançois Tigeot 14401b13d190SFrançois Tigeot ret = i915_cmd_parser_init_ring(ring); 14411b13d190SFrançois Tigeot if (ret) 14421b13d190SFrançois Tigeot return ret; 14431b13d190SFrançois Tigeot 14441b13d190SFrançois Tigeot ret = intel_lr_context_deferred_create(ring->default_context, ring); 14451b13d190SFrançois Tigeot 14461b13d190SFrançois Tigeot return ret; 14471b13d190SFrançois Tigeot } 14481b13d190SFrançois Tigeot 14491b13d190SFrançois Tigeot static int logical_render_ring_init(struct drm_device *dev) 14501b13d190SFrançois Tigeot { 14511b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 14521b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[RCS]; 14532c9916cdSFrançois Tigeot int ret; 14541b13d190SFrançois Tigeot 14551b13d190SFrançois Tigeot ring->name = "render ring"; 14561b13d190SFrançois Tigeot ring->id = RCS; 14571b13d190SFrançois Tigeot ring->mmio_base = RENDER_RING_BASE; 14581b13d190SFrançois Tigeot ring->irq_enable_mask = 14591b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_RCS_IRQ_SHIFT; 14601b13d190SFrançois Tigeot ring->irq_keep_mask = 14611b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_RCS_IRQ_SHIFT; 14621b13d190SFrançois Tigeot if (HAS_L3_DPF(dev)) 14631b13d190SFrançois Tigeot ring->irq_keep_mask |= GT_RENDER_L3_PARITY_ERROR_INTERRUPT; 14641b13d190SFrançois Tigeot 1465477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->gen >= 9) 1466477eb7f9SFrançois Tigeot ring->init_hw = gen9_init_render_ring; 1467477eb7f9SFrançois Tigeot else 14682c9916cdSFrançois Tigeot ring->init_hw = gen8_init_render_ring; 14692c9916cdSFrançois Tigeot ring->init_context = gen8_init_rcs_context; 14701b13d190SFrançois Tigeot ring->cleanup = intel_fini_pipe_control; 14711b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 14721b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 14731b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 14741b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush_render; 14751b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 14761b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 14771b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 14781b13d190SFrançois Tigeot 14792c9916cdSFrançois Tigeot ring->dev = dev; 14802c9916cdSFrançois Tigeot ret = logical_ring_init(dev, ring); 14812c9916cdSFrançois Tigeot if (ret) 14822c9916cdSFrançois Tigeot return ret; 14832c9916cdSFrançois Tigeot 14842c9916cdSFrançois Tigeot return intel_init_pipe_control(ring); 14851b13d190SFrançois Tigeot } 14861b13d190SFrançois Tigeot 14871b13d190SFrançois Tigeot static int logical_bsd_ring_init(struct drm_device *dev) 14881b13d190SFrançois Tigeot { 14891b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 14901b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VCS]; 14911b13d190SFrançois Tigeot 14921b13d190SFrançois Tigeot ring->name = "bsd ring"; 14931b13d190SFrançois Tigeot ring->id = VCS; 14941b13d190SFrançois Tigeot ring->mmio_base = GEN6_BSD_RING_BASE; 14951b13d190SFrançois Tigeot ring->irq_enable_mask = 14961b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VCS1_IRQ_SHIFT; 14971b13d190SFrançois Tigeot ring->irq_keep_mask = 14981b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS1_IRQ_SHIFT; 14991b13d190SFrançois Tigeot 15002c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 15011b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 15021b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 15031b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 15041b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 15051b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 15061b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 15071b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 15081b13d190SFrançois Tigeot 15091b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 15101b13d190SFrançois Tigeot } 15111b13d190SFrançois Tigeot 15121b13d190SFrançois Tigeot static int logical_bsd2_ring_init(struct drm_device *dev) 15131b13d190SFrançois Tigeot { 15141b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 15151b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VCS2]; 15161b13d190SFrançois Tigeot 15171b13d190SFrançois Tigeot ring->name = "bds2 ring"; 15181b13d190SFrançois Tigeot ring->id = VCS2; 15191b13d190SFrançois Tigeot ring->mmio_base = GEN8_BSD2_RING_BASE; 15201b13d190SFrançois Tigeot ring->irq_enable_mask = 15211b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VCS2_IRQ_SHIFT; 15221b13d190SFrançois Tigeot ring->irq_keep_mask = 15231b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS2_IRQ_SHIFT; 15241b13d190SFrançois Tigeot 15252c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 15261b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 15271b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 15281b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 15291b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 15301b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 15311b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 15321b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 15331b13d190SFrançois Tigeot 15341b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 15351b13d190SFrançois Tigeot } 15361b13d190SFrançois Tigeot 15371b13d190SFrançois Tigeot static int logical_blt_ring_init(struct drm_device *dev) 15381b13d190SFrançois Tigeot { 15391b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 15401b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[BCS]; 15411b13d190SFrançois Tigeot 15421b13d190SFrançois Tigeot ring->name = "blitter ring"; 15431b13d190SFrançois Tigeot ring->id = BCS; 15441b13d190SFrançois Tigeot ring->mmio_base = BLT_RING_BASE; 15451b13d190SFrançois Tigeot ring->irq_enable_mask = 15461b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_BCS_IRQ_SHIFT; 15471b13d190SFrançois Tigeot ring->irq_keep_mask = 15481b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_BCS_IRQ_SHIFT; 15491b13d190SFrançois Tigeot 15502c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 15511b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 15521b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 15531b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 15541b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 15551b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 15561b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 15571b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 15581b13d190SFrançois Tigeot 15591b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 15601b13d190SFrançois Tigeot } 15611b13d190SFrançois Tigeot 15621b13d190SFrançois Tigeot static int logical_vebox_ring_init(struct drm_device *dev) 15631b13d190SFrançois Tigeot { 15641b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 15651b13d190SFrançois Tigeot struct intel_engine_cs *ring = &dev_priv->ring[VECS]; 15661b13d190SFrançois Tigeot 15671b13d190SFrançois Tigeot ring->name = "video enhancement ring"; 15681b13d190SFrançois Tigeot ring->id = VECS; 15691b13d190SFrançois Tigeot ring->mmio_base = VEBOX_RING_BASE; 15701b13d190SFrançois Tigeot ring->irq_enable_mask = 15711b13d190SFrançois Tigeot GT_RENDER_USER_INTERRUPT << GEN8_VECS_IRQ_SHIFT; 15721b13d190SFrançois Tigeot ring->irq_keep_mask = 15731b13d190SFrançois Tigeot GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VECS_IRQ_SHIFT; 15741b13d190SFrançois Tigeot 15752c9916cdSFrançois Tigeot ring->init_hw = gen8_init_common_ring; 15761b13d190SFrançois Tigeot ring->get_seqno = gen8_get_seqno; 15771b13d190SFrançois Tigeot ring->set_seqno = gen8_set_seqno; 15781b13d190SFrançois Tigeot ring->emit_request = gen8_emit_request; 15791b13d190SFrançois Tigeot ring->emit_flush = gen8_emit_flush; 15801b13d190SFrançois Tigeot ring->irq_get = gen8_logical_ring_get_irq; 15811b13d190SFrançois Tigeot ring->irq_put = gen8_logical_ring_put_irq; 15821b13d190SFrançois Tigeot ring->emit_bb_start = gen8_emit_bb_start; 15831b13d190SFrançois Tigeot 15841b13d190SFrançois Tigeot return logical_ring_init(dev, ring); 15851b13d190SFrançois Tigeot } 15861b13d190SFrançois Tigeot 15871b13d190SFrançois Tigeot /** 15881b13d190SFrançois Tigeot * intel_logical_rings_init() - allocate, populate and init the Engine Command Streamers 15891b13d190SFrançois Tigeot * @dev: DRM device. 15901b13d190SFrançois Tigeot * 15911b13d190SFrançois Tigeot * This function inits the engines for an Execlists submission style (the equivalent in the 15921b13d190SFrançois Tigeot * legacy ringbuffer submission world would be i915_gem_init_rings). It does it only for 15931b13d190SFrançois Tigeot * those engines that are present in the hardware. 15941b13d190SFrançois Tigeot * 15951b13d190SFrançois Tigeot * Return: non-zero if the initialization failed. 15961b13d190SFrançois Tigeot */ 15971b13d190SFrançois Tigeot int intel_logical_rings_init(struct drm_device *dev) 15981b13d190SFrançois Tigeot { 15991b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 16001b13d190SFrançois Tigeot int ret; 16011b13d190SFrançois Tigeot 16021b13d190SFrançois Tigeot ret = logical_render_ring_init(dev); 16031b13d190SFrançois Tigeot if (ret) 16041b13d190SFrançois Tigeot return ret; 16051b13d190SFrançois Tigeot 16061b13d190SFrançois Tigeot if (HAS_BSD(dev)) { 16071b13d190SFrançois Tigeot ret = logical_bsd_ring_init(dev); 16081b13d190SFrançois Tigeot if (ret) 16091b13d190SFrançois Tigeot goto cleanup_render_ring; 16101b13d190SFrançois Tigeot } 16111b13d190SFrançois Tigeot 16121b13d190SFrançois Tigeot if (HAS_BLT(dev)) { 16131b13d190SFrançois Tigeot ret = logical_blt_ring_init(dev); 16141b13d190SFrançois Tigeot if (ret) 16151b13d190SFrançois Tigeot goto cleanup_bsd_ring; 16161b13d190SFrançois Tigeot } 16171b13d190SFrançois Tigeot 16181b13d190SFrançois Tigeot if (HAS_VEBOX(dev)) { 16191b13d190SFrançois Tigeot ret = logical_vebox_ring_init(dev); 16201b13d190SFrançois Tigeot if (ret) 16211b13d190SFrançois Tigeot goto cleanup_blt_ring; 16221b13d190SFrançois Tigeot } 16231b13d190SFrançois Tigeot 16241b13d190SFrançois Tigeot if (HAS_BSD2(dev)) { 16251b13d190SFrançois Tigeot ret = logical_bsd2_ring_init(dev); 16261b13d190SFrançois Tigeot if (ret) 16271b13d190SFrançois Tigeot goto cleanup_vebox_ring; 16281b13d190SFrançois Tigeot } 16291b13d190SFrançois Tigeot 16301b13d190SFrançois Tigeot ret = i915_gem_set_seqno(dev, ((u32)~0 - 0x1000)); 16311b13d190SFrançois Tigeot if (ret) 16321b13d190SFrançois Tigeot goto cleanup_bsd2_ring; 16331b13d190SFrançois Tigeot 16341b13d190SFrançois Tigeot return 0; 16351b13d190SFrançois Tigeot 16361b13d190SFrançois Tigeot cleanup_bsd2_ring: 16371b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[VCS2]); 16381b13d190SFrançois Tigeot cleanup_vebox_ring: 16391b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[VECS]); 16401b13d190SFrançois Tigeot cleanup_blt_ring: 16411b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[BCS]); 16421b13d190SFrançois Tigeot cleanup_bsd_ring: 16431b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[VCS]); 16441b13d190SFrançois Tigeot cleanup_render_ring: 16451b13d190SFrançois Tigeot intel_logical_ring_cleanup(&dev_priv->ring[RCS]); 16461b13d190SFrançois Tigeot 16471b13d190SFrançois Tigeot return ret; 16481b13d190SFrançois Tigeot } 16491b13d190SFrançois Tigeot 1650477eb7f9SFrançois Tigeot static u32 1651477eb7f9SFrançois Tigeot make_rpcs(struct drm_device *dev) 16521b13d190SFrançois Tigeot { 1653477eb7f9SFrançois Tigeot u32 rpcs = 0; 16541b13d190SFrançois Tigeot 1655477eb7f9SFrançois Tigeot /* 1656477eb7f9SFrançois Tigeot * No explicit RPCS request is needed to ensure full 1657477eb7f9SFrançois Tigeot * slice/subslice/EU enablement prior to Gen9. 1658477eb7f9SFrançois Tigeot */ 1659477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->gen < 9) 16601b13d190SFrançois Tigeot return 0; 16611b13d190SFrançois Tigeot 1662477eb7f9SFrançois Tigeot /* 1663477eb7f9SFrançois Tigeot * Starting in Gen9, render power gating can leave 1664477eb7f9SFrançois Tigeot * slice/subslice/EU in a partially enabled state. We 1665477eb7f9SFrançois Tigeot * must make an explicit request through RPCS for full 1666477eb7f9SFrançois Tigeot * enablement. 1667477eb7f9SFrançois Tigeot */ 1668477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_slice_pg) { 1669477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_S_CNT_ENABLE; 1670477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->slice_total << 1671477eb7f9SFrançois Tigeot GEN8_RPCS_S_CNT_SHIFT; 1672477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 1673477eb7f9SFrançois Tigeot } 16741b13d190SFrançois Tigeot 1675477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_subslice_pg) { 1676477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_SS_CNT_ENABLE; 1677477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->subslice_per_slice << 1678477eb7f9SFrançois Tigeot GEN8_RPCS_SS_CNT_SHIFT; 1679477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 1680477eb7f9SFrançois Tigeot } 16811b13d190SFrançois Tigeot 1682477eb7f9SFrançois Tigeot if (INTEL_INFO(dev)->has_eu_pg) { 1683477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->eu_per_subslice << 1684477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MIN_SHIFT; 1685477eb7f9SFrançois Tigeot rpcs |= INTEL_INFO(dev)->eu_per_subslice << 1686477eb7f9SFrançois Tigeot GEN8_RPCS_EU_MAX_SHIFT; 1687477eb7f9SFrançois Tigeot rpcs |= GEN8_RPCS_ENABLE; 1688477eb7f9SFrançois Tigeot } 1689477eb7f9SFrançois Tigeot 1690477eb7f9SFrançois Tigeot return rpcs; 16911b13d190SFrançois Tigeot } 16921b13d190SFrançois Tigeot 16931b13d190SFrançois Tigeot static int 16941b13d190SFrançois Tigeot populate_lr_context(struct intel_context *ctx, struct drm_i915_gem_object *ctx_obj, 16951b13d190SFrançois Tigeot struct intel_engine_cs *ring, struct intel_ringbuffer *ringbuf) 16961b13d190SFrançois Tigeot { 16971b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 16981b13d190SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 16991b13d190SFrançois Tigeot struct i915_hw_ppgtt *ppgtt = ctx->ppgtt; 17001b13d190SFrançois Tigeot struct vm_page *page; 17011b13d190SFrançois Tigeot uint32_t *reg_state; 17021b13d190SFrançois Tigeot int ret; 17031b13d190SFrançois Tigeot 17041b13d190SFrançois Tigeot if (!ppgtt) 17051b13d190SFrançois Tigeot ppgtt = dev_priv->mm.aliasing_ppgtt; 17061b13d190SFrançois Tigeot 17071b13d190SFrançois Tigeot ret = i915_gem_object_set_to_cpu_domain(ctx_obj, true); 17081b13d190SFrançois Tigeot if (ret) { 17091b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Could not set to CPU domain\n"); 17101b13d190SFrançois Tigeot return ret; 17111b13d190SFrançois Tigeot } 17121b13d190SFrançois Tigeot 17131b13d190SFrançois Tigeot ret = i915_gem_object_get_pages(ctx_obj); 17141b13d190SFrançois Tigeot if (ret) { 17151b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Could not get object pages\n"); 17161b13d190SFrançois Tigeot return ret; 17171b13d190SFrançois Tigeot } 17181b13d190SFrançois Tigeot 17191b13d190SFrançois Tigeot i915_gem_object_pin_pages(ctx_obj); 17201b13d190SFrançois Tigeot 17211b13d190SFrançois Tigeot /* The second page of the context object contains some fields which must 17221b13d190SFrançois Tigeot * be set up prior to the first execution. */ 17231b13d190SFrançois Tigeot page = i915_gem_object_get_page(ctx_obj, 1); 17241b13d190SFrançois Tigeot reg_state = kmap_atomic(page); 17251b13d190SFrançois Tigeot 17261b13d190SFrançois Tigeot /* A context is actually a big batch buffer with several MI_LOAD_REGISTER_IMM 17271b13d190SFrançois Tigeot * commands followed by (reg, value) pairs. The values we are setting here are 17281b13d190SFrançois Tigeot * only for the first context restore: on a subsequent save, the GPU will 17291b13d190SFrançois Tigeot * recreate this batchbuffer with new values (including all the missing 17301b13d190SFrançois Tigeot * MI_LOAD_REGISTER_IMM commands that we are not initializing here). */ 17311b13d190SFrançois Tigeot if (ring->id == RCS) 17321b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_0] = MI_LOAD_REGISTER_IMM(14); 17331b13d190SFrançois Tigeot else 17341b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_0] = MI_LOAD_REGISTER_IMM(11); 17351b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_0] |= MI_LRI_FORCE_POSTED; 17361b13d190SFrançois Tigeot reg_state[CTX_CONTEXT_CONTROL] = RING_CONTEXT_CONTROL(ring); 17371b13d190SFrançois Tigeot reg_state[CTX_CONTEXT_CONTROL+1] = 1738477eb7f9SFrançois Tigeot _MASKED_BIT_ENABLE(CTX_CTRL_INHIBIT_SYN_CTX_SWITCH | 1739477eb7f9SFrançois Tigeot CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT); 17401b13d190SFrançois Tigeot reg_state[CTX_RING_HEAD] = RING_HEAD(ring->mmio_base); 17411b13d190SFrançois Tigeot reg_state[CTX_RING_HEAD+1] = 0; 17421b13d190SFrançois Tigeot reg_state[CTX_RING_TAIL] = RING_TAIL(ring->mmio_base); 17431b13d190SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = 0; 17441b13d190SFrançois Tigeot reg_state[CTX_RING_BUFFER_START] = RING_START(ring->mmio_base); 17452c9916cdSFrançois Tigeot /* Ring buffer start address is not known until the buffer is pinned. 17462c9916cdSFrançois Tigeot * It is written to the context image in execlists_update_context() 17472c9916cdSFrançois Tigeot */ 17481b13d190SFrançois Tigeot reg_state[CTX_RING_BUFFER_CONTROL] = RING_CTL(ring->mmio_base); 17491b13d190SFrançois Tigeot reg_state[CTX_RING_BUFFER_CONTROL+1] = 17501b13d190SFrançois Tigeot ((ringbuf->size - PAGE_SIZE) & RING_NR_PAGES) | RING_VALID; 17511b13d190SFrançois Tigeot reg_state[CTX_BB_HEAD_U] = ring->mmio_base + 0x168; 17521b13d190SFrançois Tigeot reg_state[CTX_BB_HEAD_U+1] = 0; 17531b13d190SFrançois Tigeot reg_state[CTX_BB_HEAD_L] = ring->mmio_base + 0x140; 17541b13d190SFrançois Tigeot reg_state[CTX_BB_HEAD_L+1] = 0; 17551b13d190SFrançois Tigeot reg_state[CTX_BB_STATE] = ring->mmio_base + 0x110; 17561b13d190SFrançois Tigeot reg_state[CTX_BB_STATE+1] = (1<<5); 17571b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_HEAD_U] = ring->mmio_base + 0x11c; 17581b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_HEAD_U+1] = 0; 17591b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_HEAD_L] = ring->mmio_base + 0x114; 17601b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_HEAD_L+1] = 0; 17611b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_STATE] = ring->mmio_base + 0x118; 17621b13d190SFrançois Tigeot reg_state[CTX_SECOND_BB_STATE+1] = 0; 17631b13d190SFrançois Tigeot if (ring->id == RCS) { 17641b13d190SFrançois Tigeot /* TODO: according to BSpec, the register state context 17651b13d190SFrançois Tigeot * for CHV does not have these. OTOH, these registers do 17661b13d190SFrançois Tigeot * exist in CHV. I'm waiting for a clarification */ 17671b13d190SFrançois Tigeot reg_state[CTX_BB_PER_CTX_PTR] = ring->mmio_base + 0x1c0; 17681b13d190SFrançois Tigeot reg_state[CTX_BB_PER_CTX_PTR+1] = 0; 17691b13d190SFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX] = ring->mmio_base + 0x1c4; 17701b13d190SFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX+1] = 0; 17711b13d190SFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX_OFFSET] = ring->mmio_base + 0x1c8; 17721b13d190SFrançois Tigeot reg_state[CTX_RCS_INDIRECT_CTX_OFFSET+1] = 0; 17731b13d190SFrançois Tigeot } 17741b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_1] = MI_LOAD_REGISTER_IMM(9); 17751b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_1] |= MI_LRI_FORCE_POSTED; 17761b13d190SFrançois Tigeot reg_state[CTX_CTX_TIMESTAMP] = ring->mmio_base + 0x3a8; 17771b13d190SFrançois Tigeot reg_state[CTX_CTX_TIMESTAMP+1] = 0; 17781b13d190SFrançois Tigeot reg_state[CTX_PDP3_UDW] = GEN8_RING_PDP_UDW(ring, 3); 17791b13d190SFrançois Tigeot reg_state[CTX_PDP3_LDW] = GEN8_RING_PDP_LDW(ring, 3); 17801b13d190SFrançois Tigeot reg_state[CTX_PDP2_UDW] = GEN8_RING_PDP_UDW(ring, 2); 17811b13d190SFrançois Tigeot reg_state[CTX_PDP2_LDW] = GEN8_RING_PDP_LDW(ring, 2); 17821b13d190SFrançois Tigeot reg_state[CTX_PDP1_UDW] = GEN8_RING_PDP_UDW(ring, 1); 17831b13d190SFrançois Tigeot reg_state[CTX_PDP1_LDW] = GEN8_RING_PDP_LDW(ring, 1); 17841b13d190SFrançois Tigeot reg_state[CTX_PDP0_UDW] = GEN8_RING_PDP_UDW(ring, 0); 17851b13d190SFrançois Tigeot reg_state[CTX_PDP0_LDW] = GEN8_RING_PDP_LDW(ring, 0); 1786*19c468b4SFrançois Tigeot 1787*19c468b4SFrançois Tigeot /* With dynamic page allocation, PDPs may not be allocated at this point, 1788*19c468b4SFrançois Tigeot * Point the unallocated PDPs to the scratch page 1789*19c468b4SFrançois Tigeot */ 1790*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 3); 1791*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 2); 1792*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 1); 1793*19c468b4SFrançois Tigeot ASSIGN_CTX_PDP(ppgtt, reg_state, 0); 17941b13d190SFrançois Tigeot if (ring->id == RCS) { 17951b13d190SFrançois Tigeot reg_state[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1); 1796477eb7f9SFrançois Tigeot reg_state[CTX_R_PWR_CLK_STATE] = GEN8_R_PWR_CLK_STATE; 1797477eb7f9SFrançois Tigeot reg_state[CTX_R_PWR_CLK_STATE+1] = make_rpcs(dev); 17981b13d190SFrançois Tigeot } 17991b13d190SFrançois Tigeot 18001b13d190SFrançois Tigeot kunmap_atomic(reg_state); 18011b13d190SFrançois Tigeot 18021b13d190SFrançois Tigeot ctx_obj->dirty = 1; 18031b13d190SFrançois Tigeot set_page_dirty(page); 18041b13d190SFrançois Tigeot i915_gem_object_unpin_pages(ctx_obj); 18051b13d190SFrançois Tigeot 18061b13d190SFrançois Tigeot return 0; 18071b13d190SFrançois Tigeot } 18081b13d190SFrançois Tigeot 18091b13d190SFrançois Tigeot /** 18101b13d190SFrançois Tigeot * intel_lr_context_free() - free the LRC specific bits of a context 18111b13d190SFrançois Tigeot * @ctx: the LR context to free. 18121b13d190SFrançois Tigeot * 18131b13d190SFrançois Tigeot * The real context freeing is done in i915_gem_context_free: this only 18141b13d190SFrançois Tigeot * takes care of the bits that are LRC related: the per-engine backing 18151b13d190SFrançois Tigeot * objects and the logical ringbuffer. 18161b13d190SFrançois Tigeot */ 18171b13d190SFrançois Tigeot void intel_lr_context_free(struct intel_context *ctx) 18181b13d190SFrançois Tigeot { 18191b13d190SFrançois Tigeot int i; 18201b13d190SFrançois Tigeot 18211b13d190SFrançois Tigeot for (i = 0; i < I915_NUM_RINGS; i++) { 18221b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = ctx->engine[i].state; 18231b13d190SFrançois Tigeot 18241b13d190SFrançois Tigeot if (ctx_obj) { 18252c9916cdSFrançois Tigeot struct intel_ringbuffer *ringbuf = 18262c9916cdSFrançois Tigeot ctx->engine[i].ringbuf; 18272c9916cdSFrançois Tigeot struct intel_engine_cs *ring = ringbuf->ring; 18282c9916cdSFrançois Tigeot 18292c9916cdSFrançois Tigeot if (ctx == ring->default_context) { 18302c9916cdSFrançois Tigeot intel_unpin_ringbuffer_obj(ringbuf); 18312c9916cdSFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 18322c9916cdSFrançois Tigeot } 18332c9916cdSFrançois Tigeot WARN_ON(ctx->engine[ring->id].pin_count); 18341b13d190SFrançois Tigeot intel_destroy_ringbuffer_obj(ringbuf); 18351b13d190SFrançois Tigeot kfree(ringbuf); 18361b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 18371b13d190SFrançois Tigeot } 18381b13d190SFrançois Tigeot } 18391b13d190SFrançois Tigeot } 18401b13d190SFrançois Tigeot 18411b13d190SFrançois Tigeot static uint32_t get_lr_context_size(struct intel_engine_cs *ring) 18421b13d190SFrançois Tigeot { 18431b13d190SFrançois Tigeot int ret = 0; 18441b13d190SFrançois Tigeot 18452c9916cdSFrançois Tigeot WARN_ON(INTEL_INFO(ring->dev)->gen < 8); 18461b13d190SFrançois Tigeot 18471b13d190SFrançois Tigeot switch (ring->id) { 18481b13d190SFrançois Tigeot case RCS: 18492c9916cdSFrançois Tigeot if (INTEL_INFO(ring->dev)->gen >= 9) 18502c9916cdSFrançois Tigeot ret = GEN9_LR_CONTEXT_RENDER_SIZE; 18512c9916cdSFrançois Tigeot else 18521b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_RENDER_SIZE; 18531b13d190SFrançois Tigeot break; 18541b13d190SFrançois Tigeot case VCS: 18551b13d190SFrançois Tigeot case BCS: 18561b13d190SFrançois Tigeot case VECS: 18571b13d190SFrançois Tigeot case VCS2: 18581b13d190SFrançois Tigeot ret = GEN8_LR_CONTEXT_OTHER_SIZE; 18591b13d190SFrançois Tigeot break; 18601b13d190SFrançois Tigeot } 18611b13d190SFrançois Tigeot 18621b13d190SFrançois Tigeot return ret; 18631b13d190SFrançois Tigeot } 18641b13d190SFrançois Tigeot 18652c9916cdSFrançois Tigeot static void lrc_setup_hardware_status_page(struct intel_engine_cs *ring, 18662c9916cdSFrançois Tigeot struct drm_i915_gem_object *default_ctx_obj) 18672c9916cdSFrançois Tigeot { 18682c9916cdSFrançois Tigeot struct drm_i915_private *dev_priv = ring->dev->dev_private; 18692c9916cdSFrançois Tigeot 18702c9916cdSFrançois Tigeot /* The status page is offset 0 from the default context object 18712c9916cdSFrançois Tigeot * in LRC mode. */ 18722c9916cdSFrançois Tigeot ring->status_page.gfx_addr = i915_gem_obj_ggtt_offset(default_ctx_obj); 18732c9916cdSFrançois Tigeot ring->status_page.page_addr = 18742c9916cdSFrançois Tigeot kmap(default_ctx_obj->pages[0]); 18752c9916cdSFrançois Tigeot ring->status_page.obj = default_ctx_obj; 18762c9916cdSFrançois Tigeot 18772c9916cdSFrançois Tigeot I915_WRITE(RING_HWS_PGA(ring->mmio_base), 18782c9916cdSFrançois Tigeot (u32)ring->status_page.gfx_addr); 18792c9916cdSFrançois Tigeot POSTING_READ(RING_HWS_PGA(ring->mmio_base)); 18802c9916cdSFrançois Tigeot } 18812c9916cdSFrançois Tigeot 18821b13d190SFrançois Tigeot /** 18831b13d190SFrançois Tigeot * intel_lr_context_deferred_create() - create the LRC specific bits of a context 18841b13d190SFrançois Tigeot * @ctx: LR context to create. 18851b13d190SFrançois Tigeot * @ring: engine to be used with the context. 18861b13d190SFrançois Tigeot * 18871b13d190SFrançois Tigeot * This function can be called more than once, with different engines, if we plan 18881b13d190SFrançois Tigeot * to use the context with them. The context backing objects and the ringbuffers 18891b13d190SFrançois Tigeot * (specially the ringbuffer backing objects) suck a lot of memory up, and that's why 18901b13d190SFrançois Tigeot * the creation is a deferred call: it's better to make sure first that we need to use 18911b13d190SFrançois Tigeot * a given ring with the context. 18921b13d190SFrançois Tigeot * 18932c9916cdSFrançois Tigeot * Return: non-zero on error. 18941b13d190SFrançois Tigeot */ 18951b13d190SFrançois Tigeot int intel_lr_context_deferred_create(struct intel_context *ctx, 18961b13d190SFrançois Tigeot struct intel_engine_cs *ring) 18971b13d190SFrançois Tigeot { 18982c9916cdSFrançois Tigeot const bool is_global_default_ctx = (ctx == ring->default_context); 18991b13d190SFrançois Tigeot struct drm_device *dev = ring->dev; 19001b13d190SFrançois Tigeot struct drm_i915_gem_object *ctx_obj; 19011b13d190SFrançois Tigeot uint32_t context_size; 19021b13d190SFrançois Tigeot struct intel_ringbuffer *ringbuf; 19031b13d190SFrançois Tigeot int ret; 19041b13d190SFrançois Tigeot 19051b13d190SFrançois Tigeot WARN_ON(ctx->legacy_hw_ctx.rcs_state != NULL); 19062c9916cdSFrançois Tigeot WARN_ON(ctx->engine[ring->id].state); 19071b13d190SFrançois Tigeot 19081b13d190SFrançois Tigeot context_size = round_up(get_lr_context_size(ring), 4096); 19091b13d190SFrançois Tigeot 1910*19c468b4SFrançois Tigeot ctx_obj = i915_gem_alloc_object(dev, context_size); 1911*19c468b4SFrançois Tigeot if (!ctx_obj) { 1912*19c468b4SFrançois Tigeot DRM_DEBUG_DRIVER("Alloc LRC backing obj failed.\n"); 1913*19c468b4SFrançois Tigeot return -ENOMEM; 19141b13d190SFrançois Tigeot } 19151b13d190SFrançois Tigeot 19162c9916cdSFrançois Tigeot if (is_global_default_ctx) { 19171b13d190SFrançois Tigeot ret = i915_gem_obj_ggtt_pin(ctx_obj, GEN8_LR_CONTEXT_ALIGN, 0); 19181b13d190SFrançois Tigeot if (ret) { 19192c9916cdSFrançois Tigeot DRM_DEBUG_DRIVER("Pin LRC backing obj failed: %d\n", 19202c9916cdSFrançois Tigeot ret); 19211b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 19221b13d190SFrançois Tigeot return ret; 19231b13d190SFrançois Tigeot } 19242c9916cdSFrançois Tigeot } 19251b13d190SFrançois Tigeot 19261b13d190SFrançois Tigeot ringbuf = kzalloc(sizeof(*ringbuf), GFP_KERNEL); 19271b13d190SFrançois Tigeot if (!ringbuf) { 19281b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Failed to allocate ringbuffer %s\n", 19291b13d190SFrançois Tigeot ring->name); 19301b13d190SFrançois Tigeot ret = -ENOMEM; 19312c9916cdSFrançois Tigeot goto error_unpin_ctx; 19321b13d190SFrançois Tigeot } 19331b13d190SFrançois Tigeot 19341b13d190SFrançois Tigeot ringbuf->ring = ring; 19351b13d190SFrançois Tigeot 19361b13d190SFrançois Tigeot ringbuf->size = 32 * PAGE_SIZE; 19371b13d190SFrançois Tigeot ringbuf->effective_size = ringbuf->size; 19381b13d190SFrançois Tigeot ringbuf->head = 0; 19391b13d190SFrançois Tigeot ringbuf->tail = 0; 19401b13d190SFrançois Tigeot ringbuf->last_retired_head = -1; 19412c9916cdSFrançois Tigeot intel_ring_update_space(ringbuf); 19421b13d190SFrançois Tigeot 19432c9916cdSFrançois Tigeot if (ringbuf->obj == NULL) { 19441b13d190SFrançois Tigeot ret = intel_alloc_ringbuffer_obj(dev, ringbuf); 19451b13d190SFrançois Tigeot if (ret) { 19462c9916cdSFrançois Tigeot DRM_DEBUG_DRIVER( 19472c9916cdSFrançois Tigeot "Failed to allocate ringbuffer obj %s: %d\n", 19481b13d190SFrançois Tigeot ring->name, ret); 19492c9916cdSFrançois Tigeot goto error_free_rbuf; 19502c9916cdSFrançois Tigeot } 19512c9916cdSFrançois Tigeot 19522c9916cdSFrançois Tigeot if (is_global_default_ctx) { 19532c9916cdSFrançois Tigeot ret = intel_pin_and_map_ringbuffer_obj(dev, ringbuf); 19542c9916cdSFrançois Tigeot if (ret) { 19552c9916cdSFrançois Tigeot DRM_ERROR( 19562c9916cdSFrançois Tigeot "Failed to pin and map ringbuffer %s: %d\n", 19572c9916cdSFrançois Tigeot ring->name, ret); 19582c9916cdSFrançois Tigeot goto error_destroy_rbuf; 19592c9916cdSFrançois Tigeot } 19602c9916cdSFrançois Tigeot } 19612c9916cdSFrançois Tigeot 19621b13d190SFrançois Tigeot } 19631b13d190SFrançois Tigeot 19641b13d190SFrançois Tigeot ret = populate_lr_context(ctx, ctx_obj, ring, ringbuf); 19651b13d190SFrançois Tigeot if (ret) { 19661b13d190SFrançois Tigeot DRM_DEBUG_DRIVER("Failed to populate LRC: %d\n", ret); 19671b13d190SFrançois Tigeot goto error; 19681b13d190SFrançois Tigeot } 19691b13d190SFrançois Tigeot 19701b13d190SFrançois Tigeot ctx->engine[ring->id].ringbuf = ringbuf; 19711b13d190SFrançois Tigeot ctx->engine[ring->id].state = ctx_obj; 19721b13d190SFrançois Tigeot 19732c9916cdSFrançois Tigeot if (ctx == ring->default_context) 19742c9916cdSFrançois Tigeot lrc_setup_hardware_status_page(ring, ctx_obj); 19752c9916cdSFrançois Tigeot else if (ring->id == RCS && !ctx->rcs_initialized) { 19762c9916cdSFrançois Tigeot if (ring->init_context) { 19772c9916cdSFrançois Tigeot ret = ring->init_context(ring, ctx); 19781b13d190SFrançois Tigeot if (ret) { 19792c9916cdSFrançois Tigeot DRM_ERROR("ring init context: %d\n", ret); 19801b13d190SFrançois Tigeot ctx->engine[ring->id].ringbuf = NULL; 19811b13d190SFrançois Tigeot ctx->engine[ring->id].state = NULL; 19821b13d190SFrançois Tigeot goto error; 19831b13d190SFrançois Tigeot } 19842c9916cdSFrançois Tigeot } 19852c9916cdSFrançois Tigeot 19861b13d190SFrançois Tigeot ctx->rcs_initialized = true; 19871b13d190SFrançois Tigeot } 19881b13d190SFrançois Tigeot 19891b13d190SFrançois Tigeot return 0; 19901b13d190SFrançois Tigeot 19911b13d190SFrançois Tigeot error: 19922c9916cdSFrançois Tigeot if (is_global_default_ctx) 19932c9916cdSFrançois Tigeot intel_unpin_ringbuffer_obj(ringbuf); 19942c9916cdSFrançois Tigeot error_destroy_rbuf: 19952c9916cdSFrançois Tigeot intel_destroy_ringbuffer_obj(ringbuf); 19962c9916cdSFrançois Tigeot error_free_rbuf: 19971b13d190SFrançois Tigeot kfree(ringbuf); 19982c9916cdSFrançois Tigeot error_unpin_ctx: 19992c9916cdSFrançois Tigeot if (is_global_default_ctx) 20001b13d190SFrançois Tigeot i915_gem_object_ggtt_unpin(ctx_obj); 20011b13d190SFrançois Tigeot drm_gem_object_unreference(&ctx_obj->base); 20021b13d190SFrançois Tigeot return ret; 20031b13d190SFrançois Tigeot } 2004477eb7f9SFrançois Tigeot 2005477eb7f9SFrançois Tigeot void intel_lr_context_reset(struct drm_device *dev, 2006477eb7f9SFrançois Tigeot struct intel_context *ctx) 2007477eb7f9SFrançois Tigeot { 2008477eb7f9SFrançois Tigeot struct drm_i915_private *dev_priv = dev->dev_private; 2009477eb7f9SFrançois Tigeot struct intel_engine_cs *ring; 2010477eb7f9SFrançois Tigeot int i; 2011477eb7f9SFrançois Tigeot 2012477eb7f9SFrançois Tigeot for_each_ring(ring, dev_priv, i) { 2013477eb7f9SFrançois Tigeot struct drm_i915_gem_object *ctx_obj = 2014477eb7f9SFrançois Tigeot ctx->engine[ring->id].state; 2015477eb7f9SFrançois Tigeot struct intel_ringbuffer *ringbuf = 2016477eb7f9SFrançois Tigeot ctx->engine[ring->id].ringbuf; 2017477eb7f9SFrançois Tigeot uint32_t *reg_state; 2018477eb7f9SFrançois Tigeot struct vm_page *page; 2019477eb7f9SFrançois Tigeot 2020477eb7f9SFrançois Tigeot if (!ctx_obj) 2021477eb7f9SFrançois Tigeot continue; 2022477eb7f9SFrançois Tigeot 2023477eb7f9SFrançois Tigeot if (i915_gem_object_get_pages(ctx_obj)) { 2024477eb7f9SFrançois Tigeot WARN(1, "Failed get_pages for context obj\n"); 2025477eb7f9SFrançois Tigeot continue; 2026477eb7f9SFrançois Tigeot } 2027477eb7f9SFrançois Tigeot page = i915_gem_object_get_page(ctx_obj, 1); 2028477eb7f9SFrançois Tigeot reg_state = kmap_atomic(page); 2029477eb7f9SFrançois Tigeot 2030477eb7f9SFrançois Tigeot reg_state[CTX_RING_HEAD+1] = 0; 2031477eb7f9SFrançois Tigeot reg_state[CTX_RING_TAIL+1] = 0; 2032477eb7f9SFrançois Tigeot 2033477eb7f9SFrançois Tigeot kunmap_atomic(reg_state); 2034477eb7f9SFrançois Tigeot 2035477eb7f9SFrançois Tigeot ringbuf->head = 0; 2036477eb7f9SFrançois Tigeot ringbuf->tail = 0; 2037477eb7f9SFrançois Tigeot } 2038477eb7f9SFrançois Tigeot } 2039