1 //===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2 //
3 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4 // See https://llvm.org/LICENSE.txt for license information.
5 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6 //
7 //===----------------------------------------------------------------------===//
8 //
9 /// \file
10 /// This file is a part of MemorySanitizer, a detector of uninitialized
11 /// reads.
12 ///
13 /// The algorithm of the tool is similar to Memcheck
14 /// (http://goo.gl/QKbem). We associate a few shadow bits with every
15 /// byte of the application memory, poison the shadow of the malloc-ed
16 /// or alloca-ed memory, load the shadow bits on every memory read,
17 /// propagate the shadow bits through some of the arithmetic
18 /// instruction (including MOV), store the shadow bits on every memory
19 /// write, report a bug on some other instructions (e.g. JMP) if the
20 /// associated shadow is poisoned.
21 ///
22 /// But there are differences too. The first and the major one:
23 /// compiler instrumentation instead of binary instrumentation. This
24 /// gives us much better register allocation, possible compiler
25 /// optimizations and a fast start-up. But this brings the major issue
26 /// as well: msan needs to see all program events, including system
27 /// calls and reads/writes in system libraries, so we either need to
28 /// compile *everything* with msan or use a binary translation
29 /// component (e.g. DynamoRIO) to instrument pre-built libraries.
30 /// Another difference from Memcheck is that we use 8 shadow bits per
31 /// byte of application memory and use a direct shadow mapping. This
32 /// greatly simplifies the instrumentation code and avoids races on
33 /// shadow updates (Memcheck is single-threaded so races are not a
34 /// concern there. Memcheck uses 2 shadow bits per byte with a slow
35 /// path storage that uses 8 bits per byte).
36 ///
37 /// The default value of shadow is 0, which means "clean" (not poisoned).
38 ///
39 /// Every module initializer should call __msan_init to ensure that the
40 /// shadow memory is ready. On error, __msan_warning is called. Since
41 /// parameters and return values may be passed via registers, we have a
42 /// specialized thread-local shadow for return values
43 /// (__msan_retval_tls) and parameters (__msan_param_tls).
44 ///
45 /// Origin tracking.
46 ///
47 /// MemorySanitizer can track origins (allocation points) of all uninitialized
48 /// values. This behavior is controlled with a flag (msan-track-origins) and is
49 /// disabled by default.
50 ///
51 /// Origins are 4-byte values created and interpreted by the runtime library.
52 /// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53 /// of application memory. Propagation of origins is basically a bunch of
54 /// "select" instructions that pick the origin of a dirty argument, if an
55 /// instruction has one.
56 ///
57 /// Every 4 aligned, consecutive bytes of application memory have one origin
58 /// value associated with them. If these bytes contain uninitialized data
59 /// coming from 2 different allocations, the last store wins. Because of this,
60 /// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61 /// practice.
62 ///
63 /// Origins are meaningless for fully initialized values, so MemorySanitizer
64 /// avoids storing origin to memory when a fully initialized value is stored.
65 /// This way it avoids needless overwriting origin of the 4-byte region on
66 /// a short (i.e. 1 byte) clean store, and it is also good for performance.
67 ///
68 /// Atomic handling.
69 ///
70 /// Ideally, every atomic store of application value should update the
71 /// corresponding shadow location in an atomic way. Unfortunately, atomic store
72 /// of two disjoint locations can not be done without severe slowdown.
73 ///
74 /// Therefore, we implement an approximation that may err on the safe side.
75 /// In this implementation, every atomically accessed location in the program
76 /// may only change from (partially) uninitialized to fully initialized, but
77 /// not the other way around. We load the shadow _after_ the application load,
78 /// and we store the shadow _before_ the app store. Also, we always store clean
79 /// shadow (if the application store is atomic). This way, if the store-load
80 /// pair constitutes a happens-before arc, shadow store and load are correctly
81 /// ordered such that the load will get either the value that was stored, or
82 /// some later value (which is always clean).
83 ///
84 /// This does not work very well with Compare-And-Swap (CAS) and
85 /// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86 /// must store the new shadow before the app operation, and load the shadow
87 /// after the app operation. Computers don't work this way. Current
88 /// implementation ignores the load aspect of CAS/RMW, always returning a clean
89 /// value. It implements the store part as a simple atomic store by storing a
90 /// clean shadow.
91 ///
92 /// Instrumenting inline assembly.
93 ///
94 /// For inline assembly code LLVM has little idea about which memory locations
95 /// become initialized depending on the arguments. It can be possible to figure
96 /// out which arguments are meant to point to inputs and outputs, but the
97 /// actual semantics can be only visible at runtime. In the Linux kernel it's
98 /// also possible that the arguments only indicate the offset for a base taken
99 /// from a segment register, so it's dangerous to treat any asm() arguments as
100 /// pointers. We take a conservative approach generating calls to
101 /// __msan_instrument_asm_store(ptr, size)
102 /// , which defer the memory unpoisoning to the runtime library.
103 /// The latter can perform more complex address checks to figure out whether
104 /// it's safe to touch the shadow memory.
105 /// Like with atomic operations, we call __msan_instrument_asm_store() before
106 /// the assembly call, so that changes to the shadow memory will be seen by
107 /// other threads together with main memory initialization.
108 ///
109 /// KernelMemorySanitizer (KMSAN) implementation.
110 ///
111 /// The major differences between KMSAN and MSan instrumentation are:
112 /// - KMSAN always tracks the origins and implies msan-keep-going=true;
113 /// - KMSAN allocates shadow and origin memory for each page separately, so
114 /// there are no explicit accesses to shadow and origin in the
115 /// instrumentation.
116 /// Shadow and origin values for a particular X-byte memory location
117 /// (X=1,2,4,8) are accessed through pointers obtained via the
118 /// __msan_metadata_ptr_for_load_X(ptr)
119 /// __msan_metadata_ptr_for_store_X(ptr)
120 /// functions. The corresponding functions check that the X-byte accesses
121 /// are possible and returns the pointers to shadow and origin memory.
122 /// Arbitrary sized accesses are handled with:
123 /// __msan_metadata_ptr_for_load_n(ptr, size)
124 /// __msan_metadata_ptr_for_store_n(ptr, size);
125 /// - TLS variables are stored in a single per-task struct. A call to a
126 /// function __msan_get_context_state() returning a pointer to that struct
127 /// is inserted into every instrumented function before the entry block;
128 /// - __msan_warning() takes a 32-bit origin parameter;
129 /// - local variables are poisoned with __msan_poison_alloca() upon function
130 /// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
131 /// function;
132 /// - the pass doesn't declare any global variables or add global constructors
133 /// to the translation unit.
134 ///
135 /// Also, KMSAN currently ignores uninitialized memory passed into inline asm
136 /// calls, making sure we're on the safe side wrt. possible false positives.
137 ///
138 /// KernelMemorySanitizer only supports X86_64 at the moment.
139 ///
140 //
141 // FIXME: This sanitizer does not yet handle scalable vectors
142 //
143 //===----------------------------------------------------------------------===//
144
145 #include "llvm/Transforms/Instrumentation/MemorySanitizer.h"
146 #include "llvm/ADT/APInt.h"
147 #include "llvm/ADT/ArrayRef.h"
148 #include "llvm/ADT/DepthFirstIterator.h"
149 #include "llvm/ADT/SmallSet.h"
150 #include "llvm/ADT/SmallString.h"
151 #include "llvm/ADT/SmallVector.h"
152 #include "llvm/ADT/StringExtras.h"
153 #include "llvm/ADT/StringRef.h"
154 #include "llvm/ADT/Triple.h"
155 #include "llvm/Analysis/TargetLibraryInfo.h"
156 #include "llvm/Analysis/ValueTracking.h"
157 #include "llvm/IR/Argument.h"
158 #include "llvm/IR/Attributes.h"
159 #include "llvm/IR/BasicBlock.h"
160 #include "llvm/IR/CallingConv.h"
161 #include "llvm/IR/Constant.h"
162 #include "llvm/IR/Constants.h"
163 #include "llvm/IR/DataLayout.h"
164 #include "llvm/IR/DerivedTypes.h"
165 #include "llvm/IR/Function.h"
166 #include "llvm/IR/GlobalValue.h"
167 #include "llvm/IR/GlobalVariable.h"
168 #include "llvm/IR/IRBuilder.h"
169 #include "llvm/IR/InlineAsm.h"
170 #include "llvm/IR/InstVisitor.h"
171 #include "llvm/IR/InstrTypes.h"
172 #include "llvm/IR/Instruction.h"
173 #include "llvm/IR/Instructions.h"
174 #include "llvm/IR/IntrinsicInst.h"
175 #include "llvm/IR/Intrinsics.h"
176 #include "llvm/IR/IntrinsicsX86.h"
177 #include "llvm/IR/LLVMContext.h"
178 #include "llvm/IR/MDBuilder.h"
179 #include "llvm/IR/Module.h"
180 #include "llvm/IR/Type.h"
181 #include "llvm/IR/Value.h"
182 #include "llvm/IR/ValueMap.h"
183 #include "llvm/InitializePasses.h"
184 #include "llvm/Pass.h"
185 #include "llvm/Support/AtomicOrdering.h"
186 #include "llvm/Support/Casting.h"
187 #include "llvm/Support/CommandLine.h"
188 #include "llvm/Support/Compiler.h"
189 #include "llvm/Support/Debug.h"
190 #include "llvm/Support/ErrorHandling.h"
191 #include "llvm/Support/MathExtras.h"
192 #include "llvm/Support/raw_ostream.h"
193 #include "llvm/Transforms/Instrumentation.h"
194 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
195 #include "llvm/Transforms/Utils/Local.h"
196 #include "llvm/Transforms/Utils/ModuleUtils.h"
197 #include <algorithm>
198 #include <cassert>
199 #include <cstddef>
200 #include <cstdint>
201 #include <memory>
202 #include <string>
203 #include <tuple>
204
205 using namespace llvm;
206
207 #define DEBUG_TYPE "msan"
208
209 static const unsigned kOriginSize = 4;
210 static const Align kMinOriginAlignment = Align(4);
211 static const Align kShadowTLSAlignment = Align(8);
212
213 // These constants must be kept in sync with the ones in msan.h.
214 static const unsigned kParamTLSSize = 800;
215 static const unsigned kRetvalTLSSize = 800;
216
217 // Accesses sizes are powers of two: 1, 2, 4, 8.
218 static const size_t kNumberOfAccessSizes = 4;
219
220 /// Track origins of uninitialized values.
221 ///
222 /// Adds a section to MemorySanitizer report that points to the allocation
223 /// (stack or heap) the uninitialized bits came from originally.
224 static cl::opt<int> ClTrackOrigins("msan-track-origins",
225 cl::desc("Track origins (allocation sites) of poisoned memory"),
226 cl::Hidden, cl::init(0));
227
228 static cl::opt<bool> ClKeepGoing("msan-keep-going",
229 cl::desc("keep going after reporting a UMR"),
230 cl::Hidden, cl::init(false));
231
232 static cl::opt<bool> ClPoisonStack("msan-poison-stack",
233 cl::desc("poison uninitialized stack variables"),
234 cl::Hidden, cl::init(true));
235
236 static cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call",
237 cl::desc("poison uninitialized stack variables with a call"),
238 cl::Hidden, cl::init(false));
239
240 static cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern",
241 cl::desc("poison uninitialized stack variables with the given pattern"),
242 cl::Hidden, cl::init(0xff));
243
244 static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
245 cl::desc("poison undef temps"),
246 cl::Hidden, cl::init(true));
247
248 static cl::opt<bool> ClHandleICmp("msan-handle-icmp",
249 cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
250 cl::Hidden, cl::init(true));
251
252 static cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact",
253 cl::desc("exact handling of relational integer ICmp"),
254 cl::Hidden, cl::init(false));
255
256 static cl::opt<bool> ClHandleLifetimeIntrinsics(
257 "msan-handle-lifetime-intrinsics",
258 cl::desc(
259 "when possible, poison scoped variables at the beginning of the scope "
260 "(slower, but more precise)"),
261 cl::Hidden, cl::init(true));
262
263 // When compiling the Linux kernel, we sometimes see false positives related to
264 // MSan being unable to understand that inline assembly calls may initialize
265 // local variables.
266 // This flag makes the compiler conservatively unpoison every memory location
267 // passed into an assembly call. Note that this may cause false positives.
268 // Because it's impossible to figure out the array sizes, we can only unpoison
269 // the first sizeof(type) bytes for each type* pointer.
270 // The instrumentation is only enabled in KMSAN builds, and only if
271 // -msan-handle-asm-conservative is on. This is done because we may want to
272 // quickly disable assembly instrumentation when it breaks.
273 static cl::opt<bool> ClHandleAsmConservative(
274 "msan-handle-asm-conservative",
275 cl::desc("conservative handling of inline assembly"), cl::Hidden,
276 cl::init(true));
277
278 // This flag controls whether we check the shadow of the address
279 // operand of load or store. Such bugs are very rare, since load from
280 // a garbage address typically results in SEGV, but still happen
281 // (e.g. only lower bits of address are garbage, or the access happens
282 // early at program startup where malloc-ed memory is more likely to
283 // be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
284 static cl::opt<bool> ClCheckAccessAddress("msan-check-access-address",
285 cl::desc("report accesses through a pointer which has poisoned shadow"),
286 cl::Hidden, cl::init(true));
287
288 static cl::opt<bool> ClEagerChecks(
289 "msan-eager-checks",
290 cl::desc("check arguments and return values at function call boundaries"),
291 cl::Hidden, cl::init(false));
292
293 static cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions",
294 cl::desc("print out instructions with default strict semantics"),
295 cl::Hidden, cl::init(false));
296
297 static cl::opt<int> ClInstrumentationWithCallThreshold(
298 "msan-instrumentation-with-call-threshold",
299 cl::desc(
300 "If the function being instrumented requires more than "
301 "this number of checks and origin stores, use callbacks instead of "
302 "inline checks (-1 means never use callbacks)."),
303 cl::Hidden, cl::init(3500));
304
305 static cl::opt<bool>
306 ClEnableKmsan("msan-kernel",
307 cl::desc("Enable KernelMemorySanitizer instrumentation"),
308 cl::Hidden, cl::init(false));
309
310 // This is an experiment to enable handling of cases where shadow is a non-zero
311 // compile-time constant. For some unexplainable reason they were silently
312 // ignored in the instrumentation.
313 static cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow",
314 cl::desc("Insert checks for constant shadow values"),
315 cl::Hidden, cl::init(false));
316
317 // This is off by default because of a bug in gold:
318 // https://sourceware.org/bugzilla/show_bug.cgi?id=19002
319 static cl::opt<bool> ClWithComdat("msan-with-comdat",
320 cl::desc("Place MSan constructors in comdat sections"),
321 cl::Hidden, cl::init(false));
322
323 // These options allow to specify custom memory map parameters
324 // See MemoryMapParams for details.
325 static cl::opt<uint64_t> ClAndMask("msan-and-mask",
326 cl::desc("Define custom MSan AndMask"),
327 cl::Hidden, cl::init(0));
328
329 static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
330 cl::desc("Define custom MSan XorMask"),
331 cl::Hidden, cl::init(0));
332
333 static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
334 cl::desc("Define custom MSan ShadowBase"),
335 cl::Hidden, cl::init(0));
336
337 static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
338 cl::desc("Define custom MSan OriginBase"),
339 cl::Hidden, cl::init(0));
340
341 const char kMsanModuleCtorName[] = "msan.module_ctor";
342 const char kMsanInitName[] = "__msan_init";
343
344 namespace {
345
346 // Memory map parameters used in application-to-shadow address calculation.
347 // Offset = (Addr & ~AndMask) ^ XorMask
348 // Shadow = ShadowBase + Offset
349 // Origin = OriginBase + Offset
350 struct MemoryMapParams {
351 uint64_t AndMask;
352 uint64_t XorMask;
353 uint64_t ShadowBase;
354 uint64_t OriginBase;
355 };
356
357 struct PlatformMemoryMapParams {
358 const MemoryMapParams *bits32;
359 const MemoryMapParams *bits64;
360 };
361
362 } // end anonymous namespace
363
364 // i386 Linux
365 static const MemoryMapParams Linux_I386_MemoryMapParams = {
366 0x000080000000, // AndMask
367 0, // XorMask (not used)
368 0, // ShadowBase (not used)
369 0x000040000000, // OriginBase
370 };
371
372 // x86_64 Linux
373 static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
374 #ifdef MSAN_LINUX_X86_64_OLD_MAPPING
375 0x400000000000, // AndMask
376 0, // XorMask (not used)
377 0, // ShadowBase (not used)
378 0x200000000000, // OriginBase
379 #else
380 0, // AndMask (not used)
381 0x500000000000, // XorMask
382 0, // ShadowBase (not used)
383 0x100000000000, // OriginBase
384 #endif
385 };
386
387 // mips64 Linux
388 static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
389 0, // AndMask (not used)
390 0x008000000000, // XorMask
391 0, // ShadowBase (not used)
392 0x002000000000, // OriginBase
393 };
394
395 // ppc64 Linux
396 static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
397 0xE00000000000, // AndMask
398 0x100000000000, // XorMask
399 0x080000000000, // ShadowBase
400 0x1C0000000000, // OriginBase
401 };
402
403 // s390x Linux
404 static const MemoryMapParams Linux_S390X_MemoryMapParams = {
405 0xC00000000000, // AndMask
406 0, // XorMask (not used)
407 0x080000000000, // ShadowBase
408 0x1C0000000000, // OriginBase
409 };
410
411 // aarch64 Linux
412 static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
413 0, // AndMask (not used)
414 0x06000000000, // XorMask
415 0, // ShadowBase (not used)
416 0x01000000000, // OriginBase
417 };
418
419 // i386 FreeBSD
420 static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
421 0x000180000000, // AndMask
422 0x000040000000, // XorMask
423 0x000020000000, // ShadowBase
424 0x000700000000, // OriginBase
425 };
426
427 // x86_64 FreeBSD
428 static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
429 0xc00000000000, // AndMask
430 0x200000000000, // XorMask
431 0x100000000000, // ShadowBase
432 0x380000000000, // OriginBase
433 };
434
435 // x86_64 NetBSD
436 static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
437 0, // AndMask
438 0x500000000000, // XorMask
439 0, // ShadowBase
440 0x100000000000, // OriginBase
441 };
442
443 static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
444 &Linux_I386_MemoryMapParams,
445 &Linux_X86_64_MemoryMapParams,
446 };
447
448 static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
449 nullptr,
450 &Linux_MIPS64_MemoryMapParams,
451 };
452
453 static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
454 nullptr,
455 &Linux_PowerPC64_MemoryMapParams,
456 };
457
458 static const PlatformMemoryMapParams Linux_S390_MemoryMapParams = {
459 nullptr,
460 &Linux_S390X_MemoryMapParams,
461 };
462
463 static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
464 nullptr,
465 &Linux_AArch64_MemoryMapParams,
466 };
467
468 static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
469 &FreeBSD_I386_MemoryMapParams,
470 &FreeBSD_X86_64_MemoryMapParams,
471 };
472
473 static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
474 nullptr,
475 &NetBSD_X86_64_MemoryMapParams,
476 };
477
478 namespace {
479
480 /// Instrument functions of a module to detect uninitialized reads.
481 ///
482 /// Instantiating MemorySanitizer inserts the msan runtime library API function
483 /// declarations into the module if they don't exist already. Instantiating
484 /// ensures the __msan_init function is in the list of global constructors for
485 /// the module.
486 class MemorySanitizer {
487 public:
MemorySanitizer(Module & M,MemorySanitizerOptions Options)488 MemorySanitizer(Module &M, MemorySanitizerOptions Options)
489 : CompileKernel(Options.Kernel), TrackOrigins(Options.TrackOrigins),
490 Recover(Options.Recover) {
491 initializeModule(M);
492 }
493
494 // MSan cannot be moved or copied because of MapParams.
495 MemorySanitizer(MemorySanitizer &&) = delete;
496 MemorySanitizer &operator=(MemorySanitizer &&) = delete;
497 MemorySanitizer(const MemorySanitizer &) = delete;
498 MemorySanitizer &operator=(const MemorySanitizer &) = delete;
499
500 bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
501
502 private:
503 friend struct MemorySanitizerVisitor;
504 friend struct VarArgAMD64Helper;
505 friend struct VarArgMIPS64Helper;
506 friend struct VarArgAArch64Helper;
507 friend struct VarArgPowerPC64Helper;
508 friend struct VarArgSystemZHelper;
509
510 void initializeModule(Module &M);
511 void initializeCallbacks(Module &M);
512 void createKernelApi(Module &M);
513 void createUserspaceApi(Module &M);
514
515 /// True if we're compiling the Linux kernel.
516 bool CompileKernel;
517 /// Track origins (allocation points) of uninitialized values.
518 int TrackOrigins;
519 bool Recover;
520
521 LLVMContext *C;
522 Type *IntptrTy;
523 Type *OriginTy;
524
525 // XxxTLS variables represent the per-thread state in MSan and per-task state
526 // in KMSAN.
527 // For the userspace these point to thread-local globals. In the kernel land
528 // they point to the members of a per-task struct obtained via a call to
529 // __msan_get_context_state().
530
531 /// Thread-local shadow storage for function parameters.
532 Value *ParamTLS;
533
534 /// Thread-local origin storage for function parameters.
535 Value *ParamOriginTLS;
536
537 /// Thread-local shadow storage for function return value.
538 Value *RetvalTLS;
539
540 /// Thread-local origin storage for function return value.
541 Value *RetvalOriginTLS;
542
543 /// Thread-local shadow storage for in-register va_arg function
544 /// parameters (x86_64-specific).
545 Value *VAArgTLS;
546
547 /// Thread-local shadow storage for in-register va_arg function
548 /// parameters (x86_64-specific).
549 Value *VAArgOriginTLS;
550
551 /// Thread-local shadow storage for va_arg overflow area
552 /// (x86_64-specific).
553 Value *VAArgOverflowSizeTLS;
554
555 /// Are the instrumentation callbacks set up?
556 bool CallbacksInitialized = false;
557
558 /// The run-time callback to print a warning.
559 FunctionCallee WarningFn;
560
561 // These arrays are indexed by log2(AccessSize).
562 FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
563 FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
564
565 /// Run-time helper that generates a new origin value for a stack
566 /// allocation.
567 FunctionCallee MsanSetAllocaOrigin4Fn;
568
569 /// Run-time helper that poisons stack on function entry.
570 FunctionCallee MsanPoisonStackFn;
571
572 /// Run-time helper that records a store (or any event) of an
573 /// uninitialized value and returns an updated origin id encoding this info.
574 FunctionCallee MsanChainOriginFn;
575
576 /// Run-time helper that paints an origin over a region.
577 FunctionCallee MsanSetOriginFn;
578
579 /// MSan runtime replacements for memmove, memcpy and memset.
580 FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
581
582 /// KMSAN callback for task-local function argument shadow.
583 StructType *MsanContextStateTy;
584 FunctionCallee MsanGetContextStateFn;
585
586 /// Functions for poisoning/unpoisoning local variables
587 FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
588
589 /// Each of the MsanMetadataPtrXxx functions returns a pair of shadow/origin
590 /// pointers.
591 FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
592 FunctionCallee MsanMetadataPtrForLoad_1_8[4];
593 FunctionCallee MsanMetadataPtrForStore_1_8[4];
594 FunctionCallee MsanInstrumentAsmStoreFn;
595
596 /// Helper to choose between different MsanMetadataPtrXxx().
597 FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
598
599 /// Memory map parameters used in application-to-shadow calculation.
600 const MemoryMapParams *MapParams;
601
602 /// Custom memory map parameters used when -msan-shadow-base or
603 // -msan-origin-base is provided.
604 MemoryMapParams CustomMapParams;
605
606 MDNode *ColdCallWeights;
607
608 /// Branch weights for origin store.
609 MDNode *OriginStoreWeights;
610 };
611
insertModuleCtor(Module & M)612 void insertModuleCtor(Module &M) {
613 getOrCreateSanitizerCtorAndInitFunctions(
614 M, kMsanModuleCtorName, kMsanInitName,
615 /*InitArgTypes=*/{},
616 /*InitArgs=*/{},
617 // This callback is invoked when the functions are created the first
618 // time. Hook them into the global ctors list in that case:
619 [&](Function *Ctor, FunctionCallee) {
620 if (!ClWithComdat) {
621 appendToGlobalCtors(M, Ctor, 0);
622 return;
623 }
624 Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
625 Ctor->setComdat(MsanCtorComdat);
626 appendToGlobalCtors(M, Ctor, 0, Ctor);
627 });
628 }
629
630 /// A legacy function pass for msan instrumentation.
631 ///
632 /// Instruments functions to detect uninitialized reads.
633 struct MemorySanitizerLegacyPass : public FunctionPass {
634 // Pass identification, replacement for typeid.
635 static char ID;
636
MemorySanitizerLegacyPass__anonb9e144640211::MemorySanitizerLegacyPass637 MemorySanitizerLegacyPass(MemorySanitizerOptions Options = {})
638 : FunctionPass(ID), Options(Options) {
639 initializeMemorySanitizerLegacyPassPass(*PassRegistry::getPassRegistry());
640 }
getPassName__anonb9e144640211::MemorySanitizerLegacyPass641 StringRef getPassName() const override { return "MemorySanitizerLegacyPass"; }
642
getAnalysisUsage__anonb9e144640211::MemorySanitizerLegacyPass643 void getAnalysisUsage(AnalysisUsage &AU) const override {
644 AU.addRequired<TargetLibraryInfoWrapperPass>();
645 }
646
runOnFunction__anonb9e144640211::MemorySanitizerLegacyPass647 bool runOnFunction(Function &F) override {
648 return MSan->sanitizeFunction(
649 F, getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F));
650 }
651 bool doInitialization(Module &M) override;
652
653 Optional<MemorySanitizer> MSan;
654 MemorySanitizerOptions Options;
655 };
656
getOptOrDefault(const cl::opt<T> & Opt,T Default)657 template <class T> T getOptOrDefault(const cl::opt<T> &Opt, T Default) {
658 return (Opt.getNumOccurrences() > 0) ? Opt : Default;
659 }
660
661 } // end anonymous namespace
662
MemorySanitizerOptions(int TO,bool R,bool K)663 MemorySanitizerOptions::MemorySanitizerOptions(int TO, bool R, bool K)
664 : Kernel(getOptOrDefault(ClEnableKmsan, K)),
665 TrackOrigins(getOptOrDefault(ClTrackOrigins, Kernel ? 2 : TO)),
666 Recover(getOptOrDefault(ClKeepGoing, Kernel || R)) {}
667
run(Function & F,FunctionAnalysisManager & FAM)668 PreservedAnalyses MemorySanitizerPass::run(Function &F,
669 FunctionAnalysisManager &FAM) {
670 MemorySanitizer Msan(*F.getParent(), Options);
671 if (Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F)))
672 return PreservedAnalyses::none();
673 return PreservedAnalyses::all();
674 }
675
run(Module & M,ModuleAnalysisManager & AM)676 PreservedAnalyses MemorySanitizerPass::run(Module &M,
677 ModuleAnalysisManager &AM) {
678 if (Options.Kernel)
679 return PreservedAnalyses::all();
680 insertModuleCtor(M);
681 return PreservedAnalyses::none();
682 }
683
684 char MemorySanitizerLegacyPass::ID = 0;
685
686 INITIALIZE_PASS_BEGIN(MemorySanitizerLegacyPass, "msan",
687 "MemorySanitizer: detects uninitialized reads.", false,
688 false)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)689 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
690 INITIALIZE_PASS_END(MemorySanitizerLegacyPass, "msan",
691 "MemorySanitizer: detects uninitialized reads.", false,
692 false)
693
694 FunctionPass *
695 llvm::createMemorySanitizerLegacyPassPass(MemorySanitizerOptions Options) {
696 return new MemorySanitizerLegacyPass(Options);
697 }
698
699 /// Create a non-const global initialized with the given string.
700 ///
701 /// Creates a writable global for Str so that we can pass it to the
702 /// run-time lib. Runtime uses first 4 bytes of the string to store the
703 /// frame ID, so the string needs to be mutable.
createPrivateNonConstGlobalForString(Module & M,StringRef Str)704 static GlobalVariable *createPrivateNonConstGlobalForString(Module &M,
705 StringRef Str) {
706 Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
707 return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false,
708 GlobalValue::PrivateLinkage, StrConst, "");
709 }
710
711 /// Create KMSAN API callbacks.
createKernelApi(Module & M)712 void MemorySanitizer::createKernelApi(Module &M) {
713 IRBuilder<> IRB(*C);
714
715 // These will be initialized in insertKmsanPrologue().
716 RetvalTLS = nullptr;
717 RetvalOriginTLS = nullptr;
718 ParamTLS = nullptr;
719 ParamOriginTLS = nullptr;
720 VAArgTLS = nullptr;
721 VAArgOriginTLS = nullptr;
722 VAArgOverflowSizeTLS = nullptr;
723
724 WarningFn = M.getOrInsertFunction("__msan_warning", IRB.getVoidTy(),
725 IRB.getInt32Ty());
726 // Requests the per-task context state (kmsan_context_state*) from the
727 // runtime library.
728 MsanContextStateTy = StructType::get(
729 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
730 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
731 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
732 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
733 IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
734 OriginTy);
735 MsanGetContextStateFn = M.getOrInsertFunction(
736 "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
737
738 Type *RetTy = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
739 PointerType::get(IRB.getInt32Ty(), 0));
740
741 for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
742 std::string name_load =
743 "__msan_metadata_ptr_for_load_" + std::to_string(size);
744 std::string name_store =
745 "__msan_metadata_ptr_for_store_" + std::to_string(size);
746 MsanMetadataPtrForLoad_1_8[ind] = M.getOrInsertFunction(
747 name_load, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
748 MsanMetadataPtrForStore_1_8[ind] = M.getOrInsertFunction(
749 name_store, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
750 }
751
752 MsanMetadataPtrForLoadN = M.getOrInsertFunction(
753 "__msan_metadata_ptr_for_load_n", RetTy,
754 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
755 MsanMetadataPtrForStoreN = M.getOrInsertFunction(
756 "__msan_metadata_ptr_for_store_n", RetTy,
757 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
758
759 // Functions for poisoning and unpoisoning memory.
760 MsanPoisonAllocaFn =
761 M.getOrInsertFunction("__msan_poison_alloca", IRB.getVoidTy(),
762 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy());
763 MsanUnpoisonAllocaFn = M.getOrInsertFunction(
764 "__msan_unpoison_alloca", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
765 }
766
getOrInsertGlobal(Module & M,StringRef Name,Type * Ty)767 static Constant *getOrInsertGlobal(Module &M, StringRef Name, Type *Ty) {
768 return M.getOrInsertGlobal(Name, Ty, [&] {
769 return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
770 nullptr, Name, nullptr,
771 GlobalVariable::InitialExecTLSModel);
772 });
773 }
774
775 /// Insert declarations for userspace-specific functions and globals.
createUserspaceApi(Module & M)776 void MemorySanitizer::createUserspaceApi(Module &M) {
777 IRBuilder<> IRB(*C);
778
779 // Create the callback.
780 // FIXME: this function should have "Cold" calling conv,
781 // which is not yet implemented.
782 StringRef WarningFnName = Recover ? "__msan_warning_with_origin"
783 : "__msan_warning_with_origin_noreturn";
784 WarningFn =
785 M.getOrInsertFunction(WarningFnName, IRB.getVoidTy(), IRB.getInt32Ty());
786
787 // Create the global TLS variables.
788 RetvalTLS =
789 getOrInsertGlobal(M, "__msan_retval_tls",
790 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
791
792 RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
793
794 ParamTLS =
795 getOrInsertGlobal(M, "__msan_param_tls",
796 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
797
798 ParamOriginTLS =
799 getOrInsertGlobal(M, "__msan_param_origin_tls",
800 ArrayType::get(OriginTy, kParamTLSSize / 4));
801
802 VAArgTLS =
803 getOrInsertGlobal(M, "__msan_va_arg_tls",
804 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
805
806 VAArgOriginTLS =
807 getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
808 ArrayType::get(OriginTy, kParamTLSSize / 4));
809
810 VAArgOverflowSizeTLS =
811 getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
812
813 for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
814 AccessSizeIndex++) {
815 unsigned AccessSize = 1 << AccessSizeIndex;
816 std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
817 SmallVector<std::pair<unsigned, Attribute>, 2> MaybeWarningFnAttrs;
818 MaybeWarningFnAttrs.push_back(std::make_pair(
819 AttributeList::FirstArgIndex, Attribute::get(*C, Attribute::ZExt)));
820 MaybeWarningFnAttrs.push_back(std::make_pair(
821 AttributeList::FirstArgIndex + 1, Attribute::get(*C, Attribute::ZExt)));
822 MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
823 FunctionName, AttributeList::get(*C, MaybeWarningFnAttrs),
824 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt32Ty());
825
826 FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
827 SmallVector<std::pair<unsigned, Attribute>, 2> MaybeStoreOriginFnAttrs;
828 MaybeStoreOriginFnAttrs.push_back(std::make_pair(
829 AttributeList::FirstArgIndex, Attribute::get(*C, Attribute::ZExt)));
830 MaybeStoreOriginFnAttrs.push_back(std::make_pair(
831 AttributeList::FirstArgIndex + 2, Attribute::get(*C, Attribute::ZExt)));
832 MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
833 FunctionName, AttributeList::get(*C, MaybeStoreOriginFnAttrs),
834 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt8PtrTy(),
835 IRB.getInt32Ty());
836 }
837
838 MsanSetAllocaOrigin4Fn = M.getOrInsertFunction(
839 "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy,
840 IRB.getInt8PtrTy(), IntptrTy);
841 MsanPoisonStackFn =
842 M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(),
843 IRB.getInt8PtrTy(), IntptrTy);
844 }
845
846 /// Insert extern declaration of runtime-provided functions and globals.
initializeCallbacks(Module & M)847 void MemorySanitizer::initializeCallbacks(Module &M) {
848 // Only do this once.
849 if (CallbacksInitialized)
850 return;
851
852 IRBuilder<> IRB(*C);
853 // Initialize callbacks that are common for kernel and userspace
854 // instrumentation.
855 MsanChainOriginFn = M.getOrInsertFunction(
856 "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty());
857 MsanSetOriginFn =
858 M.getOrInsertFunction("__msan_set_origin", IRB.getVoidTy(),
859 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt32Ty());
860 MemmoveFn = M.getOrInsertFunction(
861 "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
862 IRB.getInt8PtrTy(), IntptrTy);
863 MemcpyFn = M.getOrInsertFunction(
864 "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
865 IntptrTy);
866 MemsetFn = M.getOrInsertFunction(
867 "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(),
868 IntptrTy);
869
870 MsanInstrumentAsmStoreFn =
871 M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
872 PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
873
874 if (CompileKernel) {
875 createKernelApi(M);
876 } else {
877 createUserspaceApi(M);
878 }
879 CallbacksInitialized = true;
880 }
881
getKmsanShadowOriginAccessFn(bool isStore,int size)882 FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
883 int size) {
884 FunctionCallee *Fns =
885 isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
886 switch (size) {
887 case 1:
888 return Fns[0];
889 case 2:
890 return Fns[1];
891 case 4:
892 return Fns[2];
893 case 8:
894 return Fns[3];
895 default:
896 return nullptr;
897 }
898 }
899
900 /// Module-level initialization.
901 ///
902 /// inserts a call to __msan_init to the module's constructor list.
initializeModule(Module & M)903 void MemorySanitizer::initializeModule(Module &M) {
904 auto &DL = M.getDataLayout();
905
906 bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
907 bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
908 // Check the overrides first
909 if (ShadowPassed || OriginPassed) {
910 CustomMapParams.AndMask = ClAndMask;
911 CustomMapParams.XorMask = ClXorMask;
912 CustomMapParams.ShadowBase = ClShadowBase;
913 CustomMapParams.OriginBase = ClOriginBase;
914 MapParams = &CustomMapParams;
915 } else {
916 Triple TargetTriple(M.getTargetTriple());
917 switch (TargetTriple.getOS()) {
918 case Triple::FreeBSD:
919 switch (TargetTriple.getArch()) {
920 case Triple::x86_64:
921 MapParams = FreeBSD_X86_MemoryMapParams.bits64;
922 break;
923 case Triple::x86:
924 MapParams = FreeBSD_X86_MemoryMapParams.bits32;
925 break;
926 default:
927 report_fatal_error("unsupported architecture");
928 }
929 break;
930 case Triple::NetBSD:
931 switch (TargetTriple.getArch()) {
932 case Triple::x86_64:
933 MapParams = NetBSD_X86_MemoryMapParams.bits64;
934 break;
935 default:
936 report_fatal_error("unsupported architecture");
937 }
938 break;
939 case Triple::Linux:
940 switch (TargetTriple.getArch()) {
941 case Triple::x86_64:
942 MapParams = Linux_X86_MemoryMapParams.bits64;
943 break;
944 case Triple::x86:
945 MapParams = Linux_X86_MemoryMapParams.bits32;
946 break;
947 case Triple::mips64:
948 case Triple::mips64el:
949 MapParams = Linux_MIPS_MemoryMapParams.bits64;
950 break;
951 case Triple::ppc64:
952 case Triple::ppc64le:
953 MapParams = Linux_PowerPC_MemoryMapParams.bits64;
954 break;
955 case Triple::systemz:
956 MapParams = Linux_S390_MemoryMapParams.bits64;
957 break;
958 case Triple::aarch64:
959 case Triple::aarch64_be:
960 MapParams = Linux_ARM_MemoryMapParams.bits64;
961 break;
962 default:
963 report_fatal_error("unsupported architecture");
964 }
965 break;
966 default:
967 report_fatal_error("unsupported operating system");
968 }
969 }
970
971 C = &(M.getContext());
972 IRBuilder<> IRB(*C);
973 IntptrTy = IRB.getIntPtrTy(DL);
974 OriginTy = IRB.getInt32Ty();
975
976 ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
977 OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
978
979 if (!CompileKernel) {
980 if (TrackOrigins)
981 M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
982 return new GlobalVariable(
983 M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
984 IRB.getInt32(TrackOrigins), "__msan_track_origins");
985 });
986
987 if (Recover)
988 M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
989 return new GlobalVariable(M, IRB.getInt32Ty(), true,
990 GlobalValue::WeakODRLinkage,
991 IRB.getInt32(Recover), "__msan_keep_going");
992 });
993 }
994 }
995
doInitialization(Module & M)996 bool MemorySanitizerLegacyPass::doInitialization(Module &M) {
997 if (!Options.Kernel)
998 insertModuleCtor(M);
999 MSan.emplace(M, Options);
1000 return true;
1001 }
1002
1003 namespace {
1004
1005 /// A helper class that handles instrumentation of VarArg
1006 /// functions on a particular platform.
1007 ///
1008 /// Implementations are expected to insert the instrumentation
1009 /// necessary to propagate argument shadow through VarArg function
1010 /// calls. Visit* methods are called during an InstVisitor pass over
1011 /// the function, and should avoid creating new basic blocks. A new
1012 /// instance of this class is created for each instrumented function.
1013 struct VarArgHelper {
1014 virtual ~VarArgHelper() = default;
1015
1016 /// Visit a CallBase.
1017 virtual void visitCallBase(CallBase &CB, IRBuilder<> &IRB) = 0;
1018
1019 /// Visit a va_start call.
1020 virtual void visitVAStartInst(VAStartInst &I) = 0;
1021
1022 /// Visit a va_copy call.
1023 virtual void visitVACopyInst(VACopyInst &I) = 0;
1024
1025 /// Finalize function instrumentation.
1026 ///
1027 /// This method is called after visiting all interesting (see above)
1028 /// instructions in a function.
1029 virtual void finalizeInstrumentation() = 0;
1030 };
1031
1032 struct MemorySanitizerVisitor;
1033
1034 } // end anonymous namespace
1035
1036 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
1037 MemorySanitizerVisitor &Visitor);
1038
TypeSizeToSizeIndex(unsigned TypeSize)1039 static unsigned TypeSizeToSizeIndex(unsigned TypeSize) {
1040 if (TypeSize <= 8) return 0;
1041 return Log2_32_Ceil((TypeSize + 7) / 8);
1042 }
1043
1044 namespace {
1045
1046 /// This class does all the work for a given function. Store and Load
1047 /// instructions store and load corresponding shadow and origin
1048 /// values. Most instructions propagate shadow from arguments to their
1049 /// return values. Certain instructions (most importantly, BranchInst)
1050 /// test their argument shadow and print reports (with a runtime call) if it's
1051 /// non-zero.
1052 struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1053 Function &F;
1054 MemorySanitizer &MS;
1055 SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1056 ValueMap<Value*, Value*> ShadowMap, OriginMap;
1057 std::unique_ptr<VarArgHelper> VAHelper;
1058 const TargetLibraryInfo *TLI;
1059 Instruction *FnPrologueEnd;
1060
1061 // The following flags disable parts of MSan instrumentation based on
1062 // exclusion list contents and command-line options.
1063 bool InsertChecks;
1064 bool PropagateShadow;
1065 bool PoisonStack;
1066 bool PoisonUndef;
1067
1068 struct ShadowOriginAndInsertPoint {
1069 Value *Shadow;
1070 Value *Origin;
1071 Instruction *OrigIns;
1072
ShadowOriginAndInsertPoint__anonb9e144640811::MemorySanitizerVisitor::ShadowOriginAndInsertPoint1073 ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1074 : Shadow(S), Origin(O), OrigIns(I) {}
1075 };
1076 SmallVector<ShadowOriginAndInsertPoint, 16> InstrumentationList;
1077 bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1078 SmallSet<AllocaInst *, 16> AllocaSet;
1079 SmallVector<std::pair<IntrinsicInst *, AllocaInst *>, 16> LifetimeStartList;
1080 SmallVector<StoreInst *, 16> StoreList;
1081
MemorySanitizerVisitor__anonb9e144640811::MemorySanitizerVisitor1082 MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1083 const TargetLibraryInfo &TLI)
1084 : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1085 bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory);
1086 InsertChecks = SanitizeFunction;
1087 PropagateShadow = SanitizeFunction;
1088 PoisonStack = SanitizeFunction && ClPoisonStack;
1089 PoisonUndef = SanitizeFunction && ClPoisonUndef;
1090
1091 // In the presence of unreachable blocks, we may see Phi nodes with
1092 // incoming nodes from such blocks. Since InstVisitor skips unreachable
1093 // blocks, such nodes will not have any shadow value associated with them.
1094 // It's easier to remove unreachable blocks than deal with missing shadow.
1095 removeUnreachableBlocks(F);
1096
1097 MS.initializeCallbacks(*F.getParent());
1098 FnPrologueEnd = IRBuilder<>(F.getEntryBlock().getFirstNonPHI())
1099 .CreateIntrinsic(Intrinsic::donothing, {}, {});
1100
1101 if (MS.CompileKernel) {
1102 IRBuilder<> IRB(FnPrologueEnd);
1103 insertKmsanPrologue(IRB);
1104 }
1105
1106 LLVM_DEBUG(if (!InsertChecks) dbgs()
1107 << "MemorySanitizer is not inserting checks into '"
1108 << F.getName() << "'\n");
1109 }
1110
isInPrologue__anonb9e144640811::MemorySanitizerVisitor1111 bool isInPrologue(Instruction &I) {
1112 return I.getParent() == FnPrologueEnd->getParent() &&
1113 (&I == FnPrologueEnd || I.comesBefore(FnPrologueEnd));
1114 }
1115
updateOrigin__anonb9e144640811::MemorySanitizerVisitor1116 Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1117 if (MS.TrackOrigins <= 1) return V;
1118 return IRB.CreateCall(MS.MsanChainOriginFn, V);
1119 }
1120
originToIntptr__anonb9e144640811::MemorySanitizerVisitor1121 Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1122 const DataLayout &DL = F.getParent()->getDataLayout();
1123 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1124 if (IntptrSize == kOriginSize) return Origin;
1125 assert(IntptrSize == kOriginSize * 2);
1126 Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1127 return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1128 }
1129
1130 /// Fill memory range with the given origin value.
paintOrigin__anonb9e144640811::MemorySanitizerVisitor1131 void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1132 unsigned Size, Align Alignment) {
1133 const DataLayout &DL = F.getParent()->getDataLayout();
1134 const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
1135 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1136 assert(IntptrAlignment >= kMinOriginAlignment);
1137 assert(IntptrSize >= kOriginSize);
1138
1139 unsigned Ofs = 0;
1140 Align CurrentAlignment = Alignment;
1141 if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1142 Value *IntptrOrigin = originToIntptr(IRB, Origin);
1143 Value *IntptrOriginPtr =
1144 IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1145 for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1146 Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1147 : IntptrOriginPtr;
1148 IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1149 Ofs += IntptrSize / kOriginSize;
1150 CurrentAlignment = IntptrAlignment;
1151 }
1152 }
1153
1154 for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1155 Value *GEP =
1156 i ? IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i) : OriginPtr;
1157 IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1158 CurrentAlignment = kMinOriginAlignment;
1159 }
1160 }
1161
storeOrigin__anonb9e144640811::MemorySanitizerVisitor1162 void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1163 Value *OriginPtr, Align Alignment, bool AsCall) {
1164 const DataLayout &DL = F.getParent()->getDataLayout();
1165 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1166 unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
1167 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1168 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1169 if (ClCheckConstantShadow && !ConstantShadow->isZeroValue())
1170 paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1171 OriginAlignment);
1172 return;
1173 }
1174
1175 unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1176 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1177 if (AsCall && SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1178 FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1179 Value *ConvertedShadow2 =
1180 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1181 CallBase *CB = IRB.CreateCall(
1182 Fn, {ConvertedShadow2,
1183 IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()), Origin});
1184 CB->addParamAttr(0, Attribute::ZExt);
1185 CB->addParamAttr(2, Attribute::ZExt);
1186 } else {
1187 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1188 Instruction *CheckTerm = SplitBlockAndInsertIfThen(
1189 Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1190 IRBuilder<> IRBNew(CheckTerm);
1191 paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1192 OriginAlignment);
1193 }
1194 }
1195
materializeStores__anonb9e144640811::MemorySanitizerVisitor1196 void materializeStores(bool InstrumentWithCalls) {
1197 for (StoreInst *SI : StoreList) {
1198 IRBuilder<> IRB(SI);
1199 Value *Val = SI->getValueOperand();
1200 Value *Addr = SI->getPointerOperand();
1201 Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1202 Value *ShadowPtr, *OriginPtr;
1203 Type *ShadowTy = Shadow->getType();
1204 const Align Alignment = assumeAligned(SI->getAlignment());
1205 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1206 std::tie(ShadowPtr, OriginPtr) =
1207 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1208
1209 StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1210 LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1211 (void)NewSI;
1212
1213 if (SI->isAtomic())
1214 SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1215
1216 if (MS.TrackOrigins && !SI->isAtomic())
1217 storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1218 OriginAlignment, InstrumentWithCalls);
1219 }
1220 }
1221
1222 /// Helper function to insert a warning at IRB's current insert point.
insertWarningFn__anonb9e144640811::MemorySanitizerVisitor1223 void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1224 if (!Origin)
1225 Origin = (Value *)IRB.getInt32(0);
1226 assert(Origin->getType()->isIntegerTy());
1227 IRB.CreateCall(MS.WarningFn, Origin)->setCannotMerge();
1228 // FIXME: Insert UnreachableInst if !MS.Recover?
1229 // This may invalidate some of the following checks and needs to be done
1230 // at the very end.
1231 }
1232
materializeOneCheck__anonb9e144640811::MemorySanitizerVisitor1233 void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin,
1234 bool AsCall) {
1235 IRBuilder<> IRB(OrigIns);
1236 LLVM_DEBUG(dbgs() << " SHAD0 : " << *Shadow << "\n");
1237 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1238 LLVM_DEBUG(dbgs() << " SHAD1 : " << *ConvertedShadow << "\n");
1239
1240 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1241 if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) {
1242 insertWarningFn(IRB, Origin);
1243 }
1244 return;
1245 }
1246
1247 const DataLayout &DL = OrigIns->getModule()->getDataLayout();
1248
1249 unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1250 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1251 if (AsCall && SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1252 FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1253 Value *ConvertedShadow2 =
1254 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1255 CallBase *CB = IRB.CreateCall(
1256 Fn, {ConvertedShadow2,
1257 MS.TrackOrigins && Origin ? Origin : (Value *)IRB.getInt32(0)});
1258 CB->addParamAttr(0, Attribute::ZExt);
1259 CB->addParamAttr(1, Attribute::ZExt);
1260 } else {
1261 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1262 Instruction *CheckTerm = SplitBlockAndInsertIfThen(
1263 Cmp, OrigIns,
1264 /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1265
1266 IRB.SetInsertPoint(CheckTerm);
1267 insertWarningFn(IRB, Origin);
1268 LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1269 }
1270 }
1271
materializeChecks__anonb9e144640811::MemorySanitizerVisitor1272 void materializeChecks(bool InstrumentWithCalls) {
1273 for (const auto &ShadowData : InstrumentationList) {
1274 Instruction *OrigIns = ShadowData.OrigIns;
1275 Value *Shadow = ShadowData.Shadow;
1276 Value *Origin = ShadowData.Origin;
1277 materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls);
1278 }
1279 LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1280 }
1281
1282 // Returns the last instruction in the new prologue
insertKmsanPrologue__anonb9e144640811::MemorySanitizerVisitor1283 void insertKmsanPrologue(IRBuilder<> &IRB) {
1284 Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1285 Constant *Zero = IRB.getInt32(0);
1286 MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1287 {Zero, IRB.getInt32(0)}, "param_shadow");
1288 MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1289 {Zero, IRB.getInt32(1)}, "retval_shadow");
1290 MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1291 {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1292 MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1293 {Zero, IRB.getInt32(3)}, "va_arg_origin");
1294 MS.VAArgOverflowSizeTLS =
1295 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1296 {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1297 MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1298 {Zero, IRB.getInt32(5)}, "param_origin");
1299 MS.RetvalOriginTLS =
1300 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1301 {Zero, IRB.getInt32(6)}, "retval_origin");
1302 }
1303
1304 /// Add MemorySanitizer instrumentation to a function.
runOnFunction__anonb9e144640811::MemorySanitizerVisitor1305 bool runOnFunction() {
1306 // Iterate all BBs in depth-first order and create shadow instructions
1307 // for all instructions (where applicable).
1308 // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1309 for (BasicBlock *BB : depth_first(FnPrologueEnd->getParent()))
1310 visit(*BB);
1311
1312 // Finalize PHI nodes.
1313 for (PHINode *PN : ShadowPHINodes) {
1314 PHINode *PNS = cast<PHINode>(getShadow(PN));
1315 PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1316 size_t NumValues = PN->getNumIncomingValues();
1317 for (size_t v = 0; v < NumValues; v++) {
1318 PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1319 if (PNO) PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1320 }
1321 }
1322
1323 VAHelper->finalizeInstrumentation();
1324
1325 // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1326 // instrumenting only allocas.
1327 if (InstrumentLifetimeStart) {
1328 for (auto Item : LifetimeStartList) {
1329 instrumentAlloca(*Item.second, Item.first);
1330 AllocaSet.erase(Item.second);
1331 }
1332 }
1333 // Poison the allocas for which we didn't instrument the corresponding
1334 // lifetime intrinsics.
1335 for (AllocaInst *AI : AllocaSet)
1336 instrumentAlloca(*AI);
1337
1338 bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 &&
1339 InstrumentationList.size() + StoreList.size() >
1340 (unsigned)ClInstrumentationWithCallThreshold;
1341
1342 // Insert shadow value checks.
1343 materializeChecks(InstrumentWithCalls);
1344
1345 // Delayed instrumentation of StoreInst.
1346 // This may not add new address checks.
1347 materializeStores(InstrumentWithCalls);
1348
1349 return true;
1350 }
1351
1352 /// Compute the shadow type that corresponds to a given Value.
getShadowTy__anonb9e144640811::MemorySanitizerVisitor1353 Type *getShadowTy(Value *V) {
1354 return getShadowTy(V->getType());
1355 }
1356
1357 /// Compute the shadow type that corresponds to a given Type.
getShadowTy__anonb9e144640811::MemorySanitizerVisitor1358 Type *getShadowTy(Type *OrigTy) {
1359 if (!OrigTy->isSized()) {
1360 return nullptr;
1361 }
1362 // For integer type, shadow is the same as the original type.
1363 // This may return weird-sized types like i1.
1364 if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1365 return IT;
1366 const DataLayout &DL = F.getParent()->getDataLayout();
1367 if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1368 uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1369 return FixedVectorType::get(IntegerType::get(*MS.C, EltSize),
1370 cast<FixedVectorType>(VT)->getNumElements());
1371 }
1372 if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1373 return ArrayType::get(getShadowTy(AT->getElementType()),
1374 AT->getNumElements());
1375 }
1376 if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1377 SmallVector<Type*, 4> Elements;
1378 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1379 Elements.push_back(getShadowTy(ST->getElementType(i)));
1380 StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1381 LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1382 return Res;
1383 }
1384 uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1385 return IntegerType::get(*MS.C, TypeSize);
1386 }
1387
1388 /// Flatten a vector type.
getShadowTyNoVec__anonb9e144640811::MemorySanitizerVisitor1389 Type *getShadowTyNoVec(Type *ty) {
1390 if (VectorType *vt = dyn_cast<VectorType>(ty))
1391 return IntegerType::get(*MS.C,
1392 vt->getPrimitiveSizeInBits().getFixedSize());
1393 return ty;
1394 }
1395
1396 /// Extract combined shadow of struct elements as a bool
collapseStructShadow__anonb9e144640811::MemorySanitizerVisitor1397 Value *collapseStructShadow(StructType *Struct, Value *Shadow,
1398 IRBuilder<> &IRB) {
1399 Value *FalseVal = IRB.getIntN(/* width */ 1, /* value */ 0);
1400 Value *Aggregator = FalseVal;
1401
1402 for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
1403 // Combine by ORing together each element's bool shadow
1404 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1405 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1406 Value *ShadowBool = convertToBool(ShadowInner, IRB);
1407
1408 if (Aggregator != FalseVal)
1409 Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
1410 else
1411 Aggregator = ShadowBool;
1412 }
1413
1414 return Aggregator;
1415 }
1416
1417 // Extract combined shadow of array elements
collapseArrayShadow__anonb9e144640811::MemorySanitizerVisitor1418 Value *collapseArrayShadow(ArrayType *Array, Value *Shadow,
1419 IRBuilder<> &IRB) {
1420 if (!Array->getNumElements())
1421 return IRB.getIntN(/* width */ 1, /* value */ 0);
1422
1423 Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
1424 Value *Aggregator = convertShadowToScalar(FirstItem, IRB);
1425
1426 for (unsigned Idx = 1; Idx < Array->getNumElements(); Idx++) {
1427 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1428 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1429 Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
1430 }
1431 return Aggregator;
1432 }
1433
1434 /// Convert a shadow value to it's flattened variant. The resulting
1435 /// shadow may not necessarily have the same bit width as the input
1436 /// value, but it will always be comparable to zero.
convertShadowToScalar__anonb9e144640811::MemorySanitizerVisitor1437 Value *convertShadowToScalar(Value *V, IRBuilder<> &IRB) {
1438 if (StructType *Struct = dyn_cast<StructType>(V->getType()))
1439 return collapseStructShadow(Struct, V, IRB);
1440 if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
1441 return collapseArrayShadow(Array, V, IRB);
1442 Type *Ty = V->getType();
1443 Type *NoVecTy = getShadowTyNoVec(Ty);
1444 if (Ty == NoVecTy) return V;
1445 return IRB.CreateBitCast(V, NoVecTy);
1446 }
1447
1448 // Convert a scalar value to an i1 by comparing with 0
convertToBool__anonb9e144640811::MemorySanitizerVisitor1449 Value *convertToBool(Value *V, IRBuilder<> &IRB, const Twine &name = "") {
1450 Type *VTy = V->getType();
1451 assert(VTy->isIntegerTy());
1452 if (VTy->getIntegerBitWidth() == 1)
1453 // Just converting a bool to a bool, so do nothing.
1454 return V;
1455 return IRB.CreateICmpNE(V, ConstantInt::get(VTy, 0), name);
1456 }
1457
1458 /// Compute the integer shadow offset that corresponds to a given
1459 /// application address.
1460 ///
1461 /// Offset = (Addr & ~AndMask) ^ XorMask
getShadowPtrOffset__anonb9e144640811::MemorySanitizerVisitor1462 Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1463 Value *OffsetLong = IRB.CreatePointerCast(Addr, MS.IntptrTy);
1464
1465 uint64_t AndMask = MS.MapParams->AndMask;
1466 if (AndMask)
1467 OffsetLong =
1468 IRB.CreateAnd(OffsetLong, ConstantInt::get(MS.IntptrTy, ~AndMask));
1469
1470 uint64_t XorMask = MS.MapParams->XorMask;
1471 if (XorMask)
1472 OffsetLong =
1473 IRB.CreateXor(OffsetLong, ConstantInt::get(MS.IntptrTy, XorMask));
1474 return OffsetLong;
1475 }
1476
1477 /// Compute the shadow and origin addresses corresponding to a given
1478 /// application address.
1479 ///
1480 /// Shadow = ShadowBase + Offset
1481 /// Origin = (OriginBase + Offset) & ~3ULL
1482 std::pair<Value *, Value *>
getShadowOriginPtrUserspace__anonb9e144640811::MemorySanitizerVisitor1483 getShadowOriginPtrUserspace(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1484 MaybeAlign Alignment) {
1485 Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1486 Value *ShadowLong = ShadowOffset;
1487 uint64_t ShadowBase = MS.MapParams->ShadowBase;
1488 if (ShadowBase != 0) {
1489 ShadowLong =
1490 IRB.CreateAdd(ShadowLong,
1491 ConstantInt::get(MS.IntptrTy, ShadowBase));
1492 }
1493 Value *ShadowPtr =
1494 IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0));
1495 Value *OriginPtr = nullptr;
1496 if (MS.TrackOrigins) {
1497 Value *OriginLong = ShadowOffset;
1498 uint64_t OriginBase = MS.MapParams->OriginBase;
1499 if (OriginBase != 0)
1500 OriginLong = IRB.CreateAdd(OriginLong,
1501 ConstantInt::get(MS.IntptrTy, OriginBase));
1502 if (!Alignment || *Alignment < kMinOriginAlignment) {
1503 uint64_t Mask = kMinOriginAlignment.value() - 1;
1504 OriginLong =
1505 IRB.CreateAnd(OriginLong, ConstantInt::get(MS.IntptrTy, ~Mask));
1506 }
1507 OriginPtr =
1508 IRB.CreateIntToPtr(OriginLong, PointerType::get(MS.OriginTy, 0));
1509 }
1510 return std::make_pair(ShadowPtr, OriginPtr);
1511 }
1512
getShadowOriginPtrKernel__anonb9e144640811::MemorySanitizerVisitor1513 std::pair<Value *, Value *> getShadowOriginPtrKernel(Value *Addr,
1514 IRBuilder<> &IRB,
1515 Type *ShadowTy,
1516 bool isStore) {
1517 Value *ShadowOriginPtrs;
1518 const DataLayout &DL = F.getParent()->getDataLayout();
1519 int Size = DL.getTypeStoreSize(ShadowTy);
1520
1521 FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1522 Value *AddrCast =
1523 IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1524 if (Getter) {
1525 ShadowOriginPtrs = IRB.CreateCall(Getter, AddrCast);
1526 } else {
1527 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1528 ShadowOriginPtrs = IRB.CreateCall(isStore ? MS.MsanMetadataPtrForStoreN
1529 : MS.MsanMetadataPtrForLoadN,
1530 {AddrCast, SizeVal});
1531 }
1532 Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1533 ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1534 Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1535
1536 return std::make_pair(ShadowPtr, OriginPtr);
1537 }
1538
getShadowOriginPtr__anonb9e144640811::MemorySanitizerVisitor1539 std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1540 Type *ShadowTy,
1541 MaybeAlign Alignment,
1542 bool isStore) {
1543 if (MS.CompileKernel)
1544 return getShadowOriginPtrKernel(Addr, IRB, ShadowTy, isStore);
1545 return getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1546 }
1547
1548 /// Compute the shadow address for a given function argument.
1549 ///
1550 /// Shadow = ParamTLS+ArgOffset.
getShadowPtrForArgument__anonb9e144640811::MemorySanitizerVisitor1551 Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB,
1552 int ArgOffset) {
1553 Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1554 if (ArgOffset)
1555 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1556 return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1557 "_msarg");
1558 }
1559
1560 /// Compute the origin address for a given function argument.
getOriginPtrForArgument__anonb9e144640811::MemorySanitizerVisitor1561 Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB,
1562 int ArgOffset) {
1563 if (!MS.TrackOrigins)
1564 return nullptr;
1565 Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1566 if (ArgOffset)
1567 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1568 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
1569 "_msarg_o");
1570 }
1571
1572 /// Compute the shadow address for a retval.
getShadowPtrForRetval__anonb9e144640811::MemorySanitizerVisitor1573 Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) {
1574 return IRB.CreatePointerCast(MS.RetvalTLS,
1575 PointerType::get(getShadowTy(A), 0),
1576 "_msret");
1577 }
1578
1579 /// Compute the origin address for a retval.
getOriginPtrForRetval__anonb9e144640811::MemorySanitizerVisitor1580 Value *getOriginPtrForRetval(IRBuilder<> &IRB) {
1581 // We keep a single origin for the entire retval. Might be too optimistic.
1582 return MS.RetvalOriginTLS;
1583 }
1584
1585 /// Set SV to be the shadow value for V.
setShadow__anonb9e144640811::MemorySanitizerVisitor1586 void setShadow(Value *V, Value *SV) {
1587 assert(!ShadowMap.count(V) && "Values may only have one shadow");
1588 ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1589 }
1590
1591 /// Set Origin to be the origin value for V.
setOrigin__anonb9e144640811::MemorySanitizerVisitor1592 void setOrigin(Value *V, Value *Origin) {
1593 if (!MS.TrackOrigins) return;
1594 assert(!OriginMap.count(V) && "Values may only have one origin");
1595 LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1596 OriginMap[V] = Origin;
1597 }
1598
getCleanShadow__anonb9e144640811::MemorySanitizerVisitor1599 Constant *getCleanShadow(Type *OrigTy) {
1600 Type *ShadowTy = getShadowTy(OrigTy);
1601 if (!ShadowTy)
1602 return nullptr;
1603 return Constant::getNullValue(ShadowTy);
1604 }
1605
1606 /// Create a clean shadow value for a given value.
1607 ///
1608 /// Clean shadow (all zeroes) means all bits of the value are defined
1609 /// (initialized).
getCleanShadow__anonb9e144640811::MemorySanitizerVisitor1610 Constant *getCleanShadow(Value *V) {
1611 return getCleanShadow(V->getType());
1612 }
1613
1614 /// Create a dirty shadow of a given shadow type.
getPoisonedShadow__anonb9e144640811::MemorySanitizerVisitor1615 Constant *getPoisonedShadow(Type *ShadowTy) {
1616 assert(ShadowTy);
1617 if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1618 return Constant::getAllOnesValue(ShadowTy);
1619 if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1620 SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1621 getPoisonedShadow(AT->getElementType()));
1622 return ConstantArray::get(AT, Vals);
1623 }
1624 if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1625 SmallVector<Constant *, 4> Vals;
1626 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1627 Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1628 return ConstantStruct::get(ST, Vals);
1629 }
1630 llvm_unreachable("Unexpected shadow type");
1631 }
1632
1633 /// Create a dirty shadow for a given value.
getPoisonedShadow__anonb9e144640811::MemorySanitizerVisitor1634 Constant *getPoisonedShadow(Value *V) {
1635 Type *ShadowTy = getShadowTy(V);
1636 if (!ShadowTy)
1637 return nullptr;
1638 return getPoisonedShadow(ShadowTy);
1639 }
1640
1641 /// Create a clean (zero) origin.
getCleanOrigin__anonb9e144640811::MemorySanitizerVisitor1642 Value *getCleanOrigin() {
1643 return Constant::getNullValue(MS.OriginTy);
1644 }
1645
1646 /// Get the shadow value for a given Value.
1647 ///
1648 /// This function either returns the value set earlier with setShadow,
1649 /// or extracts if from ParamTLS (for function arguments).
getShadow__anonb9e144640811::MemorySanitizerVisitor1650 Value *getShadow(Value *V) {
1651 if (!PropagateShadow) return getCleanShadow(V);
1652 if (Instruction *I = dyn_cast<Instruction>(V)) {
1653 if (I->getMetadata("nosanitize"))
1654 return getCleanShadow(V);
1655 // For instructions the shadow is already stored in the map.
1656 Value *Shadow = ShadowMap[V];
1657 if (!Shadow) {
1658 LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1659 (void)I;
1660 assert(Shadow && "No shadow for a value");
1661 }
1662 return Shadow;
1663 }
1664 if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1665 Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : getCleanShadow(V);
1666 LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1667 (void)U;
1668 return AllOnes;
1669 }
1670 if (Argument *A = dyn_cast<Argument>(V)) {
1671 // For arguments we compute the shadow on demand and store it in the map.
1672 Value **ShadowPtr = &ShadowMap[V];
1673 if (*ShadowPtr)
1674 return *ShadowPtr;
1675 Function *F = A->getParent();
1676 IRBuilder<> EntryIRB(FnPrologueEnd);
1677 unsigned ArgOffset = 0;
1678 const DataLayout &DL = F->getParent()->getDataLayout();
1679 for (auto &FArg : F->args()) {
1680 if (!FArg.getType()->isSized()) {
1681 LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1682 continue;
1683 }
1684
1685 bool FArgByVal = FArg.hasByValAttr();
1686 bool FArgNoUndef = FArg.hasAttribute(Attribute::NoUndef);
1687 bool FArgEagerCheck = ClEagerChecks && !FArgByVal && FArgNoUndef;
1688 unsigned Size =
1689 FArg.hasByValAttr()
1690 ? DL.getTypeAllocSize(FArg.getParamByValType())
1691 : DL.getTypeAllocSize(FArg.getType());
1692
1693 if (A == &FArg) {
1694 bool Overflow = ArgOffset + Size > kParamTLSSize;
1695 if (FArgEagerCheck) {
1696 *ShadowPtr = getCleanShadow(V);
1697 setOrigin(A, getCleanOrigin());
1698 continue;
1699 } else if (FArgByVal) {
1700 Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1701 // ByVal pointer itself has clean shadow. We copy the actual
1702 // argument shadow to the underlying memory.
1703 // Figure out maximal valid memcpy alignment.
1704 const Align ArgAlign = DL.getValueOrABITypeAlignment(
1705 MaybeAlign(FArg.getParamAlignment()), FArg.getParamByValType());
1706 Value *CpShadowPtr =
1707 getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1708 /*isStore*/ true)
1709 .first;
1710 // TODO(glider): need to copy origins.
1711 if (Overflow) {
1712 // ParamTLS overflow.
1713 EntryIRB.CreateMemSet(
1714 CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1715 Size, ArgAlign);
1716 } else {
1717 const Align CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1718 Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1719 CopyAlign, Size);
1720 LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1721 (void)Cpy;
1722 }
1723 *ShadowPtr = getCleanShadow(V);
1724 } else {
1725 // Shadow over TLS
1726 Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1727 if (Overflow) {
1728 // ParamTLS overflow.
1729 *ShadowPtr = getCleanShadow(V);
1730 } else {
1731 *ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
1732 kShadowTLSAlignment);
1733 }
1734 }
1735 LLVM_DEBUG(dbgs()
1736 << " ARG: " << FArg << " ==> " << **ShadowPtr << "\n");
1737 if (MS.TrackOrigins && !Overflow) {
1738 Value *OriginPtr =
1739 getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1740 setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
1741 } else {
1742 setOrigin(A, getCleanOrigin());
1743 }
1744
1745 break;
1746 }
1747
1748 if (!FArgEagerCheck)
1749 ArgOffset += alignTo(Size, kShadowTLSAlignment);
1750 }
1751 assert(*ShadowPtr && "Could not find shadow for an argument");
1752 return *ShadowPtr;
1753 }
1754 // For everything else the shadow is zero.
1755 return getCleanShadow(V);
1756 }
1757
1758 /// Get the shadow for i-th argument of the instruction I.
getShadow__anonb9e144640811::MemorySanitizerVisitor1759 Value *getShadow(Instruction *I, int i) {
1760 return getShadow(I->getOperand(i));
1761 }
1762
1763 /// Get the origin for a value.
getOrigin__anonb9e144640811::MemorySanitizerVisitor1764 Value *getOrigin(Value *V) {
1765 if (!MS.TrackOrigins) return nullptr;
1766 if (!PropagateShadow) return getCleanOrigin();
1767 if (isa<Constant>(V)) return getCleanOrigin();
1768 assert((isa<Instruction>(V) || isa<Argument>(V)) &&
1769 "Unexpected value type in getOrigin()");
1770 if (Instruction *I = dyn_cast<Instruction>(V)) {
1771 if (I->getMetadata("nosanitize"))
1772 return getCleanOrigin();
1773 }
1774 Value *Origin = OriginMap[V];
1775 assert(Origin && "Missing origin");
1776 return Origin;
1777 }
1778
1779 /// Get the origin for i-th argument of the instruction I.
getOrigin__anonb9e144640811::MemorySanitizerVisitor1780 Value *getOrigin(Instruction *I, int i) {
1781 return getOrigin(I->getOperand(i));
1782 }
1783
1784 /// Remember the place where a shadow check should be inserted.
1785 ///
1786 /// This location will be later instrumented with a check that will print a
1787 /// UMR warning in runtime if the shadow value is not 0.
insertShadowCheck__anonb9e144640811::MemorySanitizerVisitor1788 void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
1789 assert(Shadow);
1790 if (!InsertChecks) return;
1791 #ifndef NDEBUG
1792 Type *ShadowTy = Shadow->getType();
1793 assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy) ||
1794 isa<StructType>(ShadowTy) || isa<ArrayType>(ShadowTy)) &&
1795 "Can only insert checks for integer, vector, and aggregate shadow "
1796 "types");
1797 #endif
1798 InstrumentationList.push_back(
1799 ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
1800 }
1801
1802 /// Remember the place where a shadow check should be inserted.
1803 ///
1804 /// This location will be later instrumented with a check that will print a
1805 /// UMR warning in runtime if the value is not fully defined.
insertShadowCheck__anonb9e144640811::MemorySanitizerVisitor1806 void insertShadowCheck(Value *Val, Instruction *OrigIns) {
1807 assert(Val);
1808 Value *Shadow, *Origin;
1809 if (ClCheckConstantShadow) {
1810 Shadow = getShadow(Val);
1811 if (!Shadow) return;
1812 Origin = getOrigin(Val);
1813 } else {
1814 Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
1815 if (!Shadow) return;
1816 Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
1817 }
1818 insertShadowCheck(Shadow, Origin, OrigIns);
1819 }
1820
addReleaseOrdering__anonb9e144640811::MemorySanitizerVisitor1821 AtomicOrdering addReleaseOrdering(AtomicOrdering a) {
1822 switch (a) {
1823 case AtomicOrdering::NotAtomic:
1824 return AtomicOrdering::NotAtomic;
1825 case AtomicOrdering::Unordered:
1826 case AtomicOrdering::Monotonic:
1827 case AtomicOrdering::Release:
1828 return AtomicOrdering::Release;
1829 case AtomicOrdering::Acquire:
1830 case AtomicOrdering::AcquireRelease:
1831 return AtomicOrdering::AcquireRelease;
1832 case AtomicOrdering::SequentiallyConsistent:
1833 return AtomicOrdering::SequentiallyConsistent;
1834 }
1835 llvm_unreachable("Unknown ordering");
1836 }
1837
makeAddReleaseOrderingTable__anonb9e144640811::MemorySanitizerVisitor1838 Value *makeAddReleaseOrderingTable(IRBuilder<> &IRB) {
1839 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
1840 uint32_t OrderingTable[NumOrderings] = {};
1841
1842 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
1843 OrderingTable[(int)AtomicOrderingCABI::release] =
1844 (int)AtomicOrderingCABI::release;
1845 OrderingTable[(int)AtomicOrderingCABI::consume] =
1846 OrderingTable[(int)AtomicOrderingCABI::acquire] =
1847 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
1848 (int)AtomicOrderingCABI::acq_rel;
1849 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
1850 (int)AtomicOrderingCABI::seq_cst;
1851
1852 return ConstantDataVector::get(IRB.getContext(),
1853 makeArrayRef(OrderingTable, NumOrderings));
1854 }
1855
addAcquireOrdering__anonb9e144640811::MemorySanitizerVisitor1856 AtomicOrdering addAcquireOrdering(AtomicOrdering a) {
1857 switch (a) {
1858 case AtomicOrdering::NotAtomic:
1859 return AtomicOrdering::NotAtomic;
1860 case AtomicOrdering::Unordered:
1861 case AtomicOrdering::Monotonic:
1862 case AtomicOrdering::Acquire:
1863 return AtomicOrdering::Acquire;
1864 case AtomicOrdering::Release:
1865 case AtomicOrdering::AcquireRelease:
1866 return AtomicOrdering::AcquireRelease;
1867 case AtomicOrdering::SequentiallyConsistent:
1868 return AtomicOrdering::SequentiallyConsistent;
1869 }
1870 llvm_unreachable("Unknown ordering");
1871 }
1872
makeAddAcquireOrderingTable__anonb9e144640811::MemorySanitizerVisitor1873 Value *makeAddAcquireOrderingTable(IRBuilder<> &IRB) {
1874 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
1875 uint32_t OrderingTable[NumOrderings] = {};
1876
1877 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
1878 OrderingTable[(int)AtomicOrderingCABI::acquire] =
1879 OrderingTable[(int)AtomicOrderingCABI::consume] =
1880 (int)AtomicOrderingCABI::acquire;
1881 OrderingTable[(int)AtomicOrderingCABI::release] =
1882 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
1883 (int)AtomicOrderingCABI::acq_rel;
1884 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
1885 (int)AtomicOrderingCABI::seq_cst;
1886
1887 return ConstantDataVector::get(IRB.getContext(),
1888 makeArrayRef(OrderingTable, NumOrderings));
1889 }
1890
1891 // ------------------- Visitors.
1892 using InstVisitor<MemorySanitizerVisitor>::visit;
visit__anonb9e144640811::MemorySanitizerVisitor1893 void visit(Instruction &I) {
1894 if (I.getMetadata("nosanitize"))
1895 return;
1896 // Don't want to visit if we're in the prologue
1897 if (isInPrologue(I))
1898 return;
1899 InstVisitor<MemorySanitizerVisitor>::visit(I);
1900 }
1901
1902 /// Instrument LoadInst
1903 ///
1904 /// Loads the corresponding shadow and (optionally) origin.
1905 /// Optionally, checks that the load address is fully defined.
visitLoadInst__anonb9e144640811::MemorySanitizerVisitor1906 void visitLoadInst(LoadInst &I) {
1907 assert(I.getType()->isSized() && "Load type must have size");
1908 assert(!I.getMetadata("nosanitize"));
1909 IRBuilder<> IRB(I.getNextNode());
1910 Type *ShadowTy = getShadowTy(&I);
1911 Value *Addr = I.getPointerOperand();
1912 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
1913 const Align Alignment = assumeAligned(I.getAlignment());
1914 if (PropagateShadow) {
1915 std::tie(ShadowPtr, OriginPtr) =
1916 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
1917 setShadow(&I,
1918 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
1919 } else {
1920 setShadow(&I, getCleanShadow(&I));
1921 }
1922
1923 if (ClCheckAccessAddress)
1924 insertShadowCheck(I.getPointerOperand(), &I);
1925
1926 if (I.isAtomic())
1927 I.setOrdering(addAcquireOrdering(I.getOrdering()));
1928
1929 if (MS.TrackOrigins) {
1930 if (PropagateShadow) {
1931 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1932 setOrigin(
1933 &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
1934 } else {
1935 setOrigin(&I, getCleanOrigin());
1936 }
1937 }
1938 }
1939
1940 /// Instrument StoreInst
1941 ///
1942 /// Stores the corresponding shadow and (optionally) origin.
1943 /// Optionally, checks that the store address is fully defined.
visitStoreInst__anonb9e144640811::MemorySanitizerVisitor1944 void visitStoreInst(StoreInst &I) {
1945 StoreList.push_back(&I);
1946 if (ClCheckAccessAddress)
1947 insertShadowCheck(I.getPointerOperand(), &I);
1948 }
1949
handleCASOrRMW__anonb9e144640811::MemorySanitizerVisitor1950 void handleCASOrRMW(Instruction &I) {
1951 assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
1952
1953 IRBuilder<> IRB(&I);
1954 Value *Addr = I.getOperand(0);
1955 Value *Val = I.getOperand(1);
1956 Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, Val->getType(), Align(1),
1957 /*isStore*/ true)
1958 .first;
1959
1960 if (ClCheckAccessAddress)
1961 insertShadowCheck(Addr, &I);
1962
1963 // Only test the conditional argument of cmpxchg instruction.
1964 // The other argument can potentially be uninitialized, but we can not
1965 // detect this situation reliably without possible false positives.
1966 if (isa<AtomicCmpXchgInst>(I))
1967 insertShadowCheck(Val, &I);
1968
1969 IRB.CreateStore(getCleanShadow(Val), ShadowPtr);
1970
1971 setShadow(&I, getCleanShadow(&I));
1972 setOrigin(&I, getCleanOrigin());
1973 }
1974
visitAtomicRMWInst__anonb9e144640811::MemorySanitizerVisitor1975 void visitAtomicRMWInst(AtomicRMWInst &I) {
1976 handleCASOrRMW(I);
1977 I.setOrdering(addReleaseOrdering(I.getOrdering()));
1978 }
1979
visitAtomicCmpXchgInst__anonb9e144640811::MemorySanitizerVisitor1980 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
1981 handleCASOrRMW(I);
1982 I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
1983 }
1984
1985 // Vector manipulation.
visitExtractElementInst__anonb9e144640811::MemorySanitizerVisitor1986 void visitExtractElementInst(ExtractElementInst &I) {
1987 insertShadowCheck(I.getOperand(1), &I);
1988 IRBuilder<> IRB(&I);
1989 setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
1990 "_msprop"));
1991 setOrigin(&I, getOrigin(&I, 0));
1992 }
1993
visitInsertElementInst__anonb9e144640811::MemorySanitizerVisitor1994 void visitInsertElementInst(InsertElementInst &I) {
1995 insertShadowCheck(I.getOperand(2), &I);
1996 IRBuilder<> IRB(&I);
1997 setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1),
1998 I.getOperand(2), "_msprop"));
1999 setOriginForNaryOp(I);
2000 }
2001
visitShuffleVectorInst__anonb9e144640811::MemorySanitizerVisitor2002 void visitShuffleVectorInst(ShuffleVectorInst &I) {
2003 IRBuilder<> IRB(&I);
2004 setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1),
2005 I.getShuffleMask(), "_msprop"));
2006 setOriginForNaryOp(I);
2007 }
2008
2009 // Casts.
visitSExtInst__anonb9e144640811::MemorySanitizerVisitor2010 void visitSExtInst(SExtInst &I) {
2011 IRBuilder<> IRB(&I);
2012 setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
2013 setOrigin(&I, getOrigin(&I, 0));
2014 }
2015
visitZExtInst__anonb9e144640811::MemorySanitizerVisitor2016 void visitZExtInst(ZExtInst &I) {
2017 IRBuilder<> IRB(&I);
2018 setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
2019 setOrigin(&I, getOrigin(&I, 0));
2020 }
2021
visitTruncInst__anonb9e144640811::MemorySanitizerVisitor2022 void visitTruncInst(TruncInst &I) {
2023 IRBuilder<> IRB(&I);
2024 setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
2025 setOrigin(&I, getOrigin(&I, 0));
2026 }
2027
visitBitCastInst__anonb9e144640811::MemorySanitizerVisitor2028 void visitBitCastInst(BitCastInst &I) {
2029 // Special case: if this is the bitcast (there is exactly 1 allowed) between
2030 // a musttail call and a ret, don't instrument. New instructions are not
2031 // allowed after a musttail call.
2032 if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
2033 if (CI->isMustTailCall())
2034 return;
2035 IRBuilder<> IRB(&I);
2036 setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
2037 setOrigin(&I, getOrigin(&I, 0));
2038 }
2039
visitPtrToIntInst__anonb9e144640811::MemorySanitizerVisitor2040 void visitPtrToIntInst(PtrToIntInst &I) {
2041 IRBuilder<> IRB(&I);
2042 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2043 "_msprop_ptrtoint"));
2044 setOrigin(&I, getOrigin(&I, 0));
2045 }
2046
visitIntToPtrInst__anonb9e144640811::MemorySanitizerVisitor2047 void visitIntToPtrInst(IntToPtrInst &I) {
2048 IRBuilder<> IRB(&I);
2049 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2050 "_msprop_inttoptr"));
2051 setOrigin(&I, getOrigin(&I, 0));
2052 }
2053
visitFPToSIInst__anonb9e144640811::MemorySanitizerVisitor2054 void visitFPToSIInst(CastInst& I) { handleShadowOr(I); }
visitFPToUIInst__anonb9e144640811::MemorySanitizerVisitor2055 void visitFPToUIInst(CastInst& I) { handleShadowOr(I); }
visitSIToFPInst__anonb9e144640811::MemorySanitizerVisitor2056 void visitSIToFPInst(CastInst& I) { handleShadowOr(I); }
visitUIToFPInst__anonb9e144640811::MemorySanitizerVisitor2057 void visitUIToFPInst(CastInst& I) { handleShadowOr(I); }
visitFPExtInst__anonb9e144640811::MemorySanitizerVisitor2058 void visitFPExtInst(CastInst& I) { handleShadowOr(I); }
visitFPTruncInst__anonb9e144640811::MemorySanitizerVisitor2059 void visitFPTruncInst(CastInst& I) { handleShadowOr(I); }
2060
2061 /// Propagate shadow for bitwise AND.
2062 ///
2063 /// This code is exact, i.e. if, for example, a bit in the left argument
2064 /// is defined and 0, then neither the value not definedness of the
2065 /// corresponding bit in B don't affect the resulting shadow.
visitAnd__anonb9e144640811::MemorySanitizerVisitor2066 void visitAnd(BinaryOperator &I) {
2067 IRBuilder<> IRB(&I);
2068 // "And" of 0 and a poisoned value results in unpoisoned value.
2069 // 1&1 => 1; 0&1 => 0; p&1 => p;
2070 // 1&0 => 0; 0&0 => 0; p&0 => 0;
2071 // 1&p => p; 0&p => 0; p&p => p;
2072 // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
2073 Value *S1 = getShadow(&I, 0);
2074 Value *S2 = getShadow(&I, 1);
2075 Value *V1 = I.getOperand(0);
2076 Value *V2 = I.getOperand(1);
2077 if (V1->getType() != S1->getType()) {
2078 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2079 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2080 }
2081 Value *S1S2 = IRB.CreateAnd(S1, S2);
2082 Value *V1S2 = IRB.CreateAnd(V1, S2);
2083 Value *S1V2 = IRB.CreateAnd(S1, V2);
2084 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2085 setOriginForNaryOp(I);
2086 }
2087
visitOr__anonb9e144640811::MemorySanitizerVisitor2088 void visitOr(BinaryOperator &I) {
2089 IRBuilder<> IRB(&I);
2090 // "Or" of 1 and a poisoned value results in unpoisoned value.
2091 // 1|1 => 1; 0|1 => 1; p|1 => 1;
2092 // 1|0 => 1; 0|0 => 0; p|0 => p;
2093 // 1|p => 1; 0|p => p; p|p => p;
2094 // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
2095 Value *S1 = getShadow(&I, 0);
2096 Value *S2 = getShadow(&I, 1);
2097 Value *V1 = IRB.CreateNot(I.getOperand(0));
2098 Value *V2 = IRB.CreateNot(I.getOperand(1));
2099 if (V1->getType() != S1->getType()) {
2100 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2101 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2102 }
2103 Value *S1S2 = IRB.CreateAnd(S1, S2);
2104 Value *V1S2 = IRB.CreateAnd(V1, S2);
2105 Value *S1V2 = IRB.CreateAnd(S1, V2);
2106 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2107 setOriginForNaryOp(I);
2108 }
2109
2110 /// Default propagation of shadow and/or origin.
2111 ///
2112 /// This class implements the general case of shadow propagation, used in all
2113 /// cases where we don't know and/or don't care about what the operation
2114 /// actually does. It converts all input shadow values to a common type
2115 /// (extending or truncating as necessary), and bitwise OR's them.
2116 ///
2117 /// This is much cheaper than inserting checks (i.e. requiring inputs to be
2118 /// fully initialized), and less prone to false positives.
2119 ///
2120 /// This class also implements the general case of origin propagation. For a
2121 /// Nary operation, result origin is set to the origin of an argument that is
2122 /// not entirely initialized. If there is more than one such arguments, the
2123 /// rightmost of them is picked. It does not matter which one is picked if all
2124 /// arguments are initialized.
2125 template <bool CombineShadow>
2126 class Combiner {
2127 Value *Shadow = nullptr;
2128 Value *Origin = nullptr;
2129 IRBuilder<> &IRB;
2130 MemorySanitizerVisitor *MSV;
2131
2132 public:
Combiner(MemorySanitizerVisitor * MSV,IRBuilder<> & IRB)2133 Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
2134 : IRB(IRB), MSV(MSV) {}
2135
2136 /// Add a pair of shadow and origin values to the mix.
Add(Value * OpShadow,Value * OpOrigin)2137 Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2138 if (CombineShadow) {
2139 assert(OpShadow);
2140 if (!Shadow)
2141 Shadow = OpShadow;
2142 else {
2143 OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2144 Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2145 }
2146 }
2147
2148 if (MSV->MS.TrackOrigins) {
2149 assert(OpOrigin);
2150 if (!Origin) {
2151 Origin = OpOrigin;
2152 } else {
2153 Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2154 // No point in adding something that might result in 0 origin value.
2155 if (!ConstOrigin || !ConstOrigin->isNullValue()) {
2156 Value *FlatShadow = MSV->convertShadowToScalar(OpShadow, IRB);
2157 Value *Cond =
2158 IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
2159 Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2160 }
2161 }
2162 }
2163 return *this;
2164 }
2165
2166 /// Add an application value to the mix.
Add(Value * V)2167 Combiner &Add(Value *V) {
2168 Value *OpShadow = MSV->getShadow(V);
2169 Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2170 return Add(OpShadow, OpOrigin);
2171 }
2172
2173 /// Set the current combined values as the given instruction's shadow
2174 /// and origin.
Done(Instruction * I)2175 void Done(Instruction *I) {
2176 if (CombineShadow) {
2177 assert(Shadow);
2178 Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2179 MSV->setShadow(I, Shadow);
2180 }
2181 if (MSV->MS.TrackOrigins) {
2182 assert(Origin);
2183 MSV->setOrigin(I, Origin);
2184 }
2185 }
2186 };
2187
2188 using ShadowAndOriginCombiner = Combiner<true>;
2189 using OriginCombiner = Combiner<false>;
2190
2191 /// Propagate origin for arbitrary operation.
setOriginForNaryOp__anonb9e144640811::MemorySanitizerVisitor2192 void setOriginForNaryOp(Instruction &I) {
2193 if (!MS.TrackOrigins) return;
2194 IRBuilder<> IRB(&I);
2195 OriginCombiner OC(this, IRB);
2196 for (Use &Op : I.operands())
2197 OC.Add(Op.get());
2198 OC.Done(&I);
2199 }
2200
VectorOrPrimitiveTypeSizeInBits__anonb9e144640811::MemorySanitizerVisitor2201 size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2202 assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2203 "Vector of pointers is not a valid shadow type");
2204 return Ty->isVectorTy() ? cast<FixedVectorType>(Ty)->getNumElements() *
2205 Ty->getScalarSizeInBits()
2206 : Ty->getPrimitiveSizeInBits();
2207 }
2208
2209 /// Cast between two shadow types, extending or truncating as
2210 /// necessary.
CreateShadowCast__anonb9e144640811::MemorySanitizerVisitor2211 Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2212 bool Signed = false) {
2213 Type *srcTy = V->getType();
2214 size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2215 size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2216 if (srcSizeInBits > 1 && dstSizeInBits == 1)
2217 return IRB.CreateICmpNE(V, getCleanShadow(V));
2218
2219 if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2220 return IRB.CreateIntCast(V, dstTy, Signed);
2221 if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2222 cast<FixedVectorType>(dstTy)->getNumElements() ==
2223 cast<FixedVectorType>(srcTy)->getNumElements())
2224 return IRB.CreateIntCast(V, dstTy, Signed);
2225 Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2226 Value *V2 =
2227 IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2228 return IRB.CreateBitCast(V2, dstTy);
2229 // TODO: handle struct types.
2230 }
2231
2232 /// Cast an application value to the type of its own shadow.
CreateAppToShadowCast__anonb9e144640811::MemorySanitizerVisitor2233 Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2234 Type *ShadowTy = getShadowTy(V);
2235 if (V->getType() == ShadowTy)
2236 return V;
2237 if (V->getType()->isPtrOrPtrVectorTy())
2238 return IRB.CreatePtrToInt(V, ShadowTy);
2239 else
2240 return IRB.CreateBitCast(V, ShadowTy);
2241 }
2242
2243 /// Propagate shadow for arbitrary operation.
handleShadowOr__anonb9e144640811::MemorySanitizerVisitor2244 void handleShadowOr(Instruction &I) {
2245 IRBuilder<> IRB(&I);
2246 ShadowAndOriginCombiner SC(this, IRB);
2247 for (Use &Op : I.operands())
2248 SC.Add(Op.get());
2249 SC.Done(&I);
2250 }
2251
visitFNeg__anonb9e144640811::MemorySanitizerVisitor2252 void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2253
2254 // Handle multiplication by constant.
2255 //
2256 // Handle a special case of multiplication by constant that may have one or
2257 // more zeros in the lower bits. This makes corresponding number of lower bits
2258 // of the result zero as well. We model it by shifting the other operand
2259 // shadow left by the required number of bits. Effectively, we transform
2260 // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2261 // We use multiplication by 2**N instead of shift to cover the case of
2262 // multiplication by 0, which may occur in some elements of a vector operand.
handleMulByConstant__anonb9e144640811::MemorySanitizerVisitor2263 void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2264 Value *OtherArg) {
2265 Constant *ShadowMul;
2266 Type *Ty = ConstArg->getType();
2267 if (auto *VTy = dyn_cast<VectorType>(Ty)) {
2268 unsigned NumElements = cast<FixedVectorType>(VTy)->getNumElements();
2269 Type *EltTy = VTy->getElementType();
2270 SmallVector<Constant *, 16> Elements;
2271 for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2272 if (ConstantInt *Elt =
2273 dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2274 const APInt &V = Elt->getValue();
2275 APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2276 Elements.push_back(ConstantInt::get(EltTy, V2));
2277 } else {
2278 Elements.push_back(ConstantInt::get(EltTy, 1));
2279 }
2280 }
2281 ShadowMul = ConstantVector::get(Elements);
2282 } else {
2283 if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2284 const APInt &V = Elt->getValue();
2285 APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2286 ShadowMul = ConstantInt::get(Ty, V2);
2287 } else {
2288 ShadowMul = ConstantInt::get(Ty, 1);
2289 }
2290 }
2291
2292 IRBuilder<> IRB(&I);
2293 setShadow(&I,
2294 IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2295 setOrigin(&I, getOrigin(OtherArg));
2296 }
2297
visitMul__anonb9e144640811::MemorySanitizerVisitor2298 void visitMul(BinaryOperator &I) {
2299 Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2300 Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2301 if (constOp0 && !constOp1)
2302 handleMulByConstant(I, constOp0, I.getOperand(1));
2303 else if (constOp1 && !constOp0)
2304 handleMulByConstant(I, constOp1, I.getOperand(0));
2305 else
2306 handleShadowOr(I);
2307 }
2308
visitFAdd__anonb9e144640811::MemorySanitizerVisitor2309 void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
visitFSub__anonb9e144640811::MemorySanitizerVisitor2310 void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
visitFMul__anonb9e144640811::MemorySanitizerVisitor2311 void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
visitAdd__anonb9e144640811::MemorySanitizerVisitor2312 void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
visitSub__anonb9e144640811::MemorySanitizerVisitor2313 void visitSub(BinaryOperator &I) { handleShadowOr(I); }
visitXor__anonb9e144640811::MemorySanitizerVisitor2314 void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2315
handleIntegerDiv__anonb9e144640811::MemorySanitizerVisitor2316 void handleIntegerDiv(Instruction &I) {
2317 IRBuilder<> IRB(&I);
2318 // Strict on the second argument.
2319 insertShadowCheck(I.getOperand(1), &I);
2320 setShadow(&I, getShadow(&I, 0));
2321 setOrigin(&I, getOrigin(&I, 0));
2322 }
2323
visitUDiv__anonb9e144640811::MemorySanitizerVisitor2324 void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
visitSDiv__anonb9e144640811::MemorySanitizerVisitor2325 void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
visitURem__anonb9e144640811::MemorySanitizerVisitor2326 void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
visitSRem__anonb9e144640811::MemorySanitizerVisitor2327 void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2328
2329 // Floating point division is side-effect free. We can not require that the
2330 // divisor is fully initialized and must propagate shadow. See PR37523.
visitFDiv__anonb9e144640811::MemorySanitizerVisitor2331 void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
visitFRem__anonb9e144640811::MemorySanitizerVisitor2332 void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2333
2334 /// Instrument == and != comparisons.
2335 ///
2336 /// Sometimes the comparison result is known even if some of the bits of the
2337 /// arguments are not.
handleEqualityComparison__anonb9e144640811::MemorySanitizerVisitor2338 void handleEqualityComparison(ICmpInst &I) {
2339 IRBuilder<> IRB(&I);
2340 Value *A = I.getOperand(0);
2341 Value *B = I.getOperand(1);
2342 Value *Sa = getShadow(A);
2343 Value *Sb = getShadow(B);
2344
2345 // Get rid of pointers and vectors of pointers.
2346 // For ints (and vectors of ints), types of A and Sa match,
2347 // and this is a no-op.
2348 A = IRB.CreatePointerCast(A, Sa->getType());
2349 B = IRB.CreatePointerCast(B, Sb->getType());
2350
2351 // A == B <==> (C = A^B) == 0
2352 // A != B <==> (C = A^B) != 0
2353 // Sc = Sa | Sb
2354 Value *C = IRB.CreateXor(A, B);
2355 Value *Sc = IRB.CreateOr(Sa, Sb);
2356 // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2357 // Result is defined if one of the following is true
2358 // * there is a defined 1 bit in C
2359 // * C is fully defined
2360 // Si = !(C & ~Sc) && Sc
2361 Value *Zero = Constant::getNullValue(Sc->getType());
2362 Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2363 Value *Si =
2364 IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero),
2365 IRB.CreateICmpEQ(
2366 IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero));
2367 Si->setName("_msprop_icmp");
2368 setShadow(&I, Si);
2369 setOriginForNaryOp(I);
2370 }
2371
2372 /// Build the lowest possible value of V, taking into account V's
2373 /// uninitialized bits.
getLowestPossibleValue__anonb9e144640811::MemorySanitizerVisitor2374 Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2375 bool isSigned) {
2376 if (isSigned) {
2377 // Split shadow into sign bit and other bits.
2378 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2379 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2380 // Maximise the undefined shadow bit, minimize other undefined bits.
2381 return
2382 IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit);
2383 } else {
2384 // Minimize undefined bits.
2385 return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2386 }
2387 }
2388
2389 /// Build the highest possible value of V, taking into account V's
2390 /// uninitialized bits.
getHighestPossibleValue__anonb9e144640811::MemorySanitizerVisitor2391 Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2392 bool isSigned) {
2393 if (isSigned) {
2394 // Split shadow into sign bit and other bits.
2395 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2396 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2397 // Minimise the undefined shadow bit, maximise other undefined bits.
2398 return
2399 IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits);
2400 } else {
2401 // Maximize undefined bits.
2402 return IRB.CreateOr(A, Sa);
2403 }
2404 }
2405
2406 /// Instrument relational comparisons.
2407 ///
2408 /// This function does exact shadow propagation for all relational
2409 /// comparisons of integers, pointers and vectors of those.
2410 /// FIXME: output seems suboptimal when one of the operands is a constant
handleRelationalComparisonExact__anonb9e144640811::MemorySanitizerVisitor2411 void handleRelationalComparisonExact(ICmpInst &I) {
2412 IRBuilder<> IRB(&I);
2413 Value *A = I.getOperand(0);
2414 Value *B = I.getOperand(1);
2415 Value *Sa = getShadow(A);
2416 Value *Sb = getShadow(B);
2417
2418 // Get rid of pointers and vectors of pointers.
2419 // For ints (and vectors of ints), types of A and Sa match,
2420 // and this is a no-op.
2421 A = IRB.CreatePointerCast(A, Sa->getType());
2422 B = IRB.CreatePointerCast(B, Sb->getType());
2423
2424 // Let [a0, a1] be the interval of possible values of A, taking into account
2425 // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2426 // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2427 bool IsSigned = I.isSigned();
2428 Value *S1 = IRB.CreateICmp(I.getPredicate(),
2429 getLowestPossibleValue(IRB, A, Sa, IsSigned),
2430 getHighestPossibleValue(IRB, B, Sb, IsSigned));
2431 Value *S2 = IRB.CreateICmp(I.getPredicate(),
2432 getHighestPossibleValue(IRB, A, Sa, IsSigned),
2433 getLowestPossibleValue(IRB, B, Sb, IsSigned));
2434 Value *Si = IRB.CreateXor(S1, S2);
2435 setShadow(&I, Si);
2436 setOriginForNaryOp(I);
2437 }
2438
2439 /// Instrument signed relational comparisons.
2440 ///
2441 /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2442 /// bit of the shadow. Everything else is delegated to handleShadowOr().
handleSignedRelationalComparison__anonb9e144640811::MemorySanitizerVisitor2443 void handleSignedRelationalComparison(ICmpInst &I) {
2444 Constant *constOp;
2445 Value *op = nullptr;
2446 CmpInst::Predicate pre;
2447 if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2448 op = I.getOperand(0);
2449 pre = I.getPredicate();
2450 } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2451 op = I.getOperand(1);
2452 pre = I.getSwappedPredicate();
2453 } else {
2454 handleShadowOr(I);
2455 return;
2456 }
2457
2458 if ((constOp->isNullValue() &&
2459 (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2460 (constOp->isAllOnesValue() &&
2461 (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2462 IRBuilder<> IRB(&I);
2463 Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2464 "_msprop_icmp_s");
2465 setShadow(&I, Shadow);
2466 setOrigin(&I, getOrigin(op));
2467 } else {
2468 handleShadowOr(I);
2469 }
2470 }
2471
visitICmpInst__anonb9e144640811::MemorySanitizerVisitor2472 void visitICmpInst(ICmpInst &I) {
2473 if (!ClHandleICmp) {
2474 handleShadowOr(I);
2475 return;
2476 }
2477 if (I.isEquality()) {
2478 handleEqualityComparison(I);
2479 return;
2480 }
2481
2482 assert(I.isRelational());
2483 if (ClHandleICmpExact) {
2484 handleRelationalComparisonExact(I);
2485 return;
2486 }
2487 if (I.isSigned()) {
2488 handleSignedRelationalComparison(I);
2489 return;
2490 }
2491
2492 assert(I.isUnsigned());
2493 if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2494 handleRelationalComparisonExact(I);
2495 return;
2496 }
2497
2498 handleShadowOr(I);
2499 }
2500
visitFCmpInst__anonb9e144640811::MemorySanitizerVisitor2501 void visitFCmpInst(FCmpInst &I) {
2502 handleShadowOr(I);
2503 }
2504
handleShift__anonb9e144640811::MemorySanitizerVisitor2505 void handleShift(BinaryOperator &I) {
2506 IRBuilder<> IRB(&I);
2507 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2508 // Otherwise perform the same shift on S1.
2509 Value *S1 = getShadow(&I, 0);
2510 Value *S2 = getShadow(&I, 1);
2511 Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)),
2512 S2->getType());
2513 Value *V2 = I.getOperand(1);
2514 Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2515 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2516 setOriginForNaryOp(I);
2517 }
2518
visitShl__anonb9e144640811::MemorySanitizerVisitor2519 void visitShl(BinaryOperator &I) { handleShift(I); }
visitAShr__anonb9e144640811::MemorySanitizerVisitor2520 void visitAShr(BinaryOperator &I) { handleShift(I); }
visitLShr__anonb9e144640811::MemorySanitizerVisitor2521 void visitLShr(BinaryOperator &I) { handleShift(I); }
2522
2523 /// Instrument llvm.memmove
2524 ///
2525 /// At this point we don't know if llvm.memmove will be inlined or not.
2526 /// If we don't instrument it and it gets inlined,
2527 /// our interceptor will not kick in and we will lose the memmove.
2528 /// If we instrument the call here, but it does not get inlined,
2529 /// we will memove the shadow twice: which is bad in case
2530 /// of overlapping regions. So, we simply lower the intrinsic to a call.
2531 ///
2532 /// Similar situation exists for memcpy and memset.
visitMemMoveInst__anonb9e144640811::MemorySanitizerVisitor2533 void visitMemMoveInst(MemMoveInst &I) {
2534 IRBuilder<> IRB(&I);
2535 IRB.CreateCall(
2536 MS.MemmoveFn,
2537 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2538 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2539 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2540 I.eraseFromParent();
2541 }
2542
2543 // Similar to memmove: avoid copying shadow twice.
2544 // This is somewhat unfortunate as it may slowdown small constant memcpys.
2545 // FIXME: consider doing manual inline for small constant sizes and proper
2546 // alignment.
visitMemCpyInst__anonb9e144640811::MemorySanitizerVisitor2547 void visitMemCpyInst(MemCpyInst &I) {
2548 IRBuilder<> IRB(&I);
2549 IRB.CreateCall(
2550 MS.MemcpyFn,
2551 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2552 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2553 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2554 I.eraseFromParent();
2555 }
2556
2557 // Same as memcpy.
visitMemSetInst__anonb9e144640811::MemorySanitizerVisitor2558 void visitMemSetInst(MemSetInst &I) {
2559 IRBuilder<> IRB(&I);
2560 IRB.CreateCall(
2561 MS.MemsetFn,
2562 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2563 IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2564 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2565 I.eraseFromParent();
2566 }
2567
visitVAStartInst__anonb9e144640811::MemorySanitizerVisitor2568 void visitVAStartInst(VAStartInst &I) {
2569 VAHelper->visitVAStartInst(I);
2570 }
2571
visitVACopyInst__anonb9e144640811::MemorySanitizerVisitor2572 void visitVACopyInst(VACopyInst &I) {
2573 VAHelper->visitVACopyInst(I);
2574 }
2575
2576 /// Handle vector store-like intrinsics.
2577 ///
2578 /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2579 /// has 1 pointer argument and 1 vector argument, returns void.
handleVectorStoreIntrinsic__anonb9e144640811::MemorySanitizerVisitor2580 bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2581 IRBuilder<> IRB(&I);
2582 Value* Addr = I.getArgOperand(0);
2583 Value *Shadow = getShadow(&I, 1);
2584 Value *ShadowPtr, *OriginPtr;
2585
2586 // We don't know the pointer alignment (could be unaligned SSE store!).
2587 // Have to assume to worst case.
2588 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2589 Addr, IRB, Shadow->getType(), Align(1), /*isStore*/ true);
2590 IRB.CreateAlignedStore(Shadow, ShadowPtr, Align(1));
2591
2592 if (ClCheckAccessAddress)
2593 insertShadowCheck(Addr, &I);
2594
2595 // FIXME: factor out common code from materializeStores
2596 if (MS.TrackOrigins) IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2597 return true;
2598 }
2599
2600 /// Handle vector load-like intrinsics.
2601 ///
2602 /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2603 /// has 1 pointer argument, returns a vector.
handleVectorLoadIntrinsic__anonb9e144640811::MemorySanitizerVisitor2604 bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2605 IRBuilder<> IRB(&I);
2606 Value *Addr = I.getArgOperand(0);
2607
2608 Type *ShadowTy = getShadowTy(&I);
2609 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2610 if (PropagateShadow) {
2611 // We don't know the pointer alignment (could be unaligned SSE load!).
2612 // Have to assume to worst case.
2613 const Align Alignment = Align(1);
2614 std::tie(ShadowPtr, OriginPtr) =
2615 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2616 setShadow(&I,
2617 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2618 } else {
2619 setShadow(&I, getCleanShadow(&I));
2620 }
2621
2622 if (ClCheckAccessAddress)
2623 insertShadowCheck(Addr, &I);
2624
2625 if (MS.TrackOrigins) {
2626 if (PropagateShadow)
2627 setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2628 else
2629 setOrigin(&I, getCleanOrigin());
2630 }
2631 return true;
2632 }
2633
2634 /// Handle (SIMD arithmetic)-like intrinsics.
2635 ///
2636 /// Instrument intrinsics with any number of arguments of the same type,
2637 /// equal to the return type. The type should be simple (no aggregates or
2638 /// pointers; vectors are fine).
2639 /// Caller guarantees that this intrinsic does not access memory.
maybeHandleSimpleNomemIntrinsic__anonb9e144640811::MemorySanitizerVisitor2640 bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2641 Type *RetTy = I.getType();
2642 if (!(RetTy->isIntOrIntVectorTy() ||
2643 RetTy->isFPOrFPVectorTy() ||
2644 RetTy->isX86_MMXTy()))
2645 return false;
2646
2647 unsigned NumArgOperands = I.getNumArgOperands();
2648 for (unsigned i = 0; i < NumArgOperands; ++i) {
2649 Type *Ty = I.getArgOperand(i)->getType();
2650 if (Ty != RetTy)
2651 return false;
2652 }
2653
2654 IRBuilder<> IRB(&I);
2655 ShadowAndOriginCombiner SC(this, IRB);
2656 for (unsigned i = 0; i < NumArgOperands; ++i)
2657 SC.Add(I.getArgOperand(i));
2658 SC.Done(&I);
2659
2660 return true;
2661 }
2662
2663 /// Heuristically instrument unknown intrinsics.
2664 ///
2665 /// The main purpose of this code is to do something reasonable with all
2666 /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2667 /// We recognize several classes of intrinsics by their argument types and
2668 /// ModRefBehaviour and apply special instrumentation when we are reasonably
2669 /// sure that we know what the intrinsic does.
2670 ///
2671 /// We special-case intrinsics where this approach fails. See llvm.bswap
2672 /// handling as an example of that.
handleUnknownIntrinsic__anonb9e144640811::MemorySanitizerVisitor2673 bool handleUnknownIntrinsic(IntrinsicInst &I) {
2674 unsigned NumArgOperands = I.getNumArgOperands();
2675 if (NumArgOperands == 0)
2676 return false;
2677
2678 if (NumArgOperands == 2 &&
2679 I.getArgOperand(0)->getType()->isPointerTy() &&
2680 I.getArgOperand(1)->getType()->isVectorTy() &&
2681 I.getType()->isVoidTy() &&
2682 !I.onlyReadsMemory()) {
2683 // This looks like a vector store.
2684 return handleVectorStoreIntrinsic(I);
2685 }
2686
2687 if (NumArgOperands == 1 &&
2688 I.getArgOperand(0)->getType()->isPointerTy() &&
2689 I.getType()->isVectorTy() &&
2690 I.onlyReadsMemory()) {
2691 // This looks like a vector load.
2692 return handleVectorLoadIntrinsic(I);
2693 }
2694
2695 if (I.doesNotAccessMemory())
2696 if (maybeHandleSimpleNomemIntrinsic(I))
2697 return true;
2698
2699 // FIXME: detect and handle SSE maskstore/maskload
2700 return false;
2701 }
2702
handleInvariantGroup__anonb9e144640811::MemorySanitizerVisitor2703 void handleInvariantGroup(IntrinsicInst &I) {
2704 setShadow(&I, getShadow(&I, 0));
2705 setOrigin(&I, getOrigin(&I, 0));
2706 }
2707
handleLifetimeStart__anonb9e144640811::MemorySanitizerVisitor2708 void handleLifetimeStart(IntrinsicInst &I) {
2709 if (!PoisonStack)
2710 return;
2711 AllocaInst *AI = llvm::findAllocaForValue(I.getArgOperand(1));
2712 if (!AI)
2713 InstrumentLifetimeStart = false;
2714 LifetimeStartList.push_back(std::make_pair(&I, AI));
2715 }
2716
handleBswap__anonb9e144640811::MemorySanitizerVisitor2717 void handleBswap(IntrinsicInst &I) {
2718 IRBuilder<> IRB(&I);
2719 Value *Op = I.getArgOperand(0);
2720 Type *OpType = Op->getType();
2721 Function *BswapFunc = Intrinsic::getDeclaration(
2722 F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1));
2723 setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
2724 setOrigin(&I, getOrigin(Op));
2725 }
2726
2727 // Instrument vector convert intrinsic.
2728 //
2729 // This function instruments intrinsics like cvtsi2ss:
2730 // %Out = int_xxx_cvtyyy(%ConvertOp)
2731 // or
2732 // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
2733 // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
2734 // number \p Out elements, and (if has 2 arguments) copies the rest of the
2735 // elements from \p CopyOp.
2736 // In most cases conversion involves floating-point value which may trigger a
2737 // hardware exception when not fully initialized. For this reason we require
2738 // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
2739 // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
2740 // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
2741 // return a fully initialized value.
handleVectorConvertIntrinsic__anonb9e144640811::MemorySanitizerVisitor2742 void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements,
2743 bool HasRoundingMode = false) {
2744 IRBuilder<> IRB(&I);
2745 Value *CopyOp, *ConvertOp;
2746
2747 assert((!HasRoundingMode ||
2748 isa<ConstantInt>(I.getArgOperand(I.getNumArgOperands() - 1))) &&
2749 "Invalid rounding mode");
2750
2751 switch (I.getNumArgOperands() - HasRoundingMode) {
2752 case 2:
2753 CopyOp = I.getArgOperand(0);
2754 ConvertOp = I.getArgOperand(1);
2755 break;
2756 case 1:
2757 ConvertOp = I.getArgOperand(0);
2758 CopyOp = nullptr;
2759 break;
2760 default:
2761 llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
2762 }
2763
2764 // The first *NumUsedElements* elements of ConvertOp are converted to the
2765 // same number of output elements. The rest of the output is copied from
2766 // CopyOp, or (if not available) filled with zeroes.
2767 // Combine shadow for elements of ConvertOp that are used in this operation,
2768 // and insert a check.
2769 // FIXME: consider propagating shadow of ConvertOp, at least in the case of
2770 // int->any conversion.
2771 Value *ConvertShadow = getShadow(ConvertOp);
2772 Value *AggShadow = nullptr;
2773 if (ConvertOp->getType()->isVectorTy()) {
2774 AggShadow = IRB.CreateExtractElement(
2775 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2776 for (int i = 1; i < NumUsedElements; ++i) {
2777 Value *MoreShadow = IRB.CreateExtractElement(
2778 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2779 AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
2780 }
2781 } else {
2782 AggShadow = ConvertShadow;
2783 }
2784 assert(AggShadow->getType()->isIntegerTy());
2785 insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
2786
2787 // Build result shadow by zero-filling parts of CopyOp shadow that come from
2788 // ConvertOp.
2789 if (CopyOp) {
2790 assert(CopyOp->getType() == I.getType());
2791 assert(CopyOp->getType()->isVectorTy());
2792 Value *ResultShadow = getShadow(CopyOp);
2793 Type *EltTy = cast<VectorType>(ResultShadow->getType())->getElementType();
2794 for (int i = 0; i < NumUsedElements; ++i) {
2795 ResultShadow = IRB.CreateInsertElement(
2796 ResultShadow, ConstantInt::getNullValue(EltTy),
2797 ConstantInt::get(IRB.getInt32Ty(), i));
2798 }
2799 setShadow(&I, ResultShadow);
2800 setOrigin(&I, getOrigin(CopyOp));
2801 } else {
2802 setShadow(&I, getCleanShadow(&I));
2803 setOrigin(&I, getCleanOrigin());
2804 }
2805 }
2806
2807 // Given a scalar or vector, extract lower 64 bits (or less), and return all
2808 // zeroes if it is zero, and all ones otherwise.
Lower64ShadowExtend__anonb9e144640811::MemorySanitizerVisitor2809 Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2810 if (S->getType()->isVectorTy())
2811 S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
2812 assert(S->getType()->getPrimitiveSizeInBits() <= 64);
2813 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2814 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2815 }
2816
2817 // Given a vector, extract its first element, and return all
2818 // zeroes if it is zero, and all ones otherwise.
LowerElementShadowExtend__anonb9e144640811::MemorySanitizerVisitor2819 Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2820 Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
2821 Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
2822 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2823 }
2824
VariableShadowExtend__anonb9e144640811::MemorySanitizerVisitor2825 Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
2826 Type *T = S->getType();
2827 assert(T->isVectorTy());
2828 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2829 return IRB.CreateSExt(S2, T);
2830 }
2831
2832 // Instrument vector shift intrinsic.
2833 //
2834 // This function instruments intrinsics like int_x86_avx2_psll_w.
2835 // Intrinsic shifts %In by %ShiftSize bits.
2836 // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
2837 // size, and the rest is ignored. Behavior is defined even if shift size is
2838 // greater than register (or field) width.
handleVectorShiftIntrinsic__anonb9e144640811::MemorySanitizerVisitor2839 void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
2840 assert(I.getNumArgOperands() == 2);
2841 IRBuilder<> IRB(&I);
2842 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2843 // Otherwise perform the same shift on S1.
2844 Value *S1 = getShadow(&I, 0);
2845 Value *S2 = getShadow(&I, 1);
2846 Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
2847 : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
2848 Value *V1 = I.getOperand(0);
2849 Value *V2 = I.getOperand(1);
2850 Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledOperand(),
2851 {IRB.CreateBitCast(S1, V1->getType()), V2});
2852 Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
2853 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2854 setOriginForNaryOp(I);
2855 }
2856
2857 // Get an X86_MMX-sized vector type.
getMMXVectorTy__anonb9e144640811::MemorySanitizerVisitor2858 Type *getMMXVectorTy(unsigned EltSizeInBits) {
2859 const unsigned X86_MMXSizeInBits = 64;
2860 assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
2861 "Illegal MMX vector element size");
2862 return FixedVectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
2863 X86_MMXSizeInBits / EltSizeInBits);
2864 }
2865
2866 // Returns a signed counterpart for an (un)signed-saturate-and-pack
2867 // intrinsic.
getSignedPackIntrinsic__anonb9e144640811::MemorySanitizerVisitor2868 Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
2869 switch (id) {
2870 case Intrinsic::x86_sse2_packsswb_128:
2871 case Intrinsic::x86_sse2_packuswb_128:
2872 return Intrinsic::x86_sse2_packsswb_128;
2873
2874 case Intrinsic::x86_sse2_packssdw_128:
2875 case Intrinsic::x86_sse41_packusdw:
2876 return Intrinsic::x86_sse2_packssdw_128;
2877
2878 case Intrinsic::x86_avx2_packsswb:
2879 case Intrinsic::x86_avx2_packuswb:
2880 return Intrinsic::x86_avx2_packsswb;
2881
2882 case Intrinsic::x86_avx2_packssdw:
2883 case Intrinsic::x86_avx2_packusdw:
2884 return Intrinsic::x86_avx2_packssdw;
2885
2886 case Intrinsic::x86_mmx_packsswb:
2887 case Intrinsic::x86_mmx_packuswb:
2888 return Intrinsic::x86_mmx_packsswb;
2889
2890 case Intrinsic::x86_mmx_packssdw:
2891 return Intrinsic::x86_mmx_packssdw;
2892 default:
2893 llvm_unreachable("unexpected intrinsic id");
2894 }
2895 }
2896
2897 // Instrument vector pack intrinsic.
2898 //
2899 // This function instruments intrinsics like x86_mmx_packsswb, that
2900 // packs elements of 2 input vectors into half as many bits with saturation.
2901 // Shadow is propagated with the signed variant of the same intrinsic applied
2902 // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
2903 // EltSizeInBits is used only for x86mmx arguments.
handleVectorPackIntrinsic__anonb9e144640811::MemorySanitizerVisitor2904 void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
2905 assert(I.getNumArgOperands() == 2);
2906 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2907 IRBuilder<> IRB(&I);
2908 Value *S1 = getShadow(&I, 0);
2909 Value *S2 = getShadow(&I, 1);
2910 assert(isX86_MMX || S1->getType()->isVectorTy());
2911
2912 // SExt and ICmpNE below must apply to individual elements of input vectors.
2913 // In case of x86mmx arguments, cast them to appropriate vector types and
2914 // back.
2915 Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
2916 if (isX86_MMX) {
2917 S1 = IRB.CreateBitCast(S1, T);
2918 S2 = IRB.CreateBitCast(S2, T);
2919 }
2920 Value *S1_ext = IRB.CreateSExt(
2921 IRB.CreateICmpNE(S1, Constant::getNullValue(T)), T);
2922 Value *S2_ext = IRB.CreateSExt(
2923 IRB.CreateICmpNE(S2, Constant::getNullValue(T)), T);
2924 if (isX86_MMX) {
2925 Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
2926 S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
2927 S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
2928 }
2929
2930 Function *ShadowFn = Intrinsic::getDeclaration(
2931 F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
2932
2933 Value *S =
2934 IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
2935 if (isX86_MMX) S = IRB.CreateBitCast(S, getShadowTy(&I));
2936 setShadow(&I, S);
2937 setOriginForNaryOp(I);
2938 }
2939
2940 // Instrument sum-of-absolute-differences intrinsic.
handleVectorSadIntrinsic__anonb9e144640811::MemorySanitizerVisitor2941 void handleVectorSadIntrinsic(IntrinsicInst &I) {
2942 const unsigned SignificantBitsPerResultElement = 16;
2943 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2944 Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
2945 unsigned ZeroBitsPerResultElement =
2946 ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
2947
2948 IRBuilder<> IRB(&I);
2949 Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2950 S = IRB.CreateBitCast(S, ResTy);
2951 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2952 ResTy);
2953 S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
2954 S = IRB.CreateBitCast(S, getShadowTy(&I));
2955 setShadow(&I, S);
2956 setOriginForNaryOp(I);
2957 }
2958
2959 // Instrument multiply-add intrinsic.
handleVectorPmaddIntrinsic__anonb9e144640811::MemorySanitizerVisitor2960 void handleVectorPmaddIntrinsic(IntrinsicInst &I,
2961 unsigned EltSizeInBits = 0) {
2962 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2963 Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
2964 IRBuilder<> IRB(&I);
2965 Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2966 S = IRB.CreateBitCast(S, ResTy);
2967 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2968 ResTy);
2969 S = IRB.CreateBitCast(S, getShadowTy(&I));
2970 setShadow(&I, S);
2971 setOriginForNaryOp(I);
2972 }
2973
2974 // Instrument compare-packed intrinsic.
2975 // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
2976 // all-ones shadow.
handleVectorComparePackedIntrinsic__anonb9e144640811::MemorySanitizerVisitor2977 void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
2978 IRBuilder<> IRB(&I);
2979 Type *ResTy = getShadowTy(&I);
2980 Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2981 Value *S = IRB.CreateSExt(
2982 IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
2983 setShadow(&I, S);
2984 setOriginForNaryOp(I);
2985 }
2986
2987 // Instrument compare-scalar intrinsic.
2988 // This handles both cmp* intrinsics which return the result in the first
2989 // element of a vector, and comi* which return the result as i32.
handleVectorCompareScalarIntrinsic__anonb9e144640811::MemorySanitizerVisitor2990 void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
2991 IRBuilder<> IRB(&I);
2992 Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2993 Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
2994 setShadow(&I, S);
2995 setOriginForNaryOp(I);
2996 }
2997
2998 // Instrument generic vector reduction intrinsics
2999 // by ORing together all their fields.
handleVectorReduceIntrinsic__anonb9e144640811::MemorySanitizerVisitor3000 void handleVectorReduceIntrinsic(IntrinsicInst &I) {
3001 IRBuilder<> IRB(&I);
3002 Value *S = IRB.CreateOrReduce(getShadow(&I, 0));
3003 setShadow(&I, S);
3004 setOrigin(&I, getOrigin(&I, 0));
3005 }
3006
3007 // Instrument vector.reduce.or intrinsic.
3008 // Valid (non-poisoned) set bits in the operand pull low the
3009 // corresponding shadow bits.
handleVectorReduceOrIntrinsic__anonb9e144640811::MemorySanitizerVisitor3010 void handleVectorReduceOrIntrinsic(IntrinsicInst &I) {
3011 IRBuilder<> IRB(&I);
3012 Value *OperandShadow = getShadow(&I, 0);
3013 Value *OperandUnsetBits = IRB.CreateNot(I.getOperand(0));
3014 Value *OperandUnsetOrPoison = IRB.CreateOr(OperandUnsetBits, OperandShadow);
3015 // Bit N is clean if any field's bit N is 1 and unpoison
3016 Value *OutShadowMask = IRB.CreateAndReduce(OperandUnsetOrPoison);
3017 // Otherwise, it is clean if every field's bit N is unpoison
3018 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3019 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3020
3021 setShadow(&I, S);
3022 setOrigin(&I, getOrigin(&I, 0));
3023 }
3024
3025 // Instrument vector.reduce.and intrinsic.
3026 // Valid (non-poisoned) unset bits in the operand pull down the
3027 // corresponding shadow bits.
handleVectorReduceAndIntrinsic__anonb9e144640811::MemorySanitizerVisitor3028 void handleVectorReduceAndIntrinsic(IntrinsicInst &I) {
3029 IRBuilder<> IRB(&I);
3030 Value *OperandShadow = getShadow(&I, 0);
3031 Value *OperandSetOrPoison = IRB.CreateOr(I.getOperand(0), OperandShadow);
3032 // Bit N is clean if any field's bit N is 0 and unpoison
3033 Value *OutShadowMask = IRB.CreateAndReduce(OperandSetOrPoison);
3034 // Otherwise, it is clean if every field's bit N is unpoison
3035 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3036 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3037
3038 setShadow(&I, S);
3039 setOrigin(&I, getOrigin(&I, 0));
3040 }
3041
handleStmxcsr__anonb9e144640811::MemorySanitizerVisitor3042 void handleStmxcsr(IntrinsicInst &I) {
3043 IRBuilder<> IRB(&I);
3044 Value* Addr = I.getArgOperand(0);
3045 Type *Ty = IRB.getInt32Ty();
3046 Value *ShadowPtr =
3047 getShadowOriginPtr(Addr, IRB, Ty, Align(1), /*isStore*/ true).first;
3048
3049 IRB.CreateStore(getCleanShadow(Ty),
3050 IRB.CreatePointerCast(ShadowPtr, Ty->getPointerTo()));
3051
3052 if (ClCheckAccessAddress)
3053 insertShadowCheck(Addr, &I);
3054 }
3055
handleLdmxcsr__anonb9e144640811::MemorySanitizerVisitor3056 void handleLdmxcsr(IntrinsicInst &I) {
3057 if (!InsertChecks) return;
3058
3059 IRBuilder<> IRB(&I);
3060 Value *Addr = I.getArgOperand(0);
3061 Type *Ty = IRB.getInt32Ty();
3062 const Align Alignment = Align(1);
3063 Value *ShadowPtr, *OriginPtr;
3064 std::tie(ShadowPtr, OriginPtr) =
3065 getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
3066
3067 if (ClCheckAccessAddress)
3068 insertShadowCheck(Addr, &I);
3069
3070 Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
3071 Value *Origin = MS.TrackOrigins ? IRB.CreateLoad(MS.OriginTy, OriginPtr)
3072 : getCleanOrigin();
3073 insertShadowCheck(Shadow, Origin, &I);
3074 }
3075
handleMaskedStore__anonb9e144640811::MemorySanitizerVisitor3076 void handleMaskedStore(IntrinsicInst &I) {
3077 IRBuilder<> IRB(&I);
3078 Value *V = I.getArgOperand(0);
3079 Value *Addr = I.getArgOperand(1);
3080 const Align Alignment(
3081 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3082 Value *Mask = I.getArgOperand(3);
3083 Value *Shadow = getShadow(V);
3084
3085 Value *ShadowPtr;
3086 Value *OriginPtr;
3087 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
3088 Addr, IRB, Shadow->getType(), Alignment, /*isStore*/ true);
3089
3090 if (ClCheckAccessAddress) {
3091 insertShadowCheck(Addr, &I);
3092 // Uninitialized mask is kind of like uninitialized address, but not as
3093 // scary.
3094 insertShadowCheck(Mask, &I);
3095 }
3096
3097 IRB.CreateMaskedStore(Shadow, ShadowPtr, Alignment, Mask);
3098
3099 if (MS.TrackOrigins) {
3100 auto &DL = F.getParent()->getDataLayout();
3101 paintOrigin(IRB, getOrigin(V), OriginPtr,
3102 DL.getTypeStoreSize(Shadow->getType()),
3103 std::max(Alignment, kMinOriginAlignment));
3104 }
3105 }
3106
handleMaskedLoad__anonb9e144640811::MemorySanitizerVisitor3107 bool handleMaskedLoad(IntrinsicInst &I) {
3108 IRBuilder<> IRB(&I);
3109 Value *Addr = I.getArgOperand(0);
3110 const Align Alignment(
3111 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3112 Value *Mask = I.getArgOperand(2);
3113 Value *PassThru = I.getArgOperand(3);
3114
3115 Type *ShadowTy = getShadowTy(&I);
3116 Value *ShadowPtr, *OriginPtr;
3117 if (PropagateShadow) {
3118 std::tie(ShadowPtr, OriginPtr) =
3119 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
3120 setShadow(&I, IRB.CreateMaskedLoad(ShadowPtr, Alignment, Mask,
3121 getShadow(PassThru), "_msmaskedld"));
3122 } else {
3123 setShadow(&I, getCleanShadow(&I));
3124 }
3125
3126 if (ClCheckAccessAddress) {
3127 insertShadowCheck(Addr, &I);
3128 insertShadowCheck(Mask, &I);
3129 }
3130
3131 if (MS.TrackOrigins) {
3132 if (PropagateShadow) {
3133 // Choose between PassThru's and the loaded value's origins.
3134 Value *MaskedPassThruShadow = IRB.CreateAnd(
3135 getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
3136
3137 Value *Acc = IRB.CreateExtractElement(
3138 MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
3139 for (int i = 1, N = cast<FixedVectorType>(PassThru->getType())
3140 ->getNumElements();
3141 i < N; ++i) {
3142 Value *More = IRB.CreateExtractElement(
3143 MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), i));
3144 Acc = IRB.CreateOr(Acc, More);
3145 }
3146
3147 Value *Origin = IRB.CreateSelect(
3148 IRB.CreateICmpNE(Acc, Constant::getNullValue(Acc->getType())),
3149 getOrigin(PassThru), IRB.CreateLoad(MS.OriginTy, OriginPtr));
3150
3151 setOrigin(&I, Origin);
3152 } else {
3153 setOrigin(&I, getCleanOrigin());
3154 }
3155 }
3156 return true;
3157 }
3158
3159 // Instrument BMI / BMI2 intrinsics.
3160 // All of these intrinsics are Z = I(X, Y)
3161 // where the types of all operands and the result match, and are either i32 or i64.
3162 // The following instrumentation happens to work for all of them:
3163 // Sz = I(Sx, Y) | (sext (Sy != 0))
handleBmiIntrinsic__anonb9e144640811::MemorySanitizerVisitor3164 void handleBmiIntrinsic(IntrinsicInst &I) {
3165 IRBuilder<> IRB(&I);
3166 Type *ShadowTy = getShadowTy(&I);
3167
3168 // If any bit of the mask operand is poisoned, then the whole thing is.
3169 Value *SMask = getShadow(&I, 1);
3170 SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
3171 ShadowTy);
3172 // Apply the same intrinsic to the shadow of the first operand.
3173 Value *S = IRB.CreateCall(I.getCalledFunction(),
3174 {getShadow(&I, 0), I.getOperand(1)});
3175 S = IRB.CreateOr(SMask, S);
3176 setShadow(&I, S);
3177 setOriginForNaryOp(I);
3178 }
3179
getPclmulMask__anonb9e144640811::MemorySanitizerVisitor3180 SmallVector<int, 8> getPclmulMask(unsigned Width, bool OddElements) {
3181 SmallVector<int, 8> Mask;
3182 for (unsigned X = OddElements ? 1 : 0; X < Width; X += 2) {
3183 Mask.append(2, X);
3184 }
3185 return Mask;
3186 }
3187
3188 // Instrument pclmul intrinsics.
3189 // These intrinsics operate either on odd or on even elements of the input
3190 // vectors, depending on the constant in the 3rd argument, ignoring the rest.
3191 // Replace the unused elements with copies of the used ones, ex:
3192 // (0, 1, 2, 3) -> (0, 0, 2, 2) (even case)
3193 // or
3194 // (0, 1, 2, 3) -> (1, 1, 3, 3) (odd case)
3195 // and then apply the usual shadow combining logic.
handlePclmulIntrinsic__anonb9e144640811::MemorySanitizerVisitor3196 void handlePclmulIntrinsic(IntrinsicInst &I) {
3197 IRBuilder<> IRB(&I);
3198 unsigned Width =
3199 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3200 assert(isa<ConstantInt>(I.getArgOperand(2)) &&
3201 "pclmul 3rd operand must be a constant");
3202 unsigned Imm = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3203 Value *Shuf0 = IRB.CreateShuffleVector(getShadow(&I, 0),
3204 getPclmulMask(Width, Imm & 0x01));
3205 Value *Shuf1 = IRB.CreateShuffleVector(getShadow(&I, 1),
3206 getPclmulMask(Width, Imm & 0x10));
3207 ShadowAndOriginCombiner SOC(this, IRB);
3208 SOC.Add(Shuf0, getOrigin(&I, 0));
3209 SOC.Add(Shuf1, getOrigin(&I, 1));
3210 SOC.Done(&I);
3211 }
3212
3213 // Instrument _mm_*_sd intrinsics
handleUnarySdIntrinsic__anonb9e144640811::MemorySanitizerVisitor3214 void handleUnarySdIntrinsic(IntrinsicInst &I) {
3215 IRBuilder<> IRB(&I);
3216 Value *First = getShadow(&I, 0);
3217 Value *Second = getShadow(&I, 1);
3218 // High word of first operand, low word of second
3219 Value *Shadow =
3220 IRB.CreateShuffleVector(First, Second, llvm::makeArrayRef<int>({2, 1}));
3221
3222 setShadow(&I, Shadow);
3223 setOriginForNaryOp(I);
3224 }
3225
handleBinarySdIntrinsic__anonb9e144640811::MemorySanitizerVisitor3226 void handleBinarySdIntrinsic(IntrinsicInst &I) {
3227 IRBuilder<> IRB(&I);
3228 Value *First = getShadow(&I, 0);
3229 Value *Second = getShadow(&I, 1);
3230 Value *OrShadow = IRB.CreateOr(First, Second);
3231 // High word of first operand, low word of both OR'd together
3232 Value *Shadow = IRB.CreateShuffleVector(First, OrShadow,
3233 llvm::makeArrayRef<int>({2, 1}));
3234
3235 setShadow(&I, Shadow);
3236 setOriginForNaryOp(I);
3237 }
3238
3239 // Instrument abs intrinsic.
3240 // handleUnknownIntrinsic can't handle it because of the last
3241 // is_int_min_poison argument which does not match the result type.
handleAbsIntrinsic__anonb9e144640811::MemorySanitizerVisitor3242 void handleAbsIntrinsic(IntrinsicInst &I) {
3243 assert(I.getType()->isIntOrIntVectorTy());
3244 assert(I.getArgOperand(0)->getType() == I.getType());
3245
3246 // FIXME: Handle is_int_min_poison.
3247 IRBuilder<> IRB(&I);
3248 setShadow(&I, getShadow(&I, 0));
3249 setOrigin(&I, getOrigin(&I, 0));
3250 }
3251
visitIntrinsicInst__anonb9e144640811::MemorySanitizerVisitor3252 void visitIntrinsicInst(IntrinsicInst &I) {
3253 switch (I.getIntrinsicID()) {
3254 case Intrinsic::abs:
3255 handleAbsIntrinsic(I);
3256 break;
3257 case Intrinsic::lifetime_start:
3258 handleLifetimeStart(I);
3259 break;
3260 case Intrinsic::launder_invariant_group:
3261 case Intrinsic::strip_invariant_group:
3262 handleInvariantGroup(I);
3263 break;
3264 case Intrinsic::bswap:
3265 handleBswap(I);
3266 break;
3267 case Intrinsic::masked_store:
3268 handleMaskedStore(I);
3269 break;
3270 case Intrinsic::masked_load:
3271 handleMaskedLoad(I);
3272 break;
3273 case Intrinsic::vector_reduce_and:
3274 handleVectorReduceAndIntrinsic(I);
3275 break;
3276 case Intrinsic::vector_reduce_or:
3277 handleVectorReduceOrIntrinsic(I);
3278 break;
3279 case Intrinsic::vector_reduce_add:
3280 case Intrinsic::vector_reduce_xor:
3281 case Intrinsic::vector_reduce_mul:
3282 handleVectorReduceIntrinsic(I);
3283 break;
3284 case Intrinsic::x86_sse_stmxcsr:
3285 handleStmxcsr(I);
3286 break;
3287 case Intrinsic::x86_sse_ldmxcsr:
3288 handleLdmxcsr(I);
3289 break;
3290 case Intrinsic::x86_avx512_vcvtsd2usi64:
3291 case Intrinsic::x86_avx512_vcvtsd2usi32:
3292 case Intrinsic::x86_avx512_vcvtss2usi64:
3293 case Intrinsic::x86_avx512_vcvtss2usi32:
3294 case Intrinsic::x86_avx512_cvttss2usi64:
3295 case Intrinsic::x86_avx512_cvttss2usi:
3296 case Intrinsic::x86_avx512_cvttsd2usi64:
3297 case Intrinsic::x86_avx512_cvttsd2usi:
3298 case Intrinsic::x86_avx512_cvtusi2ss:
3299 case Intrinsic::x86_avx512_cvtusi642sd:
3300 case Intrinsic::x86_avx512_cvtusi642ss:
3301 handleVectorConvertIntrinsic(I, 1, true);
3302 break;
3303 case Intrinsic::x86_sse2_cvtsd2si64:
3304 case Intrinsic::x86_sse2_cvtsd2si:
3305 case Intrinsic::x86_sse2_cvtsd2ss:
3306 case Intrinsic::x86_sse2_cvttsd2si64:
3307 case Intrinsic::x86_sse2_cvttsd2si:
3308 case Intrinsic::x86_sse_cvtss2si64:
3309 case Intrinsic::x86_sse_cvtss2si:
3310 case Intrinsic::x86_sse_cvttss2si64:
3311 case Intrinsic::x86_sse_cvttss2si:
3312 handleVectorConvertIntrinsic(I, 1);
3313 break;
3314 case Intrinsic::x86_sse_cvtps2pi:
3315 case Intrinsic::x86_sse_cvttps2pi:
3316 handleVectorConvertIntrinsic(I, 2);
3317 break;
3318
3319 case Intrinsic::x86_avx512_psll_w_512:
3320 case Intrinsic::x86_avx512_psll_d_512:
3321 case Intrinsic::x86_avx512_psll_q_512:
3322 case Intrinsic::x86_avx512_pslli_w_512:
3323 case Intrinsic::x86_avx512_pslli_d_512:
3324 case Intrinsic::x86_avx512_pslli_q_512:
3325 case Intrinsic::x86_avx512_psrl_w_512:
3326 case Intrinsic::x86_avx512_psrl_d_512:
3327 case Intrinsic::x86_avx512_psrl_q_512:
3328 case Intrinsic::x86_avx512_psra_w_512:
3329 case Intrinsic::x86_avx512_psra_d_512:
3330 case Intrinsic::x86_avx512_psra_q_512:
3331 case Intrinsic::x86_avx512_psrli_w_512:
3332 case Intrinsic::x86_avx512_psrli_d_512:
3333 case Intrinsic::x86_avx512_psrli_q_512:
3334 case Intrinsic::x86_avx512_psrai_w_512:
3335 case Intrinsic::x86_avx512_psrai_d_512:
3336 case Intrinsic::x86_avx512_psrai_q_512:
3337 case Intrinsic::x86_avx512_psra_q_256:
3338 case Intrinsic::x86_avx512_psra_q_128:
3339 case Intrinsic::x86_avx512_psrai_q_256:
3340 case Intrinsic::x86_avx512_psrai_q_128:
3341 case Intrinsic::x86_avx2_psll_w:
3342 case Intrinsic::x86_avx2_psll_d:
3343 case Intrinsic::x86_avx2_psll_q:
3344 case Intrinsic::x86_avx2_pslli_w:
3345 case Intrinsic::x86_avx2_pslli_d:
3346 case Intrinsic::x86_avx2_pslli_q:
3347 case Intrinsic::x86_avx2_psrl_w:
3348 case Intrinsic::x86_avx2_psrl_d:
3349 case Intrinsic::x86_avx2_psrl_q:
3350 case Intrinsic::x86_avx2_psra_w:
3351 case Intrinsic::x86_avx2_psra_d:
3352 case Intrinsic::x86_avx2_psrli_w:
3353 case Intrinsic::x86_avx2_psrli_d:
3354 case Intrinsic::x86_avx2_psrli_q:
3355 case Intrinsic::x86_avx2_psrai_w:
3356 case Intrinsic::x86_avx2_psrai_d:
3357 case Intrinsic::x86_sse2_psll_w:
3358 case Intrinsic::x86_sse2_psll_d:
3359 case Intrinsic::x86_sse2_psll_q:
3360 case Intrinsic::x86_sse2_pslli_w:
3361 case Intrinsic::x86_sse2_pslli_d:
3362 case Intrinsic::x86_sse2_pslli_q:
3363 case Intrinsic::x86_sse2_psrl_w:
3364 case Intrinsic::x86_sse2_psrl_d:
3365 case Intrinsic::x86_sse2_psrl_q:
3366 case Intrinsic::x86_sse2_psra_w:
3367 case Intrinsic::x86_sse2_psra_d:
3368 case Intrinsic::x86_sse2_psrli_w:
3369 case Intrinsic::x86_sse2_psrli_d:
3370 case Intrinsic::x86_sse2_psrli_q:
3371 case Intrinsic::x86_sse2_psrai_w:
3372 case Intrinsic::x86_sse2_psrai_d:
3373 case Intrinsic::x86_mmx_psll_w:
3374 case Intrinsic::x86_mmx_psll_d:
3375 case Intrinsic::x86_mmx_psll_q:
3376 case Intrinsic::x86_mmx_pslli_w:
3377 case Intrinsic::x86_mmx_pslli_d:
3378 case Intrinsic::x86_mmx_pslli_q:
3379 case Intrinsic::x86_mmx_psrl_w:
3380 case Intrinsic::x86_mmx_psrl_d:
3381 case Intrinsic::x86_mmx_psrl_q:
3382 case Intrinsic::x86_mmx_psra_w:
3383 case Intrinsic::x86_mmx_psra_d:
3384 case Intrinsic::x86_mmx_psrli_w:
3385 case Intrinsic::x86_mmx_psrli_d:
3386 case Intrinsic::x86_mmx_psrli_q:
3387 case Intrinsic::x86_mmx_psrai_w:
3388 case Intrinsic::x86_mmx_psrai_d:
3389 handleVectorShiftIntrinsic(I, /* Variable */ false);
3390 break;
3391 case Intrinsic::x86_avx2_psllv_d:
3392 case Intrinsic::x86_avx2_psllv_d_256:
3393 case Intrinsic::x86_avx512_psllv_d_512:
3394 case Intrinsic::x86_avx2_psllv_q:
3395 case Intrinsic::x86_avx2_psllv_q_256:
3396 case Intrinsic::x86_avx512_psllv_q_512:
3397 case Intrinsic::x86_avx2_psrlv_d:
3398 case Intrinsic::x86_avx2_psrlv_d_256:
3399 case Intrinsic::x86_avx512_psrlv_d_512:
3400 case Intrinsic::x86_avx2_psrlv_q:
3401 case Intrinsic::x86_avx2_psrlv_q_256:
3402 case Intrinsic::x86_avx512_psrlv_q_512:
3403 case Intrinsic::x86_avx2_psrav_d:
3404 case Intrinsic::x86_avx2_psrav_d_256:
3405 case Intrinsic::x86_avx512_psrav_d_512:
3406 case Intrinsic::x86_avx512_psrav_q_128:
3407 case Intrinsic::x86_avx512_psrav_q_256:
3408 case Intrinsic::x86_avx512_psrav_q_512:
3409 handleVectorShiftIntrinsic(I, /* Variable */ true);
3410 break;
3411
3412 case Intrinsic::x86_sse2_packsswb_128:
3413 case Intrinsic::x86_sse2_packssdw_128:
3414 case Intrinsic::x86_sse2_packuswb_128:
3415 case Intrinsic::x86_sse41_packusdw:
3416 case Intrinsic::x86_avx2_packsswb:
3417 case Intrinsic::x86_avx2_packssdw:
3418 case Intrinsic::x86_avx2_packuswb:
3419 case Intrinsic::x86_avx2_packusdw:
3420 handleVectorPackIntrinsic(I);
3421 break;
3422
3423 case Intrinsic::x86_mmx_packsswb:
3424 case Intrinsic::x86_mmx_packuswb:
3425 handleVectorPackIntrinsic(I, 16);
3426 break;
3427
3428 case Intrinsic::x86_mmx_packssdw:
3429 handleVectorPackIntrinsic(I, 32);
3430 break;
3431
3432 case Intrinsic::x86_mmx_psad_bw:
3433 case Intrinsic::x86_sse2_psad_bw:
3434 case Intrinsic::x86_avx2_psad_bw:
3435 handleVectorSadIntrinsic(I);
3436 break;
3437
3438 case Intrinsic::x86_sse2_pmadd_wd:
3439 case Intrinsic::x86_avx2_pmadd_wd:
3440 case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3441 case Intrinsic::x86_avx2_pmadd_ub_sw:
3442 handleVectorPmaddIntrinsic(I);
3443 break;
3444
3445 case Intrinsic::x86_ssse3_pmadd_ub_sw:
3446 handleVectorPmaddIntrinsic(I, 8);
3447 break;
3448
3449 case Intrinsic::x86_mmx_pmadd_wd:
3450 handleVectorPmaddIntrinsic(I, 16);
3451 break;
3452
3453 case Intrinsic::x86_sse_cmp_ss:
3454 case Intrinsic::x86_sse2_cmp_sd:
3455 case Intrinsic::x86_sse_comieq_ss:
3456 case Intrinsic::x86_sse_comilt_ss:
3457 case Intrinsic::x86_sse_comile_ss:
3458 case Intrinsic::x86_sse_comigt_ss:
3459 case Intrinsic::x86_sse_comige_ss:
3460 case Intrinsic::x86_sse_comineq_ss:
3461 case Intrinsic::x86_sse_ucomieq_ss:
3462 case Intrinsic::x86_sse_ucomilt_ss:
3463 case Intrinsic::x86_sse_ucomile_ss:
3464 case Intrinsic::x86_sse_ucomigt_ss:
3465 case Intrinsic::x86_sse_ucomige_ss:
3466 case Intrinsic::x86_sse_ucomineq_ss:
3467 case Intrinsic::x86_sse2_comieq_sd:
3468 case Intrinsic::x86_sse2_comilt_sd:
3469 case Intrinsic::x86_sse2_comile_sd:
3470 case Intrinsic::x86_sse2_comigt_sd:
3471 case Intrinsic::x86_sse2_comige_sd:
3472 case Intrinsic::x86_sse2_comineq_sd:
3473 case Intrinsic::x86_sse2_ucomieq_sd:
3474 case Intrinsic::x86_sse2_ucomilt_sd:
3475 case Intrinsic::x86_sse2_ucomile_sd:
3476 case Intrinsic::x86_sse2_ucomigt_sd:
3477 case Intrinsic::x86_sse2_ucomige_sd:
3478 case Intrinsic::x86_sse2_ucomineq_sd:
3479 handleVectorCompareScalarIntrinsic(I);
3480 break;
3481
3482 case Intrinsic::x86_sse_cmp_ps:
3483 case Intrinsic::x86_sse2_cmp_pd:
3484 // FIXME: For x86_avx_cmp_pd_256 and x86_avx_cmp_ps_256 this function
3485 // generates reasonably looking IR that fails in the backend with "Do not
3486 // know how to split the result of this operator!".
3487 handleVectorComparePackedIntrinsic(I);
3488 break;
3489
3490 case Intrinsic::x86_bmi_bextr_32:
3491 case Intrinsic::x86_bmi_bextr_64:
3492 case Intrinsic::x86_bmi_bzhi_32:
3493 case Intrinsic::x86_bmi_bzhi_64:
3494 case Intrinsic::x86_bmi_pdep_32:
3495 case Intrinsic::x86_bmi_pdep_64:
3496 case Intrinsic::x86_bmi_pext_32:
3497 case Intrinsic::x86_bmi_pext_64:
3498 handleBmiIntrinsic(I);
3499 break;
3500
3501 case Intrinsic::x86_pclmulqdq:
3502 case Intrinsic::x86_pclmulqdq_256:
3503 case Intrinsic::x86_pclmulqdq_512:
3504 handlePclmulIntrinsic(I);
3505 break;
3506
3507 case Intrinsic::x86_sse41_round_sd:
3508 handleUnarySdIntrinsic(I);
3509 break;
3510 case Intrinsic::x86_sse2_max_sd:
3511 case Intrinsic::x86_sse2_min_sd:
3512 handleBinarySdIntrinsic(I);
3513 break;
3514
3515 case Intrinsic::is_constant:
3516 // The result of llvm.is.constant() is always defined.
3517 setShadow(&I, getCleanShadow(&I));
3518 setOrigin(&I, getCleanOrigin());
3519 break;
3520
3521 default:
3522 if (!handleUnknownIntrinsic(I))
3523 visitInstruction(I);
3524 break;
3525 }
3526 }
3527
visitLibAtomicLoad__anonb9e144640811::MemorySanitizerVisitor3528 void visitLibAtomicLoad(CallBase &CB) {
3529 // Since we use getNextNode here, we can't have CB terminate the BB.
3530 assert(isa<CallInst>(CB));
3531
3532 IRBuilder<> IRB(&CB);
3533 Value *Size = CB.getArgOperand(0);
3534 Value *SrcPtr = CB.getArgOperand(1);
3535 Value *DstPtr = CB.getArgOperand(2);
3536 Value *Ordering = CB.getArgOperand(3);
3537 // Convert the call to have at least Acquire ordering to make sure
3538 // the shadow operations aren't reordered before it.
3539 Value *NewOrdering =
3540 IRB.CreateExtractElement(makeAddAcquireOrderingTable(IRB), Ordering);
3541 CB.setArgOperand(3, NewOrdering);
3542
3543 IRBuilder<> NextIRB(CB.getNextNode());
3544 NextIRB.SetCurrentDebugLocation(CB.getDebugLoc());
3545
3546 Value *SrcShadowPtr, *SrcOriginPtr;
3547 std::tie(SrcShadowPtr, SrcOriginPtr) =
3548 getShadowOriginPtr(SrcPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
3549 /*isStore*/ false);
3550 Value *DstShadowPtr =
3551 getShadowOriginPtr(DstPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
3552 /*isStore*/ true)
3553 .first;
3554
3555 NextIRB.CreateMemCpy(DstShadowPtr, Align(1), SrcShadowPtr, Align(1), Size);
3556 if (MS.TrackOrigins) {
3557 Value *SrcOrigin = NextIRB.CreateAlignedLoad(MS.OriginTy, SrcOriginPtr,
3558 kMinOriginAlignment);
3559 Value *NewOrigin = updateOrigin(SrcOrigin, NextIRB);
3560 NextIRB.CreateCall(MS.MsanSetOriginFn, {DstPtr, Size, NewOrigin});
3561 }
3562 }
3563
visitLibAtomicStore__anonb9e144640811::MemorySanitizerVisitor3564 void visitLibAtomicStore(CallBase &CB) {
3565 IRBuilder<> IRB(&CB);
3566 Value *Size = CB.getArgOperand(0);
3567 Value *DstPtr = CB.getArgOperand(2);
3568 Value *Ordering = CB.getArgOperand(3);
3569 // Convert the call to have at least Release ordering to make sure
3570 // the shadow operations aren't reordered after it.
3571 Value *NewOrdering =
3572 IRB.CreateExtractElement(makeAddReleaseOrderingTable(IRB), Ordering);
3573 CB.setArgOperand(3, NewOrdering);
3574
3575 Value *DstShadowPtr =
3576 getShadowOriginPtr(DstPtr, IRB, IRB.getInt8Ty(), Align(1),
3577 /*isStore*/ true)
3578 .first;
3579
3580 // Atomic store always paints clean shadow/origin. See file header.
3581 IRB.CreateMemSet(DstShadowPtr, getCleanShadow(IRB.getInt8Ty()), Size,
3582 Align(1));
3583 }
3584
visitCallBase__anonb9e144640811::MemorySanitizerVisitor3585 void visitCallBase(CallBase &CB) {
3586 assert(!CB.getMetadata("nosanitize"));
3587 if (CB.isInlineAsm()) {
3588 // For inline asm (either a call to asm function, or callbr instruction),
3589 // do the usual thing: check argument shadow and mark all outputs as
3590 // clean. Note that any side effects of the inline asm that are not
3591 // immediately visible in its constraints are not handled.
3592 if (ClHandleAsmConservative && MS.CompileKernel)
3593 visitAsmInstruction(CB);
3594 else
3595 visitInstruction(CB);
3596 return;
3597 }
3598 LibFunc LF;
3599 if (TLI->getLibFunc(CB, LF)) {
3600 // libatomic.a functions need to have special handling because there isn't
3601 // a good way to intercept them or compile the library with
3602 // instrumentation.
3603 switch (LF) {
3604 case LibFunc_atomic_load:
3605 if (!isa<CallInst>(CB)) {
3606 llvm::errs() << "MSAN -- cannot instrument invoke of libatomic load."
3607 "Ignoring!\n";
3608 break;
3609 }
3610 visitLibAtomicLoad(CB);
3611 return;
3612 case LibFunc_atomic_store:
3613 visitLibAtomicStore(CB);
3614 return;
3615 default:
3616 break;
3617 }
3618 }
3619
3620 if (auto *Call = dyn_cast<CallInst>(&CB)) {
3621 assert(!isa<IntrinsicInst>(Call) && "intrinsics are handled elsewhere");
3622
3623 // We are going to insert code that relies on the fact that the callee
3624 // will become a non-readonly function after it is instrumented by us. To
3625 // prevent this code from being optimized out, mark that function
3626 // non-readonly in advance.
3627 AttrBuilder B;
3628 B.addAttribute(Attribute::ReadOnly)
3629 .addAttribute(Attribute::ReadNone)
3630 .addAttribute(Attribute::WriteOnly)
3631 .addAttribute(Attribute::ArgMemOnly)
3632 .addAttribute(Attribute::Speculatable);
3633
3634 Call->removeAttributes(AttributeList::FunctionIndex, B);
3635 if (Function *Func = Call->getCalledFunction()) {
3636 Func->removeAttributes(AttributeList::FunctionIndex, B);
3637 }
3638
3639 maybeMarkSanitizerLibraryCallNoBuiltin(Call, TLI);
3640 }
3641 IRBuilder<> IRB(&CB);
3642 bool MayCheckCall = ClEagerChecks;
3643 if (Function *Func = CB.getCalledFunction()) {
3644 // __sanitizer_unaligned_{load,store} functions may be called by users
3645 // and always expects shadows in the TLS. So don't check them.
3646 MayCheckCall &= !Func->getName().startswith("__sanitizer_unaligned_");
3647 }
3648
3649 unsigned ArgOffset = 0;
3650 LLVM_DEBUG(dbgs() << " CallSite: " << CB << "\n");
3651 for (auto ArgIt = CB.arg_begin(), End = CB.arg_end(); ArgIt != End;
3652 ++ArgIt) {
3653 Value *A = *ArgIt;
3654 unsigned i = ArgIt - CB.arg_begin();
3655 if (!A->getType()->isSized()) {
3656 LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << CB << "\n");
3657 continue;
3658 }
3659 unsigned Size = 0;
3660 Value *Store = nullptr;
3661 // Compute the Shadow for arg even if it is ByVal, because
3662 // in that case getShadow() will copy the actual arg shadow to
3663 // __msan_param_tls.
3664 Value *ArgShadow = getShadow(A);
3665 Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset);
3666 LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
3667 << " Shadow: " << *ArgShadow << "\n");
3668 bool ArgIsInitialized = false;
3669 const DataLayout &DL = F.getParent()->getDataLayout();
3670
3671 bool ByVal = CB.paramHasAttr(i, Attribute::ByVal);
3672 bool NoUndef = CB.paramHasAttr(i, Attribute::NoUndef);
3673 bool EagerCheck = MayCheckCall && !ByVal && NoUndef;
3674
3675 if (EagerCheck) {
3676 insertShadowCheck(A, &CB);
3677 continue;
3678 }
3679 if (ByVal) {
3680 // ByVal requires some special handling as it's too big for a single
3681 // load
3682 assert(A->getType()->isPointerTy() &&
3683 "ByVal argument is not a pointer!");
3684 Size = DL.getTypeAllocSize(CB.getParamByValType(i));
3685 if (ArgOffset + Size > kParamTLSSize) break;
3686 const MaybeAlign ParamAlignment(CB.getParamAlign(i));
3687 MaybeAlign Alignment = llvm::None;
3688 if (ParamAlignment)
3689 Alignment = std::min(*ParamAlignment, kShadowTLSAlignment);
3690 Value *AShadowPtr =
3691 getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
3692 /*isStore*/ false)
3693 .first;
3694
3695 Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
3696 Alignment, Size);
3697 // TODO(glider): need to copy origins.
3698 } else {
3699 // Any other parameters mean we need bit-grained tracking of uninit data
3700 Size = DL.getTypeAllocSize(A->getType());
3701 if (ArgOffset + Size > kParamTLSSize) break;
3702 Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
3703 kShadowTLSAlignment);
3704 Constant *Cst = dyn_cast<Constant>(ArgShadow);
3705 if (Cst && Cst->isNullValue()) ArgIsInitialized = true;
3706 }
3707 if (MS.TrackOrigins && !ArgIsInitialized)
3708 IRB.CreateStore(getOrigin(A),
3709 getOriginPtrForArgument(A, IRB, ArgOffset));
3710 (void)Store;
3711 assert(Size != 0 && Store != nullptr);
3712 LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
3713 ArgOffset += alignTo(Size, kShadowTLSAlignment);
3714 }
3715 LLVM_DEBUG(dbgs() << " done with call args\n");
3716
3717 FunctionType *FT = CB.getFunctionType();
3718 if (FT->isVarArg()) {
3719 VAHelper->visitCallBase(CB, IRB);
3720 }
3721
3722 // Now, get the shadow for the RetVal.
3723 if (!CB.getType()->isSized())
3724 return;
3725 // Don't emit the epilogue for musttail call returns.
3726 if (isa<CallInst>(CB) && cast<CallInst>(CB).isMustTailCall())
3727 return;
3728
3729 if (MayCheckCall && CB.hasRetAttr(Attribute::NoUndef)) {
3730 setShadow(&CB, getCleanShadow(&CB));
3731 setOrigin(&CB, getCleanOrigin());
3732 return;
3733 }
3734
3735 IRBuilder<> IRBBefore(&CB);
3736 // Until we have full dynamic coverage, make sure the retval shadow is 0.
3737 Value *Base = getShadowPtrForRetval(&CB, IRBBefore);
3738 IRBBefore.CreateAlignedStore(getCleanShadow(&CB), Base,
3739 kShadowTLSAlignment);
3740 BasicBlock::iterator NextInsn;
3741 if (isa<CallInst>(CB)) {
3742 NextInsn = ++CB.getIterator();
3743 assert(NextInsn != CB.getParent()->end());
3744 } else {
3745 BasicBlock *NormalDest = cast<InvokeInst>(CB).getNormalDest();
3746 if (!NormalDest->getSinglePredecessor()) {
3747 // FIXME: this case is tricky, so we are just conservative here.
3748 // Perhaps we need to split the edge between this BB and NormalDest,
3749 // but a naive attempt to use SplitEdge leads to a crash.
3750 setShadow(&CB, getCleanShadow(&CB));
3751 setOrigin(&CB, getCleanOrigin());
3752 return;
3753 }
3754 // FIXME: NextInsn is likely in a basic block that has not been visited yet.
3755 // Anything inserted there will be instrumented by MSan later!
3756 NextInsn = NormalDest->getFirstInsertionPt();
3757 assert(NextInsn != NormalDest->end() &&
3758 "Could not find insertion point for retval shadow load");
3759 }
3760 IRBuilder<> IRBAfter(&*NextInsn);
3761 Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
3762 getShadowTy(&CB), getShadowPtrForRetval(&CB, IRBAfter),
3763 kShadowTLSAlignment, "_msret");
3764 setShadow(&CB, RetvalShadow);
3765 if (MS.TrackOrigins)
3766 setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy,
3767 getOriginPtrForRetval(IRBAfter)));
3768 }
3769
isAMustTailRetVal__anonb9e144640811::MemorySanitizerVisitor3770 bool isAMustTailRetVal(Value *RetVal) {
3771 if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
3772 RetVal = I->getOperand(0);
3773 }
3774 if (auto *I = dyn_cast<CallInst>(RetVal)) {
3775 return I->isMustTailCall();
3776 }
3777 return false;
3778 }
3779
visitReturnInst__anonb9e144640811::MemorySanitizerVisitor3780 void visitReturnInst(ReturnInst &I) {
3781 IRBuilder<> IRB(&I);
3782 Value *RetVal = I.getReturnValue();
3783 if (!RetVal) return;
3784 // Don't emit the epilogue for musttail call returns.
3785 if (isAMustTailRetVal(RetVal)) return;
3786 Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB);
3787 bool HasNoUndef =
3788 F.hasAttribute(AttributeList::ReturnIndex, Attribute::NoUndef);
3789 bool StoreShadow = !(ClEagerChecks && HasNoUndef);
3790 // FIXME: Consider using SpecialCaseList to specify a list of functions that
3791 // must always return fully initialized values. For now, we hardcode "main".
3792 bool EagerCheck = (ClEagerChecks && HasNoUndef) || (F.getName() == "main");
3793
3794 Value *Shadow = getShadow(RetVal);
3795 bool StoreOrigin = true;
3796 if (EagerCheck) {
3797 insertShadowCheck(RetVal, &I);
3798 Shadow = getCleanShadow(RetVal);
3799 StoreOrigin = false;
3800 }
3801
3802 // The caller may still expect information passed over TLS if we pass our
3803 // check
3804 if (StoreShadow) {
3805 IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
3806 if (MS.TrackOrigins && StoreOrigin)
3807 IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
3808 }
3809 }
3810
visitPHINode__anonb9e144640811::MemorySanitizerVisitor3811 void visitPHINode(PHINode &I) {
3812 IRBuilder<> IRB(&I);
3813 if (!PropagateShadow) {
3814 setShadow(&I, getCleanShadow(&I));
3815 setOrigin(&I, getCleanOrigin());
3816 return;
3817 }
3818
3819 ShadowPHINodes.push_back(&I);
3820 setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
3821 "_msphi_s"));
3822 if (MS.TrackOrigins)
3823 setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(),
3824 "_msphi_o"));
3825 }
3826
getLocalVarDescription__anonb9e144640811::MemorySanitizerVisitor3827 Value *getLocalVarDescription(AllocaInst &I) {
3828 SmallString<2048> StackDescriptionStorage;
3829 raw_svector_ostream StackDescription(StackDescriptionStorage);
3830 // We create a string with a description of the stack allocation and
3831 // pass it into __msan_set_alloca_origin.
3832 // It will be printed by the run-time if stack-originated UMR is found.
3833 // The first 4 bytes of the string are set to '----' and will be replaced
3834 // by __msan_va_arg_overflow_size_tls at the first call.
3835 StackDescription << "----" << I.getName() << "@" << F.getName();
3836 return createPrivateNonConstGlobalForString(*F.getParent(),
3837 StackDescription.str());
3838 }
3839
poisonAllocaUserspace__anonb9e144640811::MemorySanitizerVisitor3840 void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3841 if (PoisonStack && ClPoisonStackWithCall) {
3842 IRB.CreateCall(MS.MsanPoisonStackFn,
3843 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3844 } else {
3845 Value *ShadowBase, *OriginBase;
3846 std::tie(ShadowBase, OriginBase) = getShadowOriginPtr(
3847 &I, IRB, IRB.getInt8Ty(), Align(1), /*isStore*/ true);
3848
3849 Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
3850 IRB.CreateMemSet(ShadowBase, PoisonValue, Len,
3851 MaybeAlign(I.getAlignment()));
3852 }
3853
3854 if (PoisonStack && MS.TrackOrigins) {
3855 Value *Descr = getLocalVarDescription(I);
3856 IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn,
3857 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3858 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()),
3859 IRB.CreatePointerCast(&F, MS.IntptrTy)});
3860 }
3861 }
3862
poisonAllocaKmsan__anonb9e144640811::MemorySanitizerVisitor3863 void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3864 Value *Descr = getLocalVarDescription(I);
3865 if (PoisonStack) {
3866 IRB.CreateCall(MS.MsanPoisonAllocaFn,
3867 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3868 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
3869 } else {
3870 IRB.CreateCall(MS.MsanUnpoisonAllocaFn,
3871 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3872 }
3873 }
3874
instrumentAlloca__anonb9e144640811::MemorySanitizerVisitor3875 void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
3876 if (!InsPoint)
3877 InsPoint = &I;
3878 IRBuilder<> IRB(InsPoint->getNextNode());
3879 const DataLayout &DL = F.getParent()->getDataLayout();
3880 uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
3881 Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
3882 if (I.isArrayAllocation())
3883 Len = IRB.CreateMul(Len, I.getArraySize());
3884
3885 if (MS.CompileKernel)
3886 poisonAllocaKmsan(I, IRB, Len);
3887 else
3888 poisonAllocaUserspace(I, IRB, Len);
3889 }
3890
visitAllocaInst__anonb9e144640811::MemorySanitizerVisitor3891 void visitAllocaInst(AllocaInst &I) {
3892 setShadow(&I, getCleanShadow(&I));
3893 setOrigin(&I, getCleanOrigin());
3894 // We'll get to this alloca later unless it's poisoned at the corresponding
3895 // llvm.lifetime.start.
3896 AllocaSet.insert(&I);
3897 }
3898
visitSelectInst__anonb9e144640811::MemorySanitizerVisitor3899 void visitSelectInst(SelectInst& I) {
3900 IRBuilder<> IRB(&I);
3901 // a = select b, c, d
3902 Value *B = I.getCondition();
3903 Value *C = I.getTrueValue();
3904 Value *D = I.getFalseValue();
3905 Value *Sb = getShadow(B);
3906 Value *Sc = getShadow(C);
3907 Value *Sd = getShadow(D);
3908
3909 // Result shadow if condition shadow is 0.
3910 Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
3911 Value *Sa1;
3912 if (I.getType()->isAggregateType()) {
3913 // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
3914 // an extra "select". This results in much more compact IR.
3915 // Sa = select Sb, poisoned, (select b, Sc, Sd)
3916 Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
3917 } else {
3918 // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
3919 // If Sb (condition is poisoned), look for bits in c and d that are equal
3920 // and both unpoisoned.
3921 // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
3922
3923 // Cast arguments to shadow-compatible type.
3924 C = CreateAppToShadowCast(IRB, C);
3925 D = CreateAppToShadowCast(IRB, D);
3926
3927 // Result shadow if condition shadow is 1.
3928 Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
3929 }
3930 Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
3931 setShadow(&I, Sa);
3932 if (MS.TrackOrigins) {
3933 // Origins are always i32, so any vector conditions must be flattened.
3934 // FIXME: consider tracking vector origins for app vectors?
3935 if (B->getType()->isVectorTy()) {
3936 Type *FlatTy = getShadowTyNoVec(B->getType());
3937 B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy),
3938 ConstantInt::getNullValue(FlatTy));
3939 Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy),
3940 ConstantInt::getNullValue(FlatTy));
3941 }
3942 // a = select b, c, d
3943 // Oa = Sb ? Ob : (b ? Oc : Od)
3944 setOrigin(
3945 &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
3946 IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
3947 getOrigin(I.getFalseValue()))));
3948 }
3949 }
3950
visitLandingPadInst__anonb9e144640811::MemorySanitizerVisitor3951 void visitLandingPadInst(LandingPadInst &I) {
3952 // Do nothing.
3953 // See https://github.com/google/sanitizers/issues/504
3954 setShadow(&I, getCleanShadow(&I));
3955 setOrigin(&I, getCleanOrigin());
3956 }
3957
visitCatchSwitchInst__anonb9e144640811::MemorySanitizerVisitor3958 void visitCatchSwitchInst(CatchSwitchInst &I) {
3959 setShadow(&I, getCleanShadow(&I));
3960 setOrigin(&I, getCleanOrigin());
3961 }
3962
visitFuncletPadInst__anonb9e144640811::MemorySanitizerVisitor3963 void visitFuncletPadInst(FuncletPadInst &I) {
3964 setShadow(&I, getCleanShadow(&I));
3965 setOrigin(&I, getCleanOrigin());
3966 }
3967
visitGetElementPtrInst__anonb9e144640811::MemorySanitizerVisitor3968 void visitGetElementPtrInst(GetElementPtrInst &I) {
3969 handleShadowOr(I);
3970 }
3971
visitExtractValueInst__anonb9e144640811::MemorySanitizerVisitor3972 void visitExtractValueInst(ExtractValueInst &I) {
3973 IRBuilder<> IRB(&I);
3974 Value *Agg = I.getAggregateOperand();
3975 LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
3976 Value *AggShadow = getShadow(Agg);
3977 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
3978 Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
3979 LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
3980 setShadow(&I, ResShadow);
3981 setOriginForNaryOp(I);
3982 }
3983
visitInsertValueInst__anonb9e144640811::MemorySanitizerVisitor3984 void visitInsertValueInst(InsertValueInst &I) {
3985 IRBuilder<> IRB(&I);
3986 LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
3987 Value *AggShadow = getShadow(I.getAggregateOperand());
3988 Value *InsShadow = getShadow(I.getInsertedValueOperand());
3989 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
3990 LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
3991 Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
3992 LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
3993 setShadow(&I, Res);
3994 setOriginForNaryOp(I);
3995 }
3996
dumpInst__anonb9e144640811::MemorySanitizerVisitor3997 void dumpInst(Instruction &I) {
3998 if (CallInst *CI = dyn_cast<CallInst>(&I)) {
3999 errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
4000 } else {
4001 errs() << "ZZZ " << I.getOpcodeName() << "\n";
4002 }
4003 errs() << "QQQ " << I << "\n";
4004 }
4005
visitResumeInst__anonb9e144640811::MemorySanitizerVisitor4006 void visitResumeInst(ResumeInst &I) {
4007 LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
4008 // Nothing to do here.
4009 }
4010
visitCleanupReturnInst__anonb9e144640811::MemorySanitizerVisitor4011 void visitCleanupReturnInst(CleanupReturnInst &CRI) {
4012 LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
4013 // Nothing to do here.
4014 }
4015
visitCatchReturnInst__anonb9e144640811::MemorySanitizerVisitor4016 void visitCatchReturnInst(CatchReturnInst &CRI) {
4017 LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
4018 // Nothing to do here.
4019 }
4020
instrumentAsmArgument__anonb9e144640811::MemorySanitizerVisitor4021 void instrumentAsmArgument(Value *Operand, Instruction &I, IRBuilder<> &IRB,
4022 const DataLayout &DL, bool isOutput) {
4023 // For each assembly argument, we check its value for being initialized.
4024 // If the argument is a pointer, we assume it points to a single element
4025 // of the corresponding type (or to a 8-byte word, if the type is unsized).
4026 // Each such pointer is instrumented with a call to the runtime library.
4027 Type *OpType = Operand->getType();
4028 // Check the operand value itself.
4029 insertShadowCheck(Operand, &I);
4030 if (!OpType->isPointerTy() || !isOutput) {
4031 assert(!isOutput);
4032 return;
4033 }
4034 Type *ElType = OpType->getPointerElementType();
4035 if (!ElType->isSized())
4036 return;
4037 int Size = DL.getTypeStoreSize(ElType);
4038 Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
4039 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
4040 IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
4041 }
4042
4043 /// Get the number of output arguments returned by pointers.
getNumOutputArgs__anonb9e144640811::MemorySanitizerVisitor4044 int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
4045 int NumRetOutputs = 0;
4046 int NumOutputs = 0;
4047 Type *RetTy = cast<Value>(CB)->getType();
4048 if (!RetTy->isVoidTy()) {
4049 // Register outputs are returned via the CallInst return value.
4050 auto *ST = dyn_cast<StructType>(RetTy);
4051 if (ST)
4052 NumRetOutputs = ST->getNumElements();
4053 else
4054 NumRetOutputs = 1;
4055 }
4056 InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
4057 for (const InlineAsm::ConstraintInfo &Info : Constraints) {
4058 switch (Info.Type) {
4059 case InlineAsm::isOutput:
4060 NumOutputs++;
4061 break;
4062 default:
4063 break;
4064 }
4065 }
4066 return NumOutputs - NumRetOutputs;
4067 }
4068
visitAsmInstruction__anonb9e144640811::MemorySanitizerVisitor4069 void visitAsmInstruction(Instruction &I) {
4070 // Conservative inline assembly handling: check for poisoned shadow of
4071 // asm() arguments, then unpoison the result and all the memory locations
4072 // pointed to by those arguments.
4073 // An inline asm() statement in C++ contains lists of input and output
4074 // arguments used by the assembly code. These are mapped to operands of the
4075 // CallInst as follows:
4076 // - nR register outputs ("=r) are returned by value in a single structure
4077 // (SSA value of the CallInst);
4078 // - nO other outputs ("=m" and others) are returned by pointer as first
4079 // nO operands of the CallInst;
4080 // - nI inputs ("r", "m" and others) are passed to CallInst as the
4081 // remaining nI operands.
4082 // The total number of asm() arguments in the source is nR+nO+nI, and the
4083 // corresponding CallInst has nO+nI+1 operands (the last operand is the
4084 // function to be called).
4085 const DataLayout &DL = F.getParent()->getDataLayout();
4086 CallBase *CB = cast<CallBase>(&I);
4087 IRBuilder<> IRB(&I);
4088 InlineAsm *IA = cast<InlineAsm>(CB->getCalledOperand());
4089 int OutputArgs = getNumOutputArgs(IA, CB);
4090 // The last operand of a CallInst is the function itself.
4091 int NumOperands = CB->getNumOperands() - 1;
4092
4093 // Check input arguments. Doing so before unpoisoning output arguments, so
4094 // that we won't overwrite uninit values before checking them.
4095 for (int i = OutputArgs; i < NumOperands; i++) {
4096 Value *Operand = CB->getOperand(i);
4097 instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ false);
4098 }
4099 // Unpoison output arguments. This must happen before the actual InlineAsm
4100 // call, so that the shadow for memory published in the asm() statement
4101 // remains valid.
4102 for (int i = 0; i < OutputArgs; i++) {
4103 Value *Operand = CB->getOperand(i);
4104 instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ true);
4105 }
4106
4107 setShadow(&I, getCleanShadow(&I));
4108 setOrigin(&I, getCleanOrigin());
4109 }
4110
visitFreezeInst__anonb9e144640811::MemorySanitizerVisitor4111 void visitFreezeInst(FreezeInst &I) {
4112 // Freeze always returns a fully defined value.
4113 setShadow(&I, getCleanShadow(&I));
4114 setOrigin(&I, getCleanOrigin());
4115 }
4116
visitInstruction__anonb9e144640811::MemorySanitizerVisitor4117 void visitInstruction(Instruction &I) {
4118 // Everything else: stop propagating and check for poisoned shadow.
4119 if (ClDumpStrictInstructions)
4120 dumpInst(I);
4121 LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
4122 for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
4123 Value *Operand = I.getOperand(i);
4124 if (Operand->getType()->isSized())
4125 insertShadowCheck(Operand, &I);
4126 }
4127 setShadow(&I, getCleanShadow(&I));
4128 setOrigin(&I, getCleanOrigin());
4129 }
4130 };
4131
4132 /// AMD64-specific implementation of VarArgHelper.
4133 struct VarArgAMD64Helper : public VarArgHelper {
4134 // An unfortunate workaround for asymmetric lowering of va_arg stuff.
4135 // See a comment in visitCallBase for more details.
4136 static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
4137 static const unsigned AMD64FpEndOffsetSSE = 176;
4138 // If SSE is disabled, fp_offset in va_list is zero.
4139 static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
4140
4141 unsigned AMD64FpEndOffset;
4142 Function &F;
4143 MemorySanitizer &MS;
4144 MemorySanitizerVisitor &MSV;
4145 Value *VAArgTLSCopy = nullptr;
4146 Value *VAArgTLSOriginCopy = nullptr;
4147 Value *VAArgOverflowSize = nullptr;
4148
4149 SmallVector<CallInst*, 16> VAStartInstrumentationList;
4150
4151 enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4152
VarArgAMD64Helper__anonb9e144640811::VarArgAMD64Helper4153 VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
4154 MemorySanitizerVisitor &MSV)
4155 : F(F), MS(MS), MSV(MSV) {
4156 AMD64FpEndOffset = AMD64FpEndOffsetSSE;
4157 for (const auto &Attr : F.getAttributes().getFnAttributes()) {
4158 if (Attr.isStringAttribute() &&
4159 (Attr.getKindAsString() == "target-features")) {
4160 if (Attr.getValueAsString().contains("-sse"))
4161 AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
4162 break;
4163 }
4164 }
4165 }
4166
classifyArgument__anonb9e144640811::VarArgAMD64Helper4167 ArgKind classifyArgument(Value* arg) {
4168 // A very rough approximation of X86_64 argument classification rules.
4169 Type *T = arg->getType();
4170 if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
4171 return AK_FloatingPoint;
4172 if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4173 return AK_GeneralPurpose;
4174 if (T->isPointerTy())
4175 return AK_GeneralPurpose;
4176 return AK_Memory;
4177 }
4178
4179 // For VarArg functions, store the argument shadow in an ABI-specific format
4180 // that corresponds to va_list layout.
4181 // We do this because Clang lowers va_arg in the frontend, and this pass
4182 // only sees the low level code that deals with va_list internals.
4183 // A much easier alternative (provided that Clang emits va_arg instructions)
4184 // would have been to associate each live instance of va_list with a copy of
4185 // MSanParamTLS, and extract shadow on va_arg() call in the argument list
4186 // order.
visitCallBase__anonb9e144640811::VarArgAMD64Helper4187 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4188 unsigned GpOffset = 0;
4189 unsigned FpOffset = AMD64GpEndOffset;
4190 unsigned OverflowOffset = AMD64FpEndOffset;
4191 const DataLayout &DL = F.getParent()->getDataLayout();
4192 for (auto ArgIt = CB.arg_begin(), End = CB.arg_end(); ArgIt != End;
4193 ++ArgIt) {
4194 Value *A = *ArgIt;
4195 unsigned ArgNo = CB.getArgOperandNo(ArgIt);
4196 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4197 bool IsByVal = CB.paramHasAttr(ArgNo, Attribute::ByVal);
4198 if (IsByVal) {
4199 // ByVal arguments always go to the overflow area.
4200 // Fixed arguments passed through the overflow area will be stepped
4201 // over by va_start, so don't count them towards the offset.
4202 if (IsFixed)
4203 continue;
4204 assert(A->getType()->isPointerTy());
4205 Type *RealTy = CB.getParamByValType(ArgNo);
4206 uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4207 Value *ShadowBase = getShadowPtrForVAArgument(
4208 RealTy, IRB, OverflowOffset, alignTo(ArgSize, 8));
4209 Value *OriginBase = nullptr;
4210 if (MS.TrackOrigins)
4211 OriginBase = getOriginPtrForVAArgument(RealTy, IRB, OverflowOffset);
4212 OverflowOffset += alignTo(ArgSize, 8);
4213 if (!ShadowBase)
4214 continue;
4215 Value *ShadowPtr, *OriginPtr;
4216 std::tie(ShadowPtr, OriginPtr) =
4217 MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
4218 /*isStore*/ false);
4219
4220 IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
4221 kShadowTLSAlignment, ArgSize);
4222 if (MS.TrackOrigins)
4223 IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
4224 kShadowTLSAlignment, ArgSize);
4225 } else {
4226 ArgKind AK = classifyArgument(A);
4227 if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
4228 AK = AK_Memory;
4229 if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
4230 AK = AK_Memory;
4231 Value *ShadowBase, *OriginBase = nullptr;
4232 switch (AK) {
4233 case AK_GeneralPurpose:
4234 ShadowBase =
4235 getShadowPtrForVAArgument(A->getType(), IRB, GpOffset, 8);
4236 if (MS.TrackOrigins)
4237 OriginBase =
4238 getOriginPtrForVAArgument(A->getType(), IRB, GpOffset);
4239 GpOffset += 8;
4240 break;
4241 case AK_FloatingPoint:
4242 ShadowBase =
4243 getShadowPtrForVAArgument(A->getType(), IRB, FpOffset, 16);
4244 if (MS.TrackOrigins)
4245 OriginBase =
4246 getOriginPtrForVAArgument(A->getType(), IRB, FpOffset);
4247 FpOffset += 16;
4248 break;
4249 case AK_Memory:
4250 if (IsFixed)
4251 continue;
4252 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4253 ShadowBase =
4254 getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset, 8);
4255 if (MS.TrackOrigins)
4256 OriginBase =
4257 getOriginPtrForVAArgument(A->getType(), IRB, OverflowOffset);
4258 OverflowOffset += alignTo(ArgSize, 8);
4259 }
4260 // Take fixed arguments into account for GpOffset and FpOffset,
4261 // but don't actually store shadows for them.
4262 // TODO(glider): don't call get*PtrForVAArgument() for them.
4263 if (IsFixed)
4264 continue;
4265 if (!ShadowBase)
4266 continue;
4267 Value *Shadow = MSV.getShadow(A);
4268 IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
4269 if (MS.TrackOrigins) {
4270 Value *Origin = MSV.getOrigin(A);
4271 unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
4272 MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
4273 std::max(kShadowTLSAlignment, kMinOriginAlignment));
4274 }
4275 }
4276 }
4277 Constant *OverflowSize =
4278 ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
4279 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4280 }
4281
4282 /// Compute the shadow address for a given va_arg.
getShadowPtrForVAArgument__anonb9e144640811::VarArgAMD64Helper4283 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4284 unsigned ArgOffset, unsigned ArgSize) {
4285 // Make sure we don't overflow __msan_va_arg_tls.
4286 if (ArgOffset + ArgSize > kParamTLSSize)
4287 return nullptr;
4288 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4289 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4290 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4291 "_msarg_va_s");
4292 }
4293
4294 /// Compute the origin address for a given va_arg.
getOriginPtrForVAArgument__anonb9e144640811::VarArgAMD64Helper4295 Value *getOriginPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, int ArgOffset) {
4296 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
4297 // getOriginPtrForVAArgument() is always called after
4298 // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
4299 // overflow.
4300 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4301 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
4302 "_msarg_va_o");
4303 }
4304
unpoisonVAListTagForInst__anonb9e144640811::VarArgAMD64Helper4305 void unpoisonVAListTagForInst(IntrinsicInst &I) {
4306 IRBuilder<> IRB(&I);
4307 Value *VAListTag = I.getArgOperand(0);
4308 Value *ShadowPtr, *OriginPtr;
4309 const Align Alignment = Align(8);
4310 std::tie(ShadowPtr, OriginPtr) =
4311 MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
4312 /*isStore*/ true);
4313
4314 // Unpoison the whole __va_list_tag.
4315 // FIXME: magic ABI constants.
4316 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4317 /* size */ 24, Alignment, false);
4318 // We shouldn't need to zero out the origins, as they're only checked for
4319 // nonzero shadow.
4320 }
4321
visitVAStartInst__anonb9e144640811::VarArgAMD64Helper4322 void visitVAStartInst(VAStartInst &I) override {
4323 if (F.getCallingConv() == CallingConv::Win64)
4324 return;
4325 VAStartInstrumentationList.push_back(&I);
4326 unpoisonVAListTagForInst(I);
4327 }
4328
visitVACopyInst__anonb9e144640811::VarArgAMD64Helper4329 void visitVACopyInst(VACopyInst &I) override {
4330 if (F.getCallingConv() == CallingConv::Win64) return;
4331 unpoisonVAListTagForInst(I);
4332 }
4333
finalizeInstrumentation__anonb9e144640811::VarArgAMD64Helper4334 void finalizeInstrumentation() override {
4335 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4336 "finalizeInstrumentation called twice");
4337 if (!VAStartInstrumentationList.empty()) {
4338 // If there is a va_start in this function, make a backup copy of
4339 // va_arg_tls somewhere in the function entry block.
4340 IRBuilder<> IRB(MSV.FnPrologueEnd);
4341 VAArgOverflowSize =
4342 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4343 Value *CopySize =
4344 IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset),
4345 VAArgOverflowSize);
4346 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4347 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
4348 if (MS.TrackOrigins) {
4349 VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4350 IRB.CreateMemCpy(VAArgTLSOriginCopy, Align(8), MS.VAArgOriginTLS,
4351 Align(8), CopySize);
4352 }
4353 }
4354
4355 // Instrument va_start.
4356 // Copy va_list shadow from the backup copy of the TLS contents.
4357 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4358 CallInst *OrigInst = VAStartInstrumentationList[i];
4359 IRBuilder<> IRB(OrigInst->getNextNode());
4360 Value *VAListTag = OrigInst->getArgOperand(0);
4361
4362 Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4363 Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
4364 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4365 ConstantInt::get(MS.IntptrTy, 16)),
4366 PointerType::get(RegSaveAreaPtrTy, 0));
4367 Value *RegSaveAreaPtr =
4368 IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4369 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4370 const Align Alignment = Align(16);
4371 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4372 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4373 Alignment, /*isStore*/ true);
4374 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4375 AMD64FpEndOffset);
4376 if (MS.TrackOrigins)
4377 IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
4378 Alignment, AMD64FpEndOffset);
4379 Type *OverflowArgAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4380 Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
4381 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4382 ConstantInt::get(MS.IntptrTy, 8)),
4383 PointerType::get(OverflowArgAreaPtrTy, 0));
4384 Value *OverflowArgAreaPtr =
4385 IRB.CreateLoad(OverflowArgAreaPtrTy, OverflowArgAreaPtrPtr);
4386 Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
4387 std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
4388 MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
4389 Alignment, /*isStore*/ true);
4390 Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
4391 AMD64FpEndOffset);
4392 IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
4393 VAArgOverflowSize);
4394 if (MS.TrackOrigins) {
4395 SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSOriginCopy,
4396 AMD64FpEndOffset);
4397 IRB.CreateMemCpy(OverflowArgAreaOriginPtr, Alignment, SrcPtr, Alignment,
4398 VAArgOverflowSize);
4399 }
4400 }
4401 }
4402 };
4403
4404 /// MIPS64-specific implementation of VarArgHelper.
4405 struct VarArgMIPS64Helper : public VarArgHelper {
4406 Function &F;
4407 MemorySanitizer &MS;
4408 MemorySanitizerVisitor &MSV;
4409 Value *VAArgTLSCopy = nullptr;
4410 Value *VAArgSize = nullptr;
4411
4412 SmallVector<CallInst*, 16> VAStartInstrumentationList;
4413
VarArgMIPS64Helper__anonb9e144640811::VarArgMIPS64Helper4414 VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
4415 MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4416
visitCallBase__anonb9e144640811::VarArgMIPS64Helper4417 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4418 unsigned VAArgOffset = 0;
4419 const DataLayout &DL = F.getParent()->getDataLayout();
4420 for (auto ArgIt = CB.arg_begin() + CB.getFunctionType()->getNumParams(),
4421 End = CB.arg_end();
4422 ArgIt != End; ++ArgIt) {
4423 Triple TargetTriple(F.getParent()->getTargetTriple());
4424 Value *A = *ArgIt;
4425 Value *Base;
4426 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4427 if (TargetTriple.getArch() == Triple::mips64) {
4428 // Adjusting the shadow for argument with size < 8 to match the placement
4429 // of bits in big endian system
4430 if (ArgSize < 8)
4431 VAArgOffset += (8 - ArgSize);
4432 }
4433 Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset, ArgSize);
4434 VAArgOffset += ArgSize;
4435 VAArgOffset = alignTo(VAArgOffset, 8);
4436 if (!Base)
4437 continue;
4438 IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4439 }
4440
4441 Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset);
4442 // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4443 // a new class member i.e. it is the total size of all VarArgs.
4444 IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4445 }
4446
4447 /// Compute the shadow address for a given va_arg.
getShadowPtrForVAArgument__anonb9e144640811::VarArgMIPS64Helper4448 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4449 unsigned ArgOffset, unsigned ArgSize) {
4450 // Make sure we don't overflow __msan_va_arg_tls.
4451 if (ArgOffset + ArgSize > kParamTLSSize)
4452 return nullptr;
4453 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4454 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4455 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4456 "_msarg");
4457 }
4458
visitVAStartInst__anonb9e144640811::VarArgMIPS64Helper4459 void visitVAStartInst(VAStartInst &I) override {
4460 IRBuilder<> IRB(&I);
4461 VAStartInstrumentationList.push_back(&I);
4462 Value *VAListTag = I.getArgOperand(0);
4463 Value *ShadowPtr, *OriginPtr;
4464 const Align Alignment = Align(8);
4465 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4466 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4467 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4468 /* size */ 8, Alignment, false);
4469 }
4470
visitVACopyInst__anonb9e144640811::VarArgMIPS64Helper4471 void visitVACopyInst(VACopyInst &I) override {
4472 IRBuilder<> IRB(&I);
4473 VAStartInstrumentationList.push_back(&I);
4474 Value *VAListTag = I.getArgOperand(0);
4475 Value *ShadowPtr, *OriginPtr;
4476 const Align Alignment = Align(8);
4477 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4478 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4479 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4480 /* size */ 8, Alignment, false);
4481 }
4482
finalizeInstrumentation__anonb9e144640811::VarArgMIPS64Helper4483 void finalizeInstrumentation() override {
4484 assert(!VAArgSize && !VAArgTLSCopy &&
4485 "finalizeInstrumentation called twice");
4486 IRBuilder<> IRB(MSV.FnPrologueEnd);
4487 VAArgSize = IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4488 Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
4489 VAArgSize);
4490
4491 if (!VAStartInstrumentationList.empty()) {
4492 // If there is a va_start in this function, make a backup copy of
4493 // va_arg_tls somewhere in the function entry block.
4494 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4495 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
4496 }
4497
4498 // Instrument va_start.
4499 // Copy va_list shadow from the backup copy of the TLS contents.
4500 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4501 CallInst *OrigInst = VAStartInstrumentationList[i];
4502 IRBuilder<> IRB(OrigInst->getNextNode());
4503 Value *VAListTag = OrigInst->getArgOperand(0);
4504 Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4505 Value *RegSaveAreaPtrPtr =
4506 IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4507 PointerType::get(RegSaveAreaPtrTy, 0));
4508 Value *RegSaveAreaPtr =
4509 IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4510 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4511 const Align Alignment = Align(8);
4512 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4513 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4514 Alignment, /*isStore*/ true);
4515 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4516 CopySize);
4517 }
4518 }
4519 };
4520
4521 /// AArch64-specific implementation of VarArgHelper.
4522 struct VarArgAArch64Helper : public VarArgHelper {
4523 static const unsigned kAArch64GrArgSize = 64;
4524 static const unsigned kAArch64VrArgSize = 128;
4525
4526 static const unsigned AArch64GrBegOffset = 0;
4527 static const unsigned AArch64GrEndOffset = kAArch64GrArgSize;
4528 // Make VR space aligned to 16 bytes.
4529 static const unsigned AArch64VrBegOffset = AArch64GrEndOffset;
4530 static const unsigned AArch64VrEndOffset = AArch64VrBegOffset
4531 + kAArch64VrArgSize;
4532 static const unsigned AArch64VAEndOffset = AArch64VrEndOffset;
4533
4534 Function &F;
4535 MemorySanitizer &MS;
4536 MemorySanitizerVisitor &MSV;
4537 Value *VAArgTLSCopy = nullptr;
4538 Value *VAArgOverflowSize = nullptr;
4539
4540 SmallVector<CallInst*, 16> VAStartInstrumentationList;
4541
4542 enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4543
VarArgAArch64Helper__anonb9e144640811::VarArgAArch64Helper4544 VarArgAArch64Helper(Function &F, MemorySanitizer &MS,
4545 MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4546
classifyArgument__anonb9e144640811::VarArgAArch64Helper4547 ArgKind classifyArgument(Value* arg) {
4548 Type *T = arg->getType();
4549 if (T->isFPOrFPVectorTy())
4550 return AK_FloatingPoint;
4551 if ((T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4552 || (T->isPointerTy()))
4553 return AK_GeneralPurpose;
4554 return AK_Memory;
4555 }
4556
4557 // The instrumentation stores the argument shadow in a non ABI-specific
4558 // format because it does not know which argument is named (since Clang,
4559 // like x86_64 case, lowers the va_args in the frontend and this pass only
4560 // sees the low level code that deals with va_list internals).
4561 // The first seven GR registers are saved in the first 56 bytes of the
4562 // va_arg tls arra, followers by the first 8 FP/SIMD registers, and then
4563 // the remaining arguments.
4564 // Using constant offset within the va_arg TLS array allows fast copy
4565 // in the finalize instrumentation.
visitCallBase__anonb9e144640811::VarArgAArch64Helper4566 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4567 unsigned GrOffset = AArch64GrBegOffset;
4568 unsigned VrOffset = AArch64VrBegOffset;
4569 unsigned OverflowOffset = AArch64VAEndOffset;
4570
4571 const DataLayout &DL = F.getParent()->getDataLayout();
4572 for (auto ArgIt = CB.arg_begin(), End = CB.arg_end(); ArgIt != End;
4573 ++ArgIt) {
4574 Value *A = *ArgIt;
4575 unsigned ArgNo = CB.getArgOperandNo(ArgIt);
4576 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4577 ArgKind AK = classifyArgument(A);
4578 if (AK == AK_GeneralPurpose && GrOffset >= AArch64GrEndOffset)
4579 AK = AK_Memory;
4580 if (AK == AK_FloatingPoint && VrOffset >= AArch64VrEndOffset)
4581 AK = AK_Memory;
4582 Value *Base;
4583 switch (AK) {
4584 case AK_GeneralPurpose:
4585 Base = getShadowPtrForVAArgument(A->getType(), IRB, GrOffset, 8);
4586 GrOffset += 8;
4587 break;
4588 case AK_FloatingPoint:
4589 Base = getShadowPtrForVAArgument(A->getType(), IRB, VrOffset, 8);
4590 VrOffset += 16;
4591 break;
4592 case AK_Memory:
4593 // Don't count fixed arguments in the overflow area - va_start will
4594 // skip right over them.
4595 if (IsFixed)
4596 continue;
4597 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4598 Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset,
4599 alignTo(ArgSize, 8));
4600 OverflowOffset += alignTo(ArgSize, 8);
4601 break;
4602 }
4603 // Count Gp/Vr fixed arguments to their respective offsets, but don't
4604 // bother to actually store a shadow.
4605 if (IsFixed)
4606 continue;
4607 if (!Base)
4608 continue;
4609 IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4610 }
4611 Constant *OverflowSize =
4612 ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset);
4613 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4614 }
4615
4616 /// Compute the shadow address for a given va_arg.
getShadowPtrForVAArgument__anonb9e144640811::VarArgAArch64Helper4617 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4618 unsigned ArgOffset, unsigned ArgSize) {
4619 // Make sure we don't overflow __msan_va_arg_tls.
4620 if (ArgOffset + ArgSize > kParamTLSSize)
4621 return nullptr;
4622 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4623 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4624 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4625 "_msarg");
4626 }
4627
visitVAStartInst__anonb9e144640811::VarArgAArch64Helper4628 void visitVAStartInst(VAStartInst &I) override {
4629 IRBuilder<> IRB(&I);
4630 VAStartInstrumentationList.push_back(&I);
4631 Value *VAListTag = I.getArgOperand(0);
4632 Value *ShadowPtr, *OriginPtr;
4633 const Align Alignment = Align(8);
4634 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4635 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4636 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4637 /* size */ 32, Alignment, false);
4638 }
4639
visitVACopyInst__anonb9e144640811::VarArgAArch64Helper4640 void visitVACopyInst(VACopyInst &I) override {
4641 IRBuilder<> IRB(&I);
4642 VAStartInstrumentationList.push_back(&I);
4643 Value *VAListTag = I.getArgOperand(0);
4644 Value *ShadowPtr, *OriginPtr;
4645 const Align Alignment = Align(8);
4646 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4647 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4648 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4649 /* size */ 32, Alignment, false);
4650 }
4651
4652 // Retrieve a va_list field of 'void*' size.
getVAField64__anonb9e144640811::VarArgAArch64Helper4653 Value* getVAField64(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4654 Value *SaveAreaPtrPtr =
4655 IRB.CreateIntToPtr(
4656 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4657 ConstantInt::get(MS.IntptrTy, offset)),
4658 Type::getInt64PtrTy(*MS.C));
4659 return IRB.CreateLoad(Type::getInt64Ty(*MS.C), SaveAreaPtrPtr);
4660 }
4661
4662 // Retrieve a va_list field of 'int' size.
getVAField32__anonb9e144640811::VarArgAArch64Helper4663 Value* getVAField32(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4664 Value *SaveAreaPtr =
4665 IRB.CreateIntToPtr(
4666 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4667 ConstantInt::get(MS.IntptrTy, offset)),
4668 Type::getInt32PtrTy(*MS.C));
4669 Value *SaveArea32 = IRB.CreateLoad(IRB.getInt32Ty(), SaveAreaPtr);
4670 return IRB.CreateSExt(SaveArea32, MS.IntptrTy);
4671 }
4672
finalizeInstrumentation__anonb9e144640811::VarArgAArch64Helper4673 void finalizeInstrumentation() override {
4674 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4675 "finalizeInstrumentation called twice");
4676 if (!VAStartInstrumentationList.empty()) {
4677 // If there is a va_start in this function, make a backup copy of
4678 // va_arg_tls somewhere in the function entry block.
4679 IRBuilder<> IRB(MSV.FnPrologueEnd);
4680 VAArgOverflowSize =
4681 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4682 Value *CopySize =
4683 IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset),
4684 VAArgOverflowSize);
4685 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4686 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
4687 }
4688
4689 Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize);
4690 Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize);
4691
4692 // Instrument va_start, copy va_list shadow from the backup copy of
4693 // the TLS contents.
4694 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4695 CallInst *OrigInst = VAStartInstrumentationList[i];
4696 IRBuilder<> IRB(OrigInst->getNextNode());
4697
4698 Value *VAListTag = OrigInst->getArgOperand(0);
4699
4700 // The variadic ABI for AArch64 creates two areas to save the incoming
4701 // argument registers (one for 64-bit general register xn-x7 and another
4702 // for 128-bit FP/SIMD vn-v7).
4703 // We need then to propagate the shadow arguments on both regions
4704 // 'va::__gr_top + va::__gr_offs' and 'va::__vr_top + va::__vr_offs'.
4705 // The remaining arguments are saved on shadow for 'va::stack'.
4706 // One caveat is it requires only to propagate the non-named arguments,
4707 // however on the call site instrumentation 'all' the arguments are
4708 // saved. So to copy the shadow values from the va_arg TLS array
4709 // we need to adjust the offset for both GR and VR fields based on
4710 // the __{gr,vr}_offs value (since they are stores based on incoming
4711 // named arguments).
4712
4713 // Read the stack pointer from the va_list.
4714 Value *StackSaveAreaPtr = getVAField64(IRB, VAListTag, 0);
4715
4716 // Read both the __gr_top and __gr_off and add them up.
4717 Value *GrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 8);
4718 Value *GrOffSaveArea = getVAField32(IRB, VAListTag, 24);
4719
4720 Value *GrRegSaveAreaPtr = IRB.CreateAdd(GrTopSaveAreaPtr, GrOffSaveArea);
4721
4722 // Read both the __vr_top and __vr_off and add them up.
4723 Value *VrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 16);
4724 Value *VrOffSaveArea = getVAField32(IRB, VAListTag, 28);
4725
4726 Value *VrRegSaveAreaPtr = IRB.CreateAdd(VrTopSaveAreaPtr, VrOffSaveArea);
4727
4728 // It does not know how many named arguments is being used and, on the
4729 // callsite all the arguments were saved. Since __gr_off is defined as
4730 // '0 - ((8 - named_gr) * 8)', the idea is to just propagate the variadic
4731 // argument by ignoring the bytes of shadow from named arguments.
4732 Value *GrRegSaveAreaShadowPtrOff =
4733 IRB.CreateAdd(GrArgSize, GrOffSaveArea);
4734
4735 Value *GrRegSaveAreaShadowPtr =
4736 MSV.getShadowOriginPtr(GrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4737 Align(8), /*isStore*/ true)
4738 .first;
4739
4740 Value *GrSrcPtr = IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4741 GrRegSaveAreaShadowPtrOff);
4742 Value *GrCopySize = IRB.CreateSub(GrArgSize, GrRegSaveAreaShadowPtrOff);
4743
4744 IRB.CreateMemCpy(GrRegSaveAreaShadowPtr, Align(8), GrSrcPtr, Align(8),
4745 GrCopySize);
4746
4747 // Again, but for FP/SIMD values.
4748 Value *VrRegSaveAreaShadowPtrOff =
4749 IRB.CreateAdd(VrArgSize, VrOffSaveArea);
4750
4751 Value *VrRegSaveAreaShadowPtr =
4752 MSV.getShadowOriginPtr(VrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4753 Align(8), /*isStore*/ true)
4754 .first;
4755
4756 Value *VrSrcPtr = IRB.CreateInBoundsGEP(
4757 IRB.getInt8Ty(),
4758 IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4759 IRB.getInt32(AArch64VrBegOffset)),
4760 VrRegSaveAreaShadowPtrOff);
4761 Value *VrCopySize = IRB.CreateSub(VrArgSize, VrRegSaveAreaShadowPtrOff);
4762
4763 IRB.CreateMemCpy(VrRegSaveAreaShadowPtr, Align(8), VrSrcPtr, Align(8),
4764 VrCopySize);
4765
4766 // And finally for remaining arguments.
4767 Value *StackSaveAreaShadowPtr =
4768 MSV.getShadowOriginPtr(StackSaveAreaPtr, IRB, IRB.getInt8Ty(),
4769 Align(16), /*isStore*/ true)
4770 .first;
4771
4772 Value *StackSrcPtr =
4773 IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4774 IRB.getInt32(AArch64VAEndOffset));
4775
4776 IRB.CreateMemCpy(StackSaveAreaShadowPtr, Align(16), StackSrcPtr,
4777 Align(16), VAArgOverflowSize);
4778 }
4779 }
4780 };
4781
4782 /// PowerPC64-specific implementation of VarArgHelper.
4783 struct VarArgPowerPC64Helper : public VarArgHelper {
4784 Function &F;
4785 MemorySanitizer &MS;
4786 MemorySanitizerVisitor &MSV;
4787 Value *VAArgTLSCopy = nullptr;
4788 Value *VAArgSize = nullptr;
4789
4790 SmallVector<CallInst*, 16> VAStartInstrumentationList;
4791
VarArgPowerPC64Helper__anonb9e144640811::VarArgPowerPC64Helper4792 VarArgPowerPC64Helper(Function &F, MemorySanitizer &MS,
4793 MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4794
visitCallBase__anonb9e144640811::VarArgPowerPC64Helper4795 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4796 // For PowerPC, we need to deal with alignment of stack arguments -
4797 // they are mostly aligned to 8 bytes, but vectors and i128 arrays
4798 // are aligned to 16 bytes, byvals can be aligned to 8 or 16 bytes,
4799 // For that reason, we compute current offset from stack pointer (which is
4800 // always properly aligned), and offset for the first vararg, then subtract
4801 // them.
4802 unsigned VAArgBase;
4803 Triple TargetTriple(F.getParent()->getTargetTriple());
4804 // Parameter save area starts at 48 bytes from frame pointer for ABIv1,
4805 // and 32 bytes for ABIv2. This is usually determined by target
4806 // endianness, but in theory could be overridden by function attribute.
4807 if (TargetTriple.getArch() == Triple::ppc64)
4808 VAArgBase = 48;
4809 else
4810 VAArgBase = 32;
4811 unsigned VAArgOffset = VAArgBase;
4812 const DataLayout &DL = F.getParent()->getDataLayout();
4813 for (auto ArgIt = CB.arg_begin(), End = CB.arg_end(); ArgIt != End;
4814 ++ArgIt) {
4815 Value *A = *ArgIt;
4816 unsigned ArgNo = CB.getArgOperandNo(ArgIt);
4817 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4818 bool IsByVal = CB.paramHasAttr(ArgNo, Attribute::ByVal);
4819 if (IsByVal) {
4820 assert(A->getType()->isPointerTy());
4821 Type *RealTy = CB.getParamByValType(ArgNo);
4822 uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4823 MaybeAlign ArgAlign = CB.getParamAlign(ArgNo);
4824 if (!ArgAlign || *ArgAlign < Align(8))
4825 ArgAlign = Align(8);
4826 VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4827 if (!IsFixed) {
4828 Value *Base = getShadowPtrForVAArgument(
4829 RealTy, IRB, VAArgOffset - VAArgBase, ArgSize);
4830 if (Base) {
4831 Value *AShadowPtr, *AOriginPtr;
4832 std::tie(AShadowPtr, AOriginPtr) =
4833 MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(),
4834 kShadowTLSAlignment, /*isStore*/ false);
4835
4836 IRB.CreateMemCpy(Base, kShadowTLSAlignment, AShadowPtr,
4837 kShadowTLSAlignment, ArgSize);
4838 }
4839 }
4840 VAArgOffset += alignTo(ArgSize, 8);
4841 } else {
4842 Value *Base;
4843 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4844 uint64_t ArgAlign = 8;
4845 if (A->getType()->isArrayTy()) {
4846 // Arrays are aligned to element size, except for long double
4847 // arrays, which are aligned to 8 bytes.
4848 Type *ElementTy = A->getType()->getArrayElementType();
4849 if (!ElementTy->isPPC_FP128Ty())
4850 ArgAlign = DL.getTypeAllocSize(ElementTy);
4851 } else if (A->getType()->isVectorTy()) {
4852 // Vectors are naturally aligned.
4853 ArgAlign = DL.getTypeAllocSize(A->getType());
4854 }
4855 if (ArgAlign < 8)
4856 ArgAlign = 8;
4857 VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4858 if (DL.isBigEndian()) {
4859 // Adjusting the shadow for argument with size < 8 to match the placement
4860 // of bits in big endian system
4861 if (ArgSize < 8)
4862 VAArgOffset += (8 - ArgSize);
4863 }
4864 if (!IsFixed) {
4865 Base = getShadowPtrForVAArgument(A->getType(), IRB,
4866 VAArgOffset - VAArgBase, ArgSize);
4867 if (Base)
4868 IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4869 }
4870 VAArgOffset += ArgSize;
4871 VAArgOffset = alignTo(VAArgOffset, 8);
4872 }
4873 if (IsFixed)
4874 VAArgBase = VAArgOffset;
4875 }
4876
4877 Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(),
4878 VAArgOffset - VAArgBase);
4879 // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4880 // a new class member i.e. it is the total size of all VarArgs.
4881 IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4882 }
4883
4884 /// Compute the shadow address for a given va_arg.
getShadowPtrForVAArgument__anonb9e144640811::VarArgPowerPC64Helper4885 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4886 unsigned ArgOffset, unsigned ArgSize) {
4887 // Make sure we don't overflow __msan_va_arg_tls.
4888 if (ArgOffset + ArgSize > kParamTLSSize)
4889 return nullptr;
4890 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4891 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4892 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4893 "_msarg");
4894 }
4895
visitVAStartInst__anonb9e144640811::VarArgPowerPC64Helper4896 void visitVAStartInst(VAStartInst &I) override {
4897 IRBuilder<> IRB(&I);
4898 VAStartInstrumentationList.push_back(&I);
4899 Value *VAListTag = I.getArgOperand(0);
4900 Value *ShadowPtr, *OriginPtr;
4901 const Align Alignment = Align(8);
4902 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4903 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4904 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4905 /* size */ 8, Alignment, false);
4906 }
4907
visitVACopyInst__anonb9e144640811::VarArgPowerPC64Helper4908 void visitVACopyInst(VACopyInst &I) override {
4909 IRBuilder<> IRB(&I);
4910 Value *VAListTag = I.getArgOperand(0);
4911 Value *ShadowPtr, *OriginPtr;
4912 const Align Alignment = Align(8);
4913 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4914 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4915 // Unpoison the whole __va_list_tag.
4916 // FIXME: magic ABI constants.
4917 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4918 /* size */ 8, Alignment, false);
4919 }
4920
finalizeInstrumentation__anonb9e144640811::VarArgPowerPC64Helper4921 void finalizeInstrumentation() override {
4922 assert(!VAArgSize && !VAArgTLSCopy &&
4923 "finalizeInstrumentation called twice");
4924 IRBuilder<> IRB(MSV.FnPrologueEnd);
4925 VAArgSize = IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4926 Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
4927 VAArgSize);
4928
4929 if (!VAStartInstrumentationList.empty()) {
4930 // If there is a va_start in this function, make a backup copy of
4931 // va_arg_tls somewhere in the function entry block.
4932 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4933 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
4934 }
4935
4936 // Instrument va_start.
4937 // Copy va_list shadow from the backup copy of the TLS contents.
4938 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4939 CallInst *OrigInst = VAStartInstrumentationList[i];
4940 IRBuilder<> IRB(OrigInst->getNextNode());
4941 Value *VAListTag = OrigInst->getArgOperand(0);
4942 Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4943 Value *RegSaveAreaPtrPtr =
4944 IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4945 PointerType::get(RegSaveAreaPtrTy, 0));
4946 Value *RegSaveAreaPtr =
4947 IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4948 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4949 const Align Alignment = Align(8);
4950 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4951 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4952 Alignment, /*isStore*/ true);
4953 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4954 CopySize);
4955 }
4956 }
4957 };
4958
4959 /// SystemZ-specific implementation of VarArgHelper.
4960 struct VarArgSystemZHelper : public VarArgHelper {
4961 static const unsigned SystemZGpOffset = 16;
4962 static const unsigned SystemZGpEndOffset = 56;
4963 static const unsigned SystemZFpOffset = 128;
4964 static const unsigned SystemZFpEndOffset = 160;
4965 static const unsigned SystemZMaxVrArgs = 8;
4966 static const unsigned SystemZRegSaveAreaSize = 160;
4967 static const unsigned SystemZOverflowOffset = 160;
4968 static const unsigned SystemZVAListTagSize = 32;
4969 static const unsigned SystemZOverflowArgAreaPtrOffset = 16;
4970 static const unsigned SystemZRegSaveAreaPtrOffset = 24;
4971
4972 Function &F;
4973 MemorySanitizer &MS;
4974 MemorySanitizerVisitor &MSV;
4975 Value *VAArgTLSCopy = nullptr;
4976 Value *VAArgTLSOriginCopy = nullptr;
4977 Value *VAArgOverflowSize = nullptr;
4978
4979 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4980
4981 enum class ArgKind {
4982 GeneralPurpose,
4983 FloatingPoint,
4984 Vector,
4985 Memory,
4986 Indirect,
4987 };
4988
4989 enum class ShadowExtension { None, Zero, Sign };
4990
VarArgSystemZHelper__anonb9e144640811::VarArgSystemZHelper4991 VarArgSystemZHelper(Function &F, MemorySanitizer &MS,
4992 MemorySanitizerVisitor &MSV)
4993 : F(F), MS(MS), MSV(MSV) {}
4994
classifyArgument__anonb9e144640811::VarArgSystemZHelper4995 ArgKind classifyArgument(Type *T, bool IsSoftFloatABI) {
4996 // T is a SystemZABIInfo::classifyArgumentType() output, and there are
4997 // only a few possibilities of what it can be. In particular, enums, single
4998 // element structs and large types have already been taken care of.
4999
5000 // Some i128 and fp128 arguments are converted to pointers only in the
5001 // back end.
5002 if (T->isIntegerTy(128) || T->isFP128Ty())
5003 return ArgKind::Indirect;
5004 if (T->isFloatingPointTy())
5005 return IsSoftFloatABI ? ArgKind::GeneralPurpose : ArgKind::FloatingPoint;
5006 if (T->isIntegerTy() || T->isPointerTy())
5007 return ArgKind::GeneralPurpose;
5008 if (T->isVectorTy())
5009 return ArgKind::Vector;
5010 return ArgKind::Memory;
5011 }
5012
getShadowExtension__anonb9e144640811::VarArgSystemZHelper5013 ShadowExtension getShadowExtension(const CallBase &CB, unsigned ArgNo) {
5014 // ABI says: "One of the simple integer types no more than 64 bits wide.
5015 // ... If such an argument is shorter than 64 bits, replace it by a full
5016 // 64-bit integer representing the same number, using sign or zero
5017 // extension". Shadow for an integer argument has the same type as the
5018 // argument itself, so it can be sign or zero extended as well.
5019 bool ZExt = CB.paramHasAttr(ArgNo, Attribute::ZExt);
5020 bool SExt = CB.paramHasAttr(ArgNo, Attribute::SExt);
5021 if (ZExt) {
5022 assert(!SExt);
5023 return ShadowExtension::Zero;
5024 }
5025 if (SExt) {
5026 assert(!ZExt);
5027 return ShadowExtension::Sign;
5028 }
5029 return ShadowExtension::None;
5030 }
5031
visitCallBase__anonb9e144640811::VarArgSystemZHelper5032 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
5033 bool IsSoftFloatABI = CB.getCalledFunction()
5034 ->getFnAttribute("use-soft-float")
5035 .getValueAsBool();
5036 unsigned GpOffset = SystemZGpOffset;
5037 unsigned FpOffset = SystemZFpOffset;
5038 unsigned VrIndex = 0;
5039 unsigned OverflowOffset = SystemZOverflowOffset;
5040 const DataLayout &DL = F.getParent()->getDataLayout();
5041 for (auto ArgIt = CB.arg_begin(), End = CB.arg_end(); ArgIt != End;
5042 ++ArgIt) {
5043 Value *A = *ArgIt;
5044 unsigned ArgNo = CB.getArgOperandNo(ArgIt);
5045 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
5046 // SystemZABIInfo does not produce ByVal parameters.
5047 assert(!CB.paramHasAttr(ArgNo, Attribute::ByVal));
5048 Type *T = A->getType();
5049 ArgKind AK = classifyArgument(T, IsSoftFloatABI);
5050 if (AK == ArgKind::Indirect) {
5051 T = PointerType::get(T, 0);
5052 AK = ArgKind::GeneralPurpose;
5053 }
5054 if (AK == ArgKind::GeneralPurpose && GpOffset >= SystemZGpEndOffset)
5055 AK = ArgKind::Memory;
5056 if (AK == ArgKind::FloatingPoint && FpOffset >= SystemZFpEndOffset)
5057 AK = ArgKind::Memory;
5058 if (AK == ArgKind::Vector && (VrIndex >= SystemZMaxVrArgs || !IsFixed))
5059 AK = ArgKind::Memory;
5060 Value *ShadowBase = nullptr;
5061 Value *OriginBase = nullptr;
5062 ShadowExtension SE = ShadowExtension::None;
5063 switch (AK) {
5064 case ArgKind::GeneralPurpose: {
5065 // Always keep track of GpOffset, but store shadow only for varargs.
5066 uint64_t ArgSize = 8;
5067 if (GpOffset + ArgSize <= kParamTLSSize) {
5068 if (!IsFixed) {
5069 SE = getShadowExtension(CB, ArgNo);
5070 uint64_t GapSize = 0;
5071 if (SE == ShadowExtension::None) {
5072 uint64_t ArgAllocSize = DL.getTypeAllocSize(T);
5073 assert(ArgAllocSize <= ArgSize);
5074 GapSize = ArgSize - ArgAllocSize;
5075 }
5076 ShadowBase = getShadowAddrForVAArgument(IRB, GpOffset + GapSize);
5077 if (MS.TrackOrigins)
5078 OriginBase = getOriginPtrForVAArgument(IRB, GpOffset + GapSize);
5079 }
5080 GpOffset += ArgSize;
5081 } else {
5082 GpOffset = kParamTLSSize;
5083 }
5084 break;
5085 }
5086 case ArgKind::FloatingPoint: {
5087 // Always keep track of FpOffset, but store shadow only for varargs.
5088 uint64_t ArgSize = 8;
5089 if (FpOffset + ArgSize <= kParamTLSSize) {
5090 if (!IsFixed) {
5091 // PoP says: "A short floating-point datum requires only the
5092 // left-most 32 bit positions of a floating-point register".
5093 // Therefore, in contrast to AK_GeneralPurpose and AK_Memory,
5094 // don't extend shadow and don't mind the gap.
5095 ShadowBase = getShadowAddrForVAArgument(IRB, FpOffset);
5096 if (MS.TrackOrigins)
5097 OriginBase = getOriginPtrForVAArgument(IRB, FpOffset);
5098 }
5099 FpOffset += ArgSize;
5100 } else {
5101 FpOffset = kParamTLSSize;
5102 }
5103 break;
5104 }
5105 case ArgKind::Vector: {
5106 // Keep track of VrIndex. No need to store shadow, since vector varargs
5107 // go through AK_Memory.
5108 assert(IsFixed);
5109 VrIndex++;
5110 break;
5111 }
5112 case ArgKind::Memory: {
5113 // Keep track of OverflowOffset and store shadow only for varargs.
5114 // Ignore fixed args, since we need to copy only the vararg portion of
5115 // the overflow area shadow.
5116 if (!IsFixed) {
5117 uint64_t ArgAllocSize = DL.getTypeAllocSize(T);
5118 uint64_t ArgSize = alignTo(ArgAllocSize, 8);
5119 if (OverflowOffset + ArgSize <= kParamTLSSize) {
5120 SE = getShadowExtension(CB, ArgNo);
5121 uint64_t GapSize =
5122 SE == ShadowExtension::None ? ArgSize - ArgAllocSize : 0;
5123 ShadowBase =
5124 getShadowAddrForVAArgument(IRB, OverflowOffset + GapSize);
5125 if (MS.TrackOrigins)
5126 OriginBase =
5127 getOriginPtrForVAArgument(IRB, OverflowOffset + GapSize);
5128 OverflowOffset += ArgSize;
5129 } else {
5130 OverflowOffset = kParamTLSSize;
5131 }
5132 }
5133 break;
5134 }
5135 case ArgKind::Indirect:
5136 llvm_unreachable("Indirect must be converted to GeneralPurpose");
5137 }
5138 if (ShadowBase == nullptr)
5139 continue;
5140 Value *Shadow = MSV.getShadow(A);
5141 if (SE != ShadowExtension::None)
5142 Shadow = MSV.CreateShadowCast(IRB, Shadow, IRB.getInt64Ty(),
5143 /*Signed*/ SE == ShadowExtension::Sign);
5144 ShadowBase = IRB.CreateIntToPtr(
5145 ShadowBase, PointerType::get(Shadow->getType(), 0), "_msarg_va_s");
5146 IRB.CreateStore(Shadow, ShadowBase);
5147 if (MS.TrackOrigins) {
5148 Value *Origin = MSV.getOrigin(A);
5149 unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
5150 MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
5151 kMinOriginAlignment);
5152 }
5153 }
5154 Constant *OverflowSize = ConstantInt::get(
5155 IRB.getInt64Ty(), OverflowOffset - SystemZOverflowOffset);
5156 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
5157 }
5158
getShadowAddrForVAArgument__anonb9e144640811::VarArgSystemZHelper5159 Value *getShadowAddrForVAArgument(IRBuilder<> &IRB, unsigned ArgOffset) {
5160 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
5161 return IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
5162 }
5163
getOriginPtrForVAArgument__anonb9e144640811::VarArgSystemZHelper5164 Value *getOriginPtrForVAArgument(IRBuilder<> &IRB, int ArgOffset) {
5165 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
5166 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
5167 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
5168 "_msarg_va_o");
5169 }
5170
unpoisonVAListTagForInst__anonb9e144640811::VarArgSystemZHelper5171 void unpoisonVAListTagForInst(IntrinsicInst &I) {
5172 IRBuilder<> IRB(&I);
5173 Value *VAListTag = I.getArgOperand(0);
5174 Value *ShadowPtr, *OriginPtr;
5175 const Align Alignment = Align(8);
5176 std::tie(ShadowPtr, OriginPtr) =
5177 MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
5178 /*isStore*/ true);
5179 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
5180 SystemZVAListTagSize, Alignment, false);
5181 }
5182
visitVAStartInst__anonb9e144640811::VarArgSystemZHelper5183 void visitVAStartInst(VAStartInst &I) override {
5184 VAStartInstrumentationList.push_back(&I);
5185 unpoisonVAListTagForInst(I);
5186 }
5187
visitVACopyInst__anonb9e144640811::VarArgSystemZHelper5188 void visitVACopyInst(VACopyInst &I) override { unpoisonVAListTagForInst(I); }
5189
copyRegSaveArea__anonb9e144640811::VarArgSystemZHelper5190 void copyRegSaveArea(IRBuilder<> &IRB, Value *VAListTag) {
5191 Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
5192 Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
5193 IRB.CreateAdd(
5194 IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
5195 ConstantInt::get(MS.IntptrTy, SystemZRegSaveAreaPtrOffset)),
5196 PointerType::get(RegSaveAreaPtrTy, 0));
5197 Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
5198 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
5199 const Align Alignment = Align(8);
5200 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
5201 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(), Alignment,
5202 /*isStore*/ true);
5203 // TODO(iii): copy only fragments filled by visitCallBase()
5204 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
5205 SystemZRegSaveAreaSize);
5206 if (MS.TrackOrigins)
5207 IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
5208 Alignment, SystemZRegSaveAreaSize);
5209 }
5210
copyOverflowArea__anonb9e144640811::VarArgSystemZHelper5211 void copyOverflowArea(IRBuilder<> &IRB, Value *VAListTag) {
5212 Type *OverflowArgAreaPtrTy = Type::getInt64PtrTy(*MS.C);
5213 Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
5214 IRB.CreateAdd(
5215 IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
5216 ConstantInt::get(MS.IntptrTy, SystemZOverflowArgAreaPtrOffset)),
5217 PointerType::get(OverflowArgAreaPtrTy, 0));
5218 Value *OverflowArgAreaPtr =
5219 IRB.CreateLoad(OverflowArgAreaPtrTy, OverflowArgAreaPtrPtr);
5220 Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
5221 const Align Alignment = Align(8);
5222 std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
5223 MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
5224 Alignment, /*isStore*/ true);
5225 Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
5226 SystemZOverflowOffset);
5227 IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
5228 VAArgOverflowSize);
5229 if (MS.TrackOrigins) {
5230 SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSOriginCopy,
5231 SystemZOverflowOffset);
5232 IRB.CreateMemCpy(OverflowArgAreaOriginPtr, Alignment, SrcPtr, Alignment,
5233 VAArgOverflowSize);
5234 }
5235 }
5236
finalizeInstrumentation__anonb9e144640811::VarArgSystemZHelper5237 void finalizeInstrumentation() override {
5238 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
5239 "finalizeInstrumentation called twice");
5240 if (!VAStartInstrumentationList.empty()) {
5241 // If there is a va_start in this function, make a backup copy of
5242 // va_arg_tls somewhere in the function entry block.
5243 IRBuilder<> IRB(MSV.FnPrologueEnd);
5244 VAArgOverflowSize =
5245 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
5246 Value *CopySize =
5247 IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, SystemZOverflowOffset),
5248 VAArgOverflowSize);
5249 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
5250 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
5251 if (MS.TrackOrigins) {
5252 VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
5253 IRB.CreateMemCpy(VAArgTLSOriginCopy, Align(8), MS.VAArgOriginTLS,
5254 Align(8), CopySize);
5255 }
5256 }
5257
5258 // Instrument va_start.
5259 // Copy va_list shadow from the backup copy of the TLS contents.
5260 for (size_t VaStartNo = 0, VaStartNum = VAStartInstrumentationList.size();
5261 VaStartNo < VaStartNum; VaStartNo++) {
5262 CallInst *OrigInst = VAStartInstrumentationList[VaStartNo];
5263 IRBuilder<> IRB(OrigInst->getNextNode());
5264 Value *VAListTag = OrigInst->getArgOperand(0);
5265 copyRegSaveArea(IRB, VAListTag);
5266 copyOverflowArea(IRB, VAListTag);
5267 }
5268 }
5269 };
5270
5271 /// A no-op implementation of VarArgHelper.
5272 struct VarArgNoOpHelper : public VarArgHelper {
VarArgNoOpHelper__anonb9e144640811::VarArgNoOpHelper5273 VarArgNoOpHelper(Function &F, MemorySanitizer &MS,
5274 MemorySanitizerVisitor &MSV) {}
5275
visitCallBase__anonb9e144640811::VarArgNoOpHelper5276 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {}
5277
visitVAStartInst__anonb9e144640811::VarArgNoOpHelper5278 void visitVAStartInst(VAStartInst &I) override {}
5279
visitVACopyInst__anonb9e144640811::VarArgNoOpHelper5280 void visitVACopyInst(VACopyInst &I) override {}
5281
finalizeInstrumentation__anonb9e144640811::VarArgNoOpHelper5282 void finalizeInstrumentation() override {}
5283 };
5284
5285 } // end anonymous namespace
5286
CreateVarArgHelper(Function & Func,MemorySanitizer & Msan,MemorySanitizerVisitor & Visitor)5287 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
5288 MemorySanitizerVisitor &Visitor) {
5289 // VarArg handling is only implemented on AMD64. False positives are possible
5290 // on other platforms.
5291 Triple TargetTriple(Func.getParent()->getTargetTriple());
5292 if (TargetTriple.getArch() == Triple::x86_64)
5293 return new VarArgAMD64Helper(Func, Msan, Visitor);
5294 else if (TargetTriple.isMIPS64())
5295 return new VarArgMIPS64Helper(Func, Msan, Visitor);
5296 else if (TargetTriple.getArch() == Triple::aarch64)
5297 return new VarArgAArch64Helper(Func, Msan, Visitor);
5298 else if (TargetTriple.getArch() == Triple::ppc64 ||
5299 TargetTriple.getArch() == Triple::ppc64le)
5300 return new VarArgPowerPC64Helper(Func, Msan, Visitor);
5301 else if (TargetTriple.getArch() == Triple::systemz)
5302 return new VarArgSystemZHelper(Func, Msan, Visitor);
5303 else
5304 return new VarArgNoOpHelper(Func, Msan, Visitor);
5305 }
5306
sanitizeFunction(Function & F,TargetLibraryInfo & TLI)5307 bool MemorySanitizer::sanitizeFunction(Function &F, TargetLibraryInfo &TLI) {
5308 if (!CompileKernel && F.getName() == kMsanModuleCtorName)
5309 return false;
5310
5311 MemorySanitizerVisitor Visitor(F, *this, TLI);
5312
5313 // Clear out readonly/readnone attributes.
5314 AttrBuilder B;
5315 B.addAttribute(Attribute::ReadOnly)
5316 .addAttribute(Attribute::ReadNone)
5317 .addAttribute(Attribute::WriteOnly)
5318 .addAttribute(Attribute::ArgMemOnly)
5319 .addAttribute(Attribute::Speculatable);
5320 F.removeAttributes(AttributeList::FunctionIndex, B);
5321
5322 return Visitor.runOnFunction();
5323 }
5324