1 /* Instruction scheduling pass. 2 Copyright (C) 1992-2018 Free Software Foundation, Inc. 3 Contributed by Michael Tiemann (tiemann@cygnus.com) Enhanced by, 4 and currently maintained by, Jim Wilson (wilson@cygnus.com) 5 6 This file is part of GCC. 7 8 GCC is free software; you can redistribute it and/or modify it under 9 the terms of the GNU General Public License as published by the Free 10 Software Foundation; either version 3, or (at your option) any later 11 version. 12 13 GCC is distributed in the hope that it will be useful, but WITHOUT ANY 14 WARRANTY; without even the implied warranty of MERCHANTABILITY or 15 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License 16 for more details. 17 18 You should have received a copy of the GNU General Public License 19 along with GCC; see the file COPYING3. If not see 20 <http://www.gnu.org/licenses/>. */ 21 22 /* Instruction scheduling pass. This file, along with sched-deps.c, 23 contains the generic parts. The actual entry point for 24 the normal instruction scheduling pass is found in sched-rgn.c. 25 26 We compute insn priorities based on data dependencies. Flow 27 analysis only creates a fraction of the data-dependencies we must 28 observe: namely, only those dependencies which the combiner can be 29 expected to use. For this pass, we must therefore create the 30 remaining dependencies we need to observe: register dependencies, 31 memory dependencies, dependencies to keep function calls in order, 32 and the dependence between a conditional branch and the setting of 33 condition codes are all dealt with here. 34 35 The scheduler first traverses the data flow graph, starting with 36 the last instruction, and proceeding to the first, assigning values 37 to insn_priority as it goes. This sorts the instructions 38 topologically by data dependence. 39 40 Once priorities have been established, we order the insns using 41 list scheduling. This works as follows: starting with a list of 42 all the ready insns, and sorted according to priority number, we 43 schedule the insn from the end of the list by placing its 44 predecessors in the list according to their priority order. We 45 consider this insn scheduled by setting the pointer to the "end" of 46 the list to point to the previous insn. When an insn has no 47 predecessors, we either queue it until sufficient time has elapsed 48 or add it to the ready list. As the instructions are scheduled or 49 when stalls are introduced, the queue advances and dumps insns into 50 the ready list. When all insns down to the lowest priority have 51 been scheduled, the critical path of the basic block has been made 52 as short as possible. The remaining insns are then scheduled in 53 remaining slots. 54 55 The following list shows the order in which we want to break ties 56 among insns in the ready list: 57 58 1. choose insn with the longest path to end of bb, ties 59 broken by 60 2. choose insn with least contribution to register pressure, 61 ties broken by 62 3. prefer in-block upon interblock motion, ties broken by 63 4. prefer useful upon speculative motion, ties broken by 64 5. choose insn with largest control flow probability, ties 65 broken by 66 6. choose insn with the least dependences upon the previously 67 scheduled insn, or finally 68 7 choose the insn which has the most insns dependent on it. 69 8. choose insn with lowest UID. 70 71 Memory references complicate matters. Only if we can be certain 72 that memory references are not part of the data dependency graph 73 (via true, anti, or output dependence), can we move operations past 74 memory references. To first approximation, reads can be done 75 independently, while writes introduce dependencies. Better 76 approximations will yield fewer dependencies. 77 78 Before reload, an extended analysis of interblock data dependences 79 is required for interblock scheduling. This is performed in 80 compute_block_dependences (). 81 82 Dependencies set up by memory references are treated in exactly the 83 same way as other dependencies, by using insn backward dependences 84 INSN_BACK_DEPS. INSN_BACK_DEPS are translated into forward dependences 85 INSN_FORW_DEPS for the purpose of forward list scheduling. 86 87 Having optimized the critical path, we may have also unduly 88 extended the lifetimes of some registers. If an operation requires 89 that constants be loaded into registers, it is certainly desirable 90 to load those constants as early as necessary, but no earlier. 91 I.e., it will not do to load up a bunch of registers at the 92 beginning of a basic block only to use them at the end, if they 93 could be loaded later, since this may result in excessive register 94 utilization. 95 96 Note that since branches are never in basic blocks, but only end 97 basic blocks, this pass will not move branches. But that is ok, 98 since we can use GNU's delayed branch scheduling pass to take care 99 of this case. 100 101 Also note that no further optimizations based on algebraic 102 identities are performed, so this pass would be a good one to 103 perform instruction splitting, such as breaking up a multiply 104 instruction into shifts and adds where that is profitable. 105 106 Given the memory aliasing analysis that this pass should perform, 107 it should be possible to remove redundant stores to memory, and to 108 load values from registers instead of hitting memory. 109 110 Before reload, speculative insns are moved only if a 'proof' exists 111 that no exception will be caused by this, and if no live registers 112 exist that inhibit the motion (live registers constraints are not 113 represented by data dependence edges). 114 115 This pass must update information that subsequent passes expect to 116 be correct. Namely: reg_n_refs, reg_n_sets, reg_n_deaths, 117 reg_n_calls_crossed, and reg_live_length. Also, BB_HEAD, BB_END. 118 119 The information in the line number notes is carefully retained by 120 this pass. Notes that refer to the starting and ending of 121 exception regions are also carefully retained by this pass. All 122 other NOTE insns are grouped in their same relative order at the 123 beginning of basic blocks and regions that have been scheduled. */ 124 125 #include "config.h" 126 #include "system.h" 127 #include "coretypes.h" 128 #include "backend.h" 129 #include "target.h" 130 #include "rtl.h" 131 #include "cfghooks.h" 132 #include "df.h" 133 #include "memmodel.h" 134 #include "tm_p.h" 135 #include "insn-config.h" 136 #include "regs.h" 137 #include "ira.h" 138 #include "recog.h" 139 #include "insn-attr.h" 140 #include "cfgrtl.h" 141 #include "cfgbuild.h" 142 #include "sched-int.h" 143 #include "common/common-target.h" 144 #include "params.h" 145 #include "dbgcnt.h" 146 #include "cfgloop.h" 147 #include "dumpfile.h" 148 #include "print-rtl.h" 149 150 #ifdef INSN_SCHEDULING 151 152 /* True if we do register pressure relief through live-range 153 shrinkage. */ 154 static bool live_range_shrinkage_p; 155 156 /* Switch on live range shrinkage. */ 157 void 158 initialize_live_range_shrinkage (void) 159 { 160 live_range_shrinkage_p = true; 161 } 162 163 /* Switch off live range shrinkage. */ 164 void 165 finish_live_range_shrinkage (void) 166 { 167 live_range_shrinkage_p = false; 168 } 169 170 /* issue_rate is the number of insns that can be scheduled in the same 171 machine cycle. It can be defined in the config/mach/mach.h file, 172 otherwise we set it to 1. */ 173 174 int issue_rate; 175 176 /* This can be set to true by a backend if the scheduler should not 177 enable a DCE pass. */ 178 bool sched_no_dce; 179 180 /* The current initiation interval used when modulo scheduling. */ 181 static int modulo_ii; 182 183 /* The maximum number of stages we are prepared to handle. */ 184 static int modulo_max_stages; 185 186 /* The number of insns that exist in each iteration of the loop. We use this 187 to detect when we've scheduled all insns from the first iteration. */ 188 static int modulo_n_insns; 189 190 /* The current count of insns in the first iteration of the loop that have 191 already been scheduled. */ 192 static int modulo_insns_scheduled; 193 194 /* The maximum uid of insns from the first iteration of the loop. */ 195 static int modulo_iter0_max_uid; 196 197 /* The number of times we should attempt to backtrack when modulo scheduling. 198 Decreased each time we have to backtrack. */ 199 static int modulo_backtracks_left; 200 201 /* The stage in which the last insn from the original loop was 202 scheduled. */ 203 static int modulo_last_stage; 204 205 /* sched-verbose controls the amount of debugging output the 206 scheduler prints. It is controlled by -fsched-verbose=N: 207 N=0: no debugging output. 208 N=1: default value. 209 N=2: bb's probabilities, detailed ready list info, unit/insn info. 210 N=3: rtl at abort point, control-flow, regions info. 211 N=5: dependences info. */ 212 int sched_verbose = 0; 213 214 /* Debugging file. All printouts are sent to dump. */ 215 FILE *sched_dump = 0; 216 217 /* This is a placeholder for the scheduler parameters common 218 to all schedulers. */ 219 struct common_sched_info_def *common_sched_info; 220 221 #define INSN_TICK(INSN) (HID (INSN)->tick) 222 #define INSN_EXACT_TICK(INSN) (HID (INSN)->exact_tick) 223 #define INSN_TICK_ESTIMATE(INSN) (HID (INSN)->tick_estimate) 224 #define INTER_TICK(INSN) (HID (INSN)->inter_tick) 225 #define FEEDS_BACKTRACK_INSN(INSN) (HID (INSN)->feeds_backtrack_insn) 226 #define SHADOW_P(INSN) (HID (INSN)->shadow_p) 227 #define MUST_RECOMPUTE_SPEC_P(INSN) (HID (INSN)->must_recompute_spec) 228 /* Cached cost of the instruction. Use insn_sched_cost to get cost of the 229 insn. -1 here means that the field is not initialized. */ 230 #define INSN_COST(INSN) (HID (INSN)->cost) 231 232 /* If INSN_TICK of an instruction is equal to INVALID_TICK, 233 then it should be recalculated from scratch. */ 234 #define INVALID_TICK (-(max_insn_queue_index + 1)) 235 /* The minimal value of the INSN_TICK of an instruction. */ 236 #define MIN_TICK (-max_insn_queue_index) 237 238 /* Original order of insns in the ready list. 239 Used to keep order of normal insns while separating DEBUG_INSNs. */ 240 #define INSN_RFS_DEBUG_ORIG_ORDER(INSN) (HID (INSN)->rfs_debug_orig_order) 241 242 /* The deciding reason for INSN's place in the ready list. */ 243 #define INSN_LAST_RFS_WIN(INSN) (HID (INSN)->last_rfs_win) 244 245 /* List of important notes we must keep around. This is a pointer to the 246 last element in the list. */ 247 rtx_insn *note_list; 248 249 static struct spec_info_def spec_info_var; 250 /* Description of the speculative part of the scheduling. 251 If NULL - no speculation. */ 252 spec_info_t spec_info = NULL; 253 254 /* True, if recovery block was added during scheduling of current block. 255 Used to determine, if we need to fix INSN_TICKs. */ 256 static bool haifa_recovery_bb_recently_added_p; 257 258 /* True, if recovery block was added during this scheduling pass. 259 Used to determine if we should have empty memory pools of dependencies 260 after finishing current region. */ 261 bool haifa_recovery_bb_ever_added_p; 262 263 /* Counters of different types of speculative instructions. */ 264 static int nr_begin_data, nr_be_in_data, nr_begin_control, nr_be_in_control; 265 266 /* Array used in {unlink, restore}_bb_notes. */ 267 static rtx_insn **bb_header = 0; 268 269 /* Basic block after which recovery blocks will be created. */ 270 static basic_block before_recovery; 271 272 /* Basic block just before the EXIT_BLOCK and after recovery, if we have 273 created it. */ 274 basic_block after_recovery; 275 276 /* FALSE if we add bb to another region, so we don't need to initialize it. */ 277 bool adding_bb_to_current_region_p = true; 278 279 /* Queues, etc. */ 280 281 /* An instruction is ready to be scheduled when all insns preceding it 282 have already been scheduled. It is important to ensure that all 283 insns which use its result will not be executed until its result 284 has been computed. An insn is maintained in one of four structures: 285 286 (P) the "Pending" set of insns which cannot be scheduled until 287 their dependencies have been satisfied. 288 (Q) the "Queued" set of insns that can be scheduled when sufficient 289 time has passed. 290 (R) the "Ready" list of unscheduled, uncommitted insns. 291 (S) the "Scheduled" list of insns. 292 293 Initially, all insns are either "Pending" or "Ready" depending on 294 whether their dependencies are satisfied. 295 296 Insns move from the "Ready" list to the "Scheduled" list as they 297 are committed to the schedule. As this occurs, the insns in the 298 "Pending" list have their dependencies satisfied and move to either 299 the "Ready" list or the "Queued" set depending on whether 300 sufficient time has passed to make them ready. As time passes, 301 insns move from the "Queued" set to the "Ready" list. 302 303 The "Pending" list (P) are the insns in the INSN_FORW_DEPS of the 304 unscheduled insns, i.e., those that are ready, queued, and pending. 305 The "Queued" set (Q) is implemented by the variable `insn_queue'. 306 The "Ready" list (R) is implemented by the variables `ready' and 307 `n_ready'. 308 The "Scheduled" list (S) is the new insn chain built by this pass. 309 310 The transition (R->S) is implemented in the scheduling loop in 311 `schedule_block' when the best insn to schedule is chosen. 312 The transitions (P->R and P->Q) are implemented in `schedule_insn' as 313 insns move from the ready list to the scheduled list. 314 The transition (Q->R) is implemented in 'queue_to_insn' as time 315 passes or stalls are introduced. */ 316 317 /* Implement a circular buffer to delay instructions until sufficient 318 time has passed. For the new pipeline description interface, 319 MAX_INSN_QUEUE_INDEX is a power of two minus one which is not less 320 than maximal time of instruction execution computed by genattr.c on 321 the base maximal time of functional unit reservations and getting a 322 result. This is the longest time an insn may be queued. */ 323 324 static rtx_insn_list **insn_queue; 325 static int q_ptr = 0; 326 static int q_size = 0; 327 #define NEXT_Q(X) (((X)+1) & max_insn_queue_index) 328 #define NEXT_Q_AFTER(X, C) (((X)+C) & max_insn_queue_index) 329 330 #define QUEUE_SCHEDULED (-3) 331 #define QUEUE_NOWHERE (-2) 332 #define QUEUE_READY (-1) 333 /* QUEUE_SCHEDULED - INSN is scheduled. 334 QUEUE_NOWHERE - INSN isn't scheduled yet and is neither in 335 queue or ready list. 336 QUEUE_READY - INSN is in ready list. 337 N >= 0 - INSN queued for X [where NEXT_Q_AFTER (q_ptr, X) == N] cycles. */ 338 339 #define QUEUE_INDEX(INSN) (HID (INSN)->queue_index) 340 341 /* The following variable value refers for all current and future 342 reservations of the processor units. */ 343 state_t curr_state; 344 345 /* The following variable value is size of memory representing all 346 current and future reservations of the processor units. */ 347 size_t dfa_state_size; 348 349 /* The following array is used to find the best insn from ready when 350 the automaton pipeline interface is used. */ 351 signed char *ready_try = NULL; 352 353 /* The ready list. */ 354 struct ready_list ready = {NULL, 0, 0, 0, 0}; 355 356 /* The pointer to the ready list (to be removed). */ 357 static struct ready_list *readyp = &ready; 358 359 /* Scheduling clock. */ 360 static int clock_var; 361 362 /* Clock at which the previous instruction was issued. */ 363 static int last_clock_var; 364 365 /* Set to true if, when queuing a shadow insn, we discover that it would be 366 scheduled too late. */ 367 static bool must_backtrack; 368 369 /* The following variable value is number of essential insns issued on 370 the current cycle. An insn is essential one if it changes the 371 processors state. */ 372 int cycle_issued_insns; 373 374 /* This records the actual schedule. It is built up during the main phase 375 of schedule_block, and afterwards used to reorder the insns in the RTL. */ 376 static vec<rtx_insn *> scheduled_insns; 377 378 static int may_trap_exp (const_rtx, int); 379 380 /* Nonzero iff the address is comprised from at most 1 register. */ 381 #define CONST_BASED_ADDRESS_P(x) \ 382 (REG_P (x) \ 383 || ((GET_CODE (x) == PLUS || GET_CODE (x) == MINUS \ 384 || (GET_CODE (x) == LO_SUM)) \ 385 && (CONSTANT_P (XEXP (x, 0)) \ 386 || CONSTANT_P (XEXP (x, 1))))) 387 388 /* Returns a class that insn with GET_DEST(insn)=x may belong to, 389 as found by analyzing insn's expression. */ 390 391 392 static int haifa_luid_for_non_insn (rtx x); 393 394 /* Haifa version of sched_info hooks common to all headers. */ 395 const struct common_sched_info_def haifa_common_sched_info = 396 { 397 NULL, /* fix_recovery_cfg */ 398 NULL, /* add_block */ 399 NULL, /* estimate_number_of_insns */ 400 haifa_luid_for_non_insn, /* luid_for_non_insn */ 401 SCHED_PASS_UNKNOWN /* sched_pass_id */ 402 }; 403 404 /* Mapping from instruction UID to its Logical UID. */ 405 vec<int> sched_luids; 406 407 /* Next LUID to assign to an instruction. */ 408 int sched_max_luid = 1; 409 410 /* Haifa Instruction Data. */ 411 vec<haifa_insn_data_def> h_i_d; 412 413 void (* sched_init_only_bb) (basic_block, basic_block); 414 415 /* Split block function. Different schedulers might use different functions 416 to handle their internal data consistent. */ 417 basic_block (* sched_split_block) (basic_block, rtx); 418 419 /* Create empty basic block after the specified block. */ 420 basic_block (* sched_create_empty_bb) (basic_block); 421 422 /* Return the number of cycles until INSN is expected to be ready. 423 Return zero if it already is. */ 424 static int 425 insn_delay (rtx_insn *insn) 426 { 427 return MAX (INSN_TICK (insn) - clock_var, 0); 428 } 429 430 static int 431 may_trap_exp (const_rtx x, int is_store) 432 { 433 enum rtx_code code; 434 435 if (x == 0) 436 return TRAP_FREE; 437 code = GET_CODE (x); 438 if (is_store) 439 { 440 if (code == MEM && may_trap_p (x)) 441 return TRAP_RISKY; 442 else 443 return TRAP_FREE; 444 } 445 if (code == MEM) 446 { 447 /* The insn uses memory: a volatile load. */ 448 if (MEM_VOLATILE_P (x)) 449 return IRISKY; 450 /* An exception-free load. */ 451 if (!may_trap_p (x)) 452 return IFREE; 453 /* A load with 1 base register, to be further checked. */ 454 if (CONST_BASED_ADDRESS_P (XEXP (x, 0))) 455 return PFREE_CANDIDATE; 456 /* No info on the load, to be further checked. */ 457 return PRISKY_CANDIDATE; 458 } 459 else 460 { 461 const char *fmt; 462 int i, insn_class = TRAP_FREE; 463 464 /* Neither store nor load, check if it may cause a trap. */ 465 if (may_trap_p (x)) 466 return TRAP_RISKY; 467 /* Recursive step: walk the insn... */ 468 fmt = GET_RTX_FORMAT (code); 469 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--) 470 { 471 if (fmt[i] == 'e') 472 { 473 int tmp_class = may_trap_exp (XEXP (x, i), is_store); 474 insn_class = WORST_CLASS (insn_class, tmp_class); 475 } 476 else if (fmt[i] == 'E') 477 { 478 int j; 479 for (j = 0; j < XVECLEN (x, i); j++) 480 { 481 int tmp_class = may_trap_exp (XVECEXP (x, i, j), is_store); 482 insn_class = WORST_CLASS (insn_class, tmp_class); 483 if (insn_class == TRAP_RISKY || insn_class == IRISKY) 484 break; 485 } 486 } 487 if (insn_class == TRAP_RISKY || insn_class == IRISKY) 488 break; 489 } 490 return insn_class; 491 } 492 } 493 494 /* Classifies rtx X of an insn for the purpose of verifying that X can be 495 executed speculatively (and consequently the insn can be moved 496 speculatively), by examining X, returning: 497 TRAP_RISKY: store, or risky non-load insn (e.g. division by variable). 498 TRAP_FREE: non-load insn. 499 IFREE: load from a globally safe location. 500 IRISKY: volatile load. 501 PFREE_CANDIDATE, PRISKY_CANDIDATE: load that need to be checked for 502 being either PFREE or PRISKY. */ 503 504 static int 505 haifa_classify_rtx (const_rtx x) 506 { 507 int tmp_class = TRAP_FREE; 508 int insn_class = TRAP_FREE; 509 enum rtx_code code; 510 511 if (GET_CODE (x) == PARALLEL) 512 { 513 int i, len = XVECLEN (x, 0); 514 515 for (i = len - 1; i >= 0; i--) 516 { 517 tmp_class = haifa_classify_rtx (XVECEXP (x, 0, i)); 518 insn_class = WORST_CLASS (insn_class, tmp_class); 519 if (insn_class == TRAP_RISKY || insn_class == IRISKY) 520 break; 521 } 522 } 523 else 524 { 525 code = GET_CODE (x); 526 switch (code) 527 { 528 case CLOBBER: 529 /* Test if it is a 'store'. */ 530 tmp_class = may_trap_exp (XEXP (x, 0), 1); 531 break; 532 case SET: 533 /* Test if it is a store. */ 534 tmp_class = may_trap_exp (SET_DEST (x), 1); 535 if (tmp_class == TRAP_RISKY) 536 break; 537 /* Test if it is a load. */ 538 tmp_class = 539 WORST_CLASS (tmp_class, 540 may_trap_exp (SET_SRC (x), 0)); 541 break; 542 case COND_EXEC: 543 tmp_class = haifa_classify_rtx (COND_EXEC_CODE (x)); 544 if (tmp_class == TRAP_RISKY) 545 break; 546 tmp_class = WORST_CLASS (tmp_class, 547 may_trap_exp (COND_EXEC_TEST (x), 0)); 548 break; 549 case TRAP_IF: 550 tmp_class = TRAP_RISKY; 551 break; 552 default:; 553 } 554 insn_class = tmp_class; 555 } 556 557 return insn_class; 558 } 559 560 int 561 haifa_classify_insn (const_rtx insn) 562 { 563 return haifa_classify_rtx (PATTERN (insn)); 564 } 565 566 /* After the scheduler initialization function has been called, this function 567 can be called to enable modulo scheduling. II is the initiation interval 568 we should use, it affects the delays for delay_pairs that were recorded as 569 separated by a given number of stages. 570 571 MAX_STAGES provides us with a limit 572 after which we give up scheduling; the caller must have unrolled at least 573 as many copies of the loop body and recorded delay_pairs for them. 574 575 INSNS is the number of real (non-debug) insns in one iteration of 576 the loop. MAX_UID can be used to test whether an insn belongs to 577 the first iteration of the loop; all of them have a uid lower than 578 MAX_UID. */ 579 void 580 set_modulo_params (int ii, int max_stages, int insns, int max_uid) 581 { 582 modulo_ii = ii; 583 modulo_max_stages = max_stages; 584 modulo_n_insns = insns; 585 modulo_iter0_max_uid = max_uid; 586 modulo_backtracks_left = PARAM_VALUE (PARAM_MAX_MODULO_BACKTRACK_ATTEMPTS); 587 } 588 589 /* A structure to record a pair of insns where the first one is a real 590 insn that has delay slots, and the second is its delayed shadow. 591 I1 is scheduled normally and will emit an assembly instruction, 592 while I2 describes the side effect that takes place at the 593 transition between cycles CYCLES and (CYCLES + 1) after I1. */ 594 struct delay_pair 595 { 596 struct delay_pair *next_same_i1; 597 rtx_insn *i1, *i2; 598 int cycles; 599 /* When doing modulo scheduling, we a delay_pair can also be used to 600 show that I1 and I2 are the same insn in a different stage. If that 601 is the case, STAGES will be nonzero. */ 602 int stages; 603 }; 604 605 /* Helpers for delay hashing. */ 606 607 struct delay_i1_hasher : nofree_ptr_hash <delay_pair> 608 { 609 typedef void *compare_type; 610 static inline hashval_t hash (const delay_pair *); 611 static inline bool equal (const delay_pair *, const void *); 612 }; 613 614 /* Returns a hash value for X, based on hashing just I1. */ 615 616 inline hashval_t 617 delay_i1_hasher::hash (const delay_pair *x) 618 { 619 return htab_hash_pointer (x->i1); 620 } 621 622 /* Return true if I1 of pair X is the same as that of pair Y. */ 623 624 inline bool 625 delay_i1_hasher::equal (const delay_pair *x, const void *y) 626 { 627 return x->i1 == y; 628 } 629 630 struct delay_i2_hasher : free_ptr_hash <delay_pair> 631 { 632 typedef void *compare_type; 633 static inline hashval_t hash (const delay_pair *); 634 static inline bool equal (const delay_pair *, const void *); 635 }; 636 637 /* Returns a hash value for X, based on hashing just I2. */ 638 639 inline hashval_t 640 delay_i2_hasher::hash (const delay_pair *x) 641 { 642 return htab_hash_pointer (x->i2); 643 } 644 645 /* Return true if I2 of pair X is the same as that of pair Y. */ 646 647 inline bool 648 delay_i2_hasher::equal (const delay_pair *x, const void *y) 649 { 650 return x->i2 == y; 651 } 652 653 /* Two hash tables to record delay_pairs, one indexed by I1 and the other 654 indexed by I2. */ 655 static hash_table<delay_i1_hasher> *delay_htab; 656 static hash_table<delay_i2_hasher> *delay_htab_i2; 657 658 /* Called through htab_traverse. Walk the hashtable using I2 as 659 index, and delete all elements involving an UID higher than 660 that pointed to by *DATA. */ 661 int 662 haifa_htab_i2_traverse (delay_pair **slot, int *data) 663 { 664 int maxuid = *data; 665 struct delay_pair *p = *slot; 666 if (INSN_UID (p->i2) >= maxuid || INSN_UID (p->i1) >= maxuid) 667 { 668 delay_htab_i2->clear_slot (slot); 669 } 670 return 1; 671 } 672 673 /* Called through htab_traverse. Walk the hashtable using I2 as 674 index, and delete all elements involving an UID higher than 675 that pointed to by *DATA. */ 676 int 677 haifa_htab_i1_traverse (delay_pair **pslot, int *data) 678 { 679 int maxuid = *data; 680 struct delay_pair *p, *first, **pprev; 681 682 if (INSN_UID ((*pslot)->i1) >= maxuid) 683 { 684 delay_htab->clear_slot (pslot); 685 return 1; 686 } 687 pprev = &first; 688 for (p = *pslot; p; p = p->next_same_i1) 689 { 690 if (INSN_UID (p->i2) < maxuid) 691 { 692 *pprev = p; 693 pprev = &p->next_same_i1; 694 } 695 } 696 *pprev = NULL; 697 if (first == NULL) 698 delay_htab->clear_slot (pslot); 699 else 700 *pslot = first; 701 return 1; 702 } 703 704 /* Discard all delay pairs which involve an insn with an UID higher 705 than MAX_UID. */ 706 void 707 discard_delay_pairs_above (int max_uid) 708 { 709 delay_htab->traverse <int *, haifa_htab_i1_traverse> (&max_uid); 710 delay_htab_i2->traverse <int *, haifa_htab_i2_traverse> (&max_uid); 711 } 712 713 /* This function can be called by a port just before it starts the final 714 scheduling pass. It records the fact that an instruction with delay 715 slots has been split into two insns, I1 and I2. The first one will be 716 scheduled normally and initiates the operation. The second one is a 717 shadow which must follow a specific number of cycles after I1; its only 718 purpose is to show the side effect that occurs at that cycle in the RTL. 719 If a JUMP_INSN or a CALL_INSN has been split, I1 should be a normal INSN, 720 while I2 retains the original insn type. 721 722 There are two ways in which the number of cycles can be specified, 723 involving the CYCLES and STAGES arguments to this function. If STAGES 724 is zero, we just use the value of CYCLES. Otherwise, STAGES is a factor 725 which is multiplied by MODULO_II to give the number of cycles. This is 726 only useful if the caller also calls set_modulo_params to enable modulo 727 scheduling. */ 728 729 void 730 record_delay_slot_pair (rtx_insn *i1, rtx_insn *i2, int cycles, int stages) 731 { 732 struct delay_pair *p = XNEW (struct delay_pair); 733 struct delay_pair **slot; 734 735 p->i1 = i1; 736 p->i2 = i2; 737 p->cycles = cycles; 738 p->stages = stages; 739 740 if (!delay_htab) 741 { 742 delay_htab = new hash_table<delay_i1_hasher> (10); 743 delay_htab_i2 = new hash_table<delay_i2_hasher> (10); 744 } 745 slot = delay_htab->find_slot_with_hash (i1, htab_hash_pointer (i1), INSERT); 746 p->next_same_i1 = *slot; 747 *slot = p; 748 slot = delay_htab_i2->find_slot (p, INSERT); 749 *slot = p; 750 } 751 752 /* Examine the delay pair hashtable to see if INSN is a shadow for another, 753 and return the other insn if so. Return NULL otherwise. */ 754 rtx_insn * 755 real_insn_for_shadow (rtx_insn *insn) 756 { 757 struct delay_pair *pair; 758 759 if (!delay_htab) 760 return NULL; 761 762 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn)); 763 if (!pair || pair->stages > 0) 764 return NULL; 765 return pair->i1; 766 } 767 768 /* For a pair P of insns, return the fixed distance in cycles from the first 769 insn after which the second must be scheduled. */ 770 static int 771 pair_delay (struct delay_pair *p) 772 { 773 if (p->stages == 0) 774 return p->cycles; 775 else 776 return p->stages * modulo_ii; 777 } 778 779 /* Given an insn INSN, add a dependence on its delayed shadow if it 780 has one. Also try to find situations where shadows depend on each other 781 and add dependencies to the real insns to limit the amount of backtracking 782 needed. */ 783 void 784 add_delay_dependencies (rtx_insn *insn) 785 { 786 struct delay_pair *pair; 787 sd_iterator_def sd_it; 788 dep_t dep; 789 790 if (!delay_htab) 791 return; 792 793 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn)); 794 if (!pair) 795 return; 796 add_dependence (insn, pair->i1, REG_DEP_ANTI); 797 if (pair->stages) 798 return; 799 800 FOR_EACH_DEP (pair->i2, SD_LIST_BACK, sd_it, dep) 801 { 802 rtx_insn *pro = DEP_PRO (dep); 803 struct delay_pair *other_pair 804 = delay_htab_i2->find_with_hash (pro, htab_hash_pointer (pro)); 805 if (!other_pair || other_pair->stages) 806 continue; 807 if (pair_delay (other_pair) >= pair_delay (pair)) 808 { 809 if (sched_verbose >= 4) 810 { 811 fprintf (sched_dump, ";;\tadding dependence %d <- %d\n", 812 INSN_UID (other_pair->i1), 813 INSN_UID (pair->i1)); 814 fprintf (sched_dump, ";;\tpair1 %d <- %d, cost %d\n", 815 INSN_UID (pair->i1), 816 INSN_UID (pair->i2), 817 pair_delay (pair)); 818 fprintf (sched_dump, ";;\tpair2 %d <- %d, cost %d\n", 819 INSN_UID (other_pair->i1), 820 INSN_UID (other_pair->i2), 821 pair_delay (other_pair)); 822 } 823 add_dependence (pair->i1, other_pair->i1, REG_DEP_ANTI); 824 } 825 } 826 } 827 828 /* Forward declarations. */ 829 830 static int priority (rtx_insn *); 831 static int autopref_rank_for_schedule (const rtx_insn *, const rtx_insn *); 832 static int rank_for_schedule (const void *, const void *); 833 static void swap_sort (rtx_insn **, int); 834 static void queue_insn (rtx_insn *, int, const char *); 835 static int schedule_insn (rtx_insn *); 836 static void adjust_priority (rtx_insn *); 837 static void advance_one_cycle (void); 838 static void extend_h_i_d (void); 839 840 841 /* Notes handling mechanism: 842 ========================= 843 Generally, NOTES are saved before scheduling and restored after scheduling. 844 The scheduler distinguishes between two types of notes: 845 846 (1) LOOP_BEGIN, LOOP_END, SETJMP, EHREGION_BEG, EHREGION_END notes: 847 Before scheduling a region, a pointer to the note is added to the insn 848 that follows or precedes it. (This happens as part of the data dependence 849 computation). After scheduling an insn, the pointer contained in it is 850 used for regenerating the corresponding note (in reemit_notes). 851 852 (2) All other notes (e.g. INSN_DELETED): Before scheduling a block, 853 these notes are put in a list (in rm_other_notes() and 854 unlink_other_notes ()). After scheduling the block, these notes are 855 inserted at the beginning of the block (in schedule_block()). */ 856 857 static void ready_add (struct ready_list *, rtx_insn *, bool); 858 static rtx_insn *ready_remove_first (struct ready_list *); 859 static rtx_insn *ready_remove_first_dispatch (struct ready_list *ready); 860 861 static void queue_to_ready (struct ready_list *); 862 static int early_queue_to_ready (state_t, struct ready_list *); 863 864 /* The following functions are used to implement multi-pass scheduling 865 on the first cycle. */ 866 static rtx_insn *ready_remove (struct ready_list *, int); 867 static void ready_remove_insn (rtx_insn *); 868 869 static void fix_inter_tick (rtx_insn *, rtx_insn *); 870 static int fix_tick_ready (rtx_insn *); 871 static void change_queue_index (rtx_insn *, int); 872 873 /* The following functions are used to implement scheduling of data/control 874 speculative instructions. */ 875 876 static void extend_h_i_d (void); 877 static void init_h_i_d (rtx_insn *); 878 static int haifa_speculate_insn (rtx_insn *, ds_t, rtx *); 879 static void generate_recovery_code (rtx_insn *); 880 static void process_insn_forw_deps_be_in_spec (rtx_insn *, rtx_insn *, ds_t); 881 static void begin_speculative_block (rtx_insn *); 882 static void add_to_speculative_block (rtx_insn *); 883 static void init_before_recovery (basic_block *); 884 static void create_check_block_twin (rtx_insn *, bool); 885 static void fix_recovery_deps (basic_block); 886 static bool haifa_change_pattern (rtx_insn *, rtx); 887 static void dump_new_block_header (int, basic_block, rtx_insn *, rtx_insn *); 888 static void restore_bb_notes (basic_block); 889 static void fix_jump_move (rtx_insn *); 890 static void move_block_after_check (rtx_insn *); 891 static void move_succs (vec<edge, va_gc> **, basic_block); 892 static void sched_remove_insn (rtx_insn *); 893 static void clear_priorities (rtx_insn *, rtx_vec_t *); 894 static void calc_priorities (rtx_vec_t); 895 static void add_jump_dependencies (rtx_insn *, rtx_insn *); 896 897 #endif /* INSN_SCHEDULING */ 898 899 /* Point to state used for the current scheduling pass. */ 900 struct haifa_sched_info *current_sched_info; 901 902 #ifndef INSN_SCHEDULING 903 void 904 schedule_insns (void) 905 { 906 } 907 #else 908 909 /* Do register pressure sensitive insn scheduling if the flag is set 910 up. */ 911 enum sched_pressure_algorithm sched_pressure; 912 913 /* Map regno -> its pressure class. The map defined only when 914 SCHED_PRESSURE != SCHED_PRESSURE_NONE. */ 915 enum reg_class *sched_regno_pressure_class; 916 917 /* The current register pressure. Only elements corresponding pressure 918 classes are defined. */ 919 static int curr_reg_pressure[N_REG_CLASSES]; 920 921 /* Saved value of the previous array. */ 922 static int saved_reg_pressure[N_REG_CLASSES]; 923 924 /* Register living at given scheduling point. */ 925 static bitmap curr_reg_live; 926 927 /* Saved value of the previous array. */ 928 static bitmap saved_reg_live; 929 930 /* Registers mentioned in the current region. */ 931 static bitmap region_ref_regs; 932 933 /* Temporary bitmap used for SCHED_PRESSURE_MODEL. */ 934 static bitmap tmp_bitmap; 935 936 /* Effective number of available registers of a given class (see comment 937 in sched_pressure_start_bb). */ 938 static int sched_class_regs_num[N_REG_CLASSES]; 939 /* Number of call_saved_regs and fixed_regs. Helpers for calculating of 940 sched_class_regs_num. */ 941 static int call_saved_regs_num[N_REG_CLASSES]; 942 static int fixed_regs_num[N_REG_CLASSES]; 943 944 /* Initiate register pressure relative info for scheduling the current 945 region. Currently it is only clearing register mentioned in the 946 current region. */ 947 void 948 sched_init_region_reg_pressure_info (void) 949 { 950 bitmap_clear (region_ref_regs); 951 } 952 953 /* PRESSURE[CL] describes the pressure on register class CL. Update it 954 for the birth (if BIRTH_P) or death (if !BIRTH_P) of register REGNO. 955 LIVE tracks the set of live registers; if it is null, assume that 956 every birth or death is genuine. */ 957 static inline void 958 mark_regno_birth_or_death (bitmap live, int *pressure, int regno, bool birth_p) 959 { 960 enum reg_class pressure_class; 961 962 pressure_class = sched_regno_pressure_class[regno]; 963 if (regno >= FIRST_PSEUDO_REGISTER) 964 { 965 if (pressure_class != NO_REGS) 966 { 967 if (birth_p) 968 { 969 if (!live || bitmap_set_bit (live, regno)) 970 pressure[pressure_class] 971 += (ira_reg_class_max_nregs 972 [pressure_class][PSEUDO_REGNO_MODE (regno)]); 973 } 974 else 975 { 976 if (!live || bitmap_clear_bit (live, regno)) 977 pressure[pressure_class] 978 -= (ira_reg_class_max_nregs 979 [pressure_class][PSEUDO_REGNO_MODE (regno)]); 980 } 981 } 982 } 983 else if (pressure_class != NO_REGS 984 && ! TEST_HARD_REG_BIT (ira_no_alloc_regs, regno)) 985 { 986 if (birth_p) 987 { 988 if (!live || bitmap_set_bit (live, regno)) 989 pressure[pressure_class]++; 990 } 991 else 992 { 993 if (!live || bitmap_clear_bit (live, regno)) 994 pressure[pressure_class]--; 995 } 996 } 997 } 998 999 /* Initiate current register pressure related info from living 1000 registers given by LIVE. */ 1001 static void 1002 initiate_reg_pressure_info (bitmap live) 1003 { 1004 int i; 1005 unsigned int j; 1006 bitmap_iterator bi; 1007 1008 for (i = 0; i < ira_pressure_classes_num; i++) 1009 curr_reg_pressure[ira_pressure_classes[i]] = 0; 1010 bitmap_clear (curr_reg_live); 1011 EXECUTE_IF_SET_IN_BITMAP (live, 0, j, bi) 1012 if (sched_pressure == SCHED_PRESSURE_MODEL 1013 || current_nr_blocks == 1 1014 || bitmap_bit_p (region_ref_regs, j)) 1015 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, j, true); 1016 } 1017 1018 /* Mark registers in X as mentioned in the current region. */ 1019 static void 1020 setup_ref_regs (rtx x) 1021 { 1022 int i, j; 1023 const RTX_CODE code = GET_CODE (x); 1024 const char *fmt; 1025 1026 if (REG_P (x)) 1027 { 1028 bitmap_set_range (region_ref_regs, REGNO (x), REG_NREGS (x)); 1029 return; 1030 } 1031 fmt = GET_RTX_FORMAT (code); 1032 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--) 1033 if (fmt[i] == 'e') 1034 setup_ref_regs (XEXP (x, i)); 1035 else if (fmt[i] == 'E') 1036 { 1037 for (j = 0; j < XVECLEN (x, i); j++) 1038 setup_ref_regs (XVECEXP (x, i, j)); 1039 } 1040 } 1041 1042 /* Initiate current register pressure related info at the start of 1043 basic block BB. */ 1044 static void 1045 initiate_bb_reg_pressure_info (basic_block bb) 1046 { 1047 unsigned int i ATTRIBUTE_UNUSED; 1048 rtx_insn *insn; 1049 1050 if (current_nr_blocks > 1) 1051 FOR_BB_INSNS (bb, insn) 1052 if (NONDEBUG_INSN_P (insn)) 1053 setup_ref_regs (PATTERN (insn)); 1054 initiate_reg_pressure_info (df_get_live_in (bb)); 1055 if (bb_has_eh_pred (bb)) 1056 for (i = 0; ; ++i) 1057 { 1058 unsigned int regno = EH_RETURN_DATA_REGNO (i); 1059 1060 if (regno == INVALID_REGNUM) 1061 break; 1062 if (! bitmap_bit_p (df_get_live_in (bb), regno)) 1063 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, 1064 regno, true); 1065 } 1066 } 1067 1068 /* Save current register pressure related info. */ 1069 static void 1070 save_reg_pressure (void) 1071 { 1072 int i; 1073 1074 for (i = 0; i < ira_pressure_classes_num; i++) 1075 saved_reg_pressure[ira_pressure_classes[i]] 1076 = curr_reg_pressure[ira_pressure_classes[i]]; 1077 bitmap_copy (saved_reg_live, curr_reg_live); 1078 } 1079 1080 /* Restore saved register pressure related info. */ 1081 static void 1082 restore_reg_pressure (void) 1083 { 1084 int i; 1085 1086 for (i = 0; i < ira_pressure_classes_num; i++) 1087 curr_reg_pressure[ira_pressure_classes[i]] 1088 = saved_reg_pressure[ira_pressure_classes[i]]; 1089 bitmap_copy (curr_reg_live, saved_reg_live); 1090 } 1091 1092 /* Return TRUE if the register is dying after its USE. */ 1093 static bool 1094 dying_use_p (struct reg_use_data *use) 1095 { 1096 struct reg_use_data *next; 1097 1098 for (next = use->next_regno_use; next != use; next = next->next_regno_use) 1099 if (NONDEBUG_INSN_P (next->insn) 1100 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED) 1101 return false; 1102 return true; 1103 } 1104 1105 /* Print info about the current register pressure and its excess for 1106 each pressure class. */ 1107 static void 1108 print_curr_reg_pressure (void) 1109 { 1110 int i; 1111 enum reg_class cl; 1112 1113 fprintf (sched_dump, ";;\t"); 1114 for (i = 0; i < ira_pressure_classes_num; i++) 1115 { 1116 cl = ira_pressure_classes[i]; 1117 gcc_assert (curr_reg_pressure[cl] >= 0); 1118 fprintf (sched_dump, " %s:%d(%d)", reg_class_names[cl], 1119 curr_reg_pressure[cl], 1120 curr_reg_pressure[cl] - sched_class_regs_num[cl]); 1121 } 1122 fprintf (sched_dump, "\n"); 1123 } 1124 1125 /* Determine if INSN has a condition that is clobbered if a register 1126 in SET_REGS is modified. */ 1127 static bool 1128 cond_clobbered_p (rtx_insn *insn, HARD_REG_SET set_regs) 1129 { 1130 rtx pat = PATTERN (insn); 1131 gcc_assert (GET_CODE (pat) == COND_EXEC); 1132 if (TEST_HARD_REG_BIT (set_regs, REGNO (XEXP (COND_EXEC_TEST (pat), 0)))) 1133 { 1134 sd_iterator_def sd_it; 1135 dep_t dep; 1136 haifa_change_pattern (insn, ORIG_PAT (insn)); 1137 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep) 1138 DEP_STATUS (dep) &= ~DEP_CANCELLED; 1139 TODO_SPEC (insn) = HARD_DEP; 1140 if (sched_verbose >= 2) 1141 fprintf (sched_dump, 1142 ";;\t\tdequeue insn %s because of clobbered condition\n", 1143 (*current_sched_info->print_insn) (insn, 0)); 1144 return true; 1145 } 1146 1147 return false; 1148 } 1149 1150 /* This function should be called after modifying the pattern of INSN, 1151 to update scheduler data structures as needed. */ 1152 static void 1153 update_insn_after_change (rtx_insn *insn) 1154 { 1155 sd_iterator_def sd_it; 1156 dep_t dep; 1157 1158 dfa_clear_single_insn_cache (insn); 1159 1160 sd_it = sd_iterator_start (insn, 1161 SD_LIST_FORW | SD_LIST_BACK | SD_LIST_RES_BACK); 1162 while (sd_iterator_cond (&sd_it, &dep)) 1163 { 1164 DEP_COST (dep) = UNKNOWN_DEP_COST; 1165 sd_iterator_next (&sd_it); 1166 } 1167 1168 /* Invalidate INSN_COST, so it'll be recalculated. */ 1169 INSN_COST (insn) = -1; 1170 /* Invalidate INSN_TICK, so it'll be recalculated. */ 1171 INSN_TICK (insn) = INVALID_TICK; 1172 1173 /* Invalidate autoprefetch data entry. */ 1174 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status 1175 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED; 1176 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status 1177 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED; 1178 } 1179 1180 1181 /* Two VECs, one to hold dependencies for which pattern replacements 1182 need to be applied or restored at the start of the next cycle, and 1183 another to hold an integer that is either one, to apply the 1184 corresponding replacement, or zero to restore it. */ 1185 static vec<dep_t> next_cycle_replace_deps; 1186 static vec<int> next_cycle_apply; 1187 1188 static void apply_replacement (dep_t, bool); 1189 static void restore_pattern (dep_t, bool); 1190 1191 /* Look at the remaining dependencies for insn NEXT, and compute and return 1192 the TODO_SPEC value we should use for it. This is called after one of 1193 NEXT's dependencies has been resolved. 1194 We also perform pattern replacements for predication, and for broken 1195 replacement dependencies. The latter is only done if FOR_BACKTRACK is 1196 false. */ 1197 1198 static ds_t 1199 recompute_todo_spec (rtx_insn *next, bool for_backtrack) 1200 { 1201 ds_t new_ds; 1202 sd_iterator_def sd_it; 1203 dep_t dep, modify_dep = NULL; 1204 int n_spec = 0; 1205 int n_control = 0; 1206 int n_replace = 0; 1207 bool first_p = true; 1208 1209 if (sd_lists_empty_p (next, SD_LIST_BACK)) 1210 /* NEXT has all its dependencies resolved. */ 1211 return 0; 1212 1213 if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK)) 1214 return HARD_DEP; 1215 1216 /* If NEXT is intended to sit adjacent to this instruction, we don't 1217 want to try to break any dependencies. Treat it as a HARD_DEP. */ 1218 if (SCHED_GROUP_P (next)) 1219 return HARD_DEP; 1220 1221 /* Now we've got NEXT with speculative deps only. 1222 1. Look at the deps to see what we have to do. 1223 2. Check if we can do 'todo'. */ 1224 new_ds = 0; 1225 1226 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep) 1227 { 1228 rtx_insn *pro = DEP_PRO (dep); 1229 ds_t ds = DEP_STATUS (dep) & SPECULATIVE; 1230 1231 if (DEBUG_INSN_P (pro) && !DEBUG_INSN_P (next)) 1232 continue; 1233 1234 if (ds) 1235 { 1236 n_spec++; 1237 if (first_p) 1238 { 1239 first_p = false; 1240 1241 new_ds = ds; 1242 } 1243 else 1244 new_ds = ds_merge (new_ds, ds); 1245 } 1246 else if (DEP_TYPE (dep) == REG_DEP_CONTROL) 1247 { 1248 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED) 1249 { 1250 n_control++; 1251 modify_dep = dep; 1252 } 1253 DEP_STATUS (dep) &= ~DEP_CANCELLED; 1254 } 1255 else if (DEP_REPLACE (dep) != NULL) 1256 { 1257 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED) 1258 { 1259 n_replace++; 1260 modify_dep = dep; 1261 } 1262 DEP_STATUS (dep) &= ~DEP_CANCELLED; 1263 } 1264 } 1265 1266 if (n_replace > 0 && n_control == 0 && n_spec == 0) 1267 { 1268 if (!dbg_cnt (sched_breakdep)) 1269 return HARD_DEP; 1270 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep) 1271 { 1272 struct dep_replacement *desc = DEP_REPLACE (dep); 1273 if (desc != NULL) 1274 { 1275 if (desc->insn == next && !for_backtrack) 1276 { 1277 gcc_assert (n_replace == 1); 1278 apply_replacement (dep, true); 1279 } 1280 DEP_STATUS (dep) |= DEP_CANCELLED; 1281 } 1282 } 1283 return 0; 1284 } 1285 1286 else if (n_control == 1 && n_replace == 0 && n_spec == 0) 1287 { 1288 rtx_insn *pro, *other; 1289 rtx new_pat; 1290 rtx cond = NULL_RTX; 1291 bool success; 1292 rtx_insn *prev = NULL; 1293 int i; 1294 unsigned regno; 1295 1296 if ((current_sched_info->flags & DO_PREDICATION) == 0 1297 || (ORIG_PAT (next) != NULL_RTX 1298 && PREDICATED_PAT (next) == NULL_RTX)) 1299 return HARD_DEP; 1300 1301 pro = DEP_PRO (modify_dep); 1302 other = real_insn_for_shadow (pro); 1303 if (other != NULL_RTX) 1304 pro = other; 1305 1306 cond = sched_get_reverse_condition_uncached (pro); 1307 regno = REGNO (XEXP (cond, 0)); 1308 1309 /* Find the last scheduled insn that modifies the condition register. 1310 We can stop looking once we find the insn we depend on through the 1311 REG_DEP_CONTROL; if the condition register isn't modified after it, 1312 we know that it still has the right value. */ 1313 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED) 1314 FOR_EACH_VEC_ELT_REVERSE (scheduled_insns, i, prev) 1315 { 1316 HARD_REG_SET t; 1317 1318 find_all_hard_reg_sets (prev, &t, true); 1319 if (TEST_HARD_REG_BIT (t, regno)) 1320 return HARD_DEP; 1321 if (prev == pro) 1322 break; 1323 } 1324 if (ORIG_PAT (next) == NULL_RTX) 1325 { 1326 ORIG_PAT (next) = PATTERN (next); 1327 1328 new_pat = gen_rtx_COND_EXEC (VOIDmode, cond, PATTERN (next)); 1329 success = haifa_change_pattern (next, new_pat); 1330 if (!success) 1331 return HARD_DEP; 1332 PREDICATED_PAT (next) = new_pat; 1333 } 1334 else if (PATTERN (next) != PREDICATED_PAT (next)) 1335 { 1336 bool success = haifa_change_pattern (next, 1337 PREDICATED_PAT (next)); 1338 gcc_assert (success); 1339 } 1340 DEP_STATUS (modify_dep) |= DEP_CANCELLED; 1341 return DEP_CONTROL; 1342 } 1343 1344 if (PREDICATED_PAT (next) != NULL_RTX) 1345 { 1346 int tick = INSN_TICK (next); 1347 bool success = haifa_change_pattern (next, 1348 ORIG_PAT (next)); 1349 INSN_TICK (next) = tick; 1350 gcc_assert (success); 1351 } 1352 1353 /* We can't handle the case where there are both speculative and control 1354 dependencies, so we return HARD_DEP in such a case. Also fail if 1355 we have speculative dependencies with not enough points, or more than 1356 one control dependency. */ 1357 if ((n_spec > 0 && (n_control > 0 || n_replace > 0)) 1358 || (n_spec > 0 1359 /* Too few points? */ 1360 && ds_weak (new_ds) < spec_info->data_weakness_cutoff) 1361 || n_control > 0 1362 || n_replace > 0) 1363 return HARD_DEP; 1364 1365 return new_ds; 1366 } 1367 1368 /* Pointer to the last instruction scheduled. */ 1369 static rtx_insn *last_scheduled_insn; 1370 1371 /* Pointer to the last nondebug instruction scheduled within the 1372 block, or the prev_head of the scheduling block. Used by 1373 rank_for_schedule, so that insns independent of the last scheduled 1374 insn will be preferred over dependent instructions. */ 1375 static rtx_insn *last_nondebug_scheduled_insn; 1376 1377 /* Pointer that iterates through the list of unscheduled insns if we 1378 have a dbg_cnt enabled. It always points at an insn prior to the 1379 first unscheduled one. */ 1380 static rtx_insn *nonscheduled_insns_begin; 1381 1382 /* Compute cost of executing INSN. 1383 This is the number of cycles between instruction issue and 1384 instruction results. */ 1385 int 1386 insn_sched_cost (rtx_insn *insn) 1387 { 1388 int cost; 1389 1390 if (sched_fusion) 1391 return 0; 1392 1393 if (sel_sched_p ()) 1394 { 1395 if (recog_memoized (insn) < 0) 1396 return 0; 1397 1398 cost = insn_default_latency (insn); 1399 if (cost < 0) 1400 cost = 0; 1401 1402 return cost; 1403 } 1404 1405 cost = INSN_COST (insn); 1406 1407 if (cost < 0) 1408 { 1409 /* A USE insn, or something else we don't need to 1410 understand. We can't pass these directly to 1411 result_ready_cost or insn_default_latency because it will 1412 trigger a fatal error for unrecognizable insns. */ 1413 if (recog_memoized (insn) < 0) 1414 { 1415 INSN_COST (insn) = 0; 1416 return 0; 1417 } 1418 else 1419 { 1420 cost = insn_default_latency (insn); 1421 if (cost < 0) 1422 cost = 0; 1423 1424 INSN_COST (insn) = cost; 1425 } 1426 } 1427 1428 return cost; 1429 } 1430 1431 /* Compute cost of dependence LINK. 1432 This is the number of cycles between instruction issue and 1433 instruction results. 1434 ??? We also use this function to call recog_memoized on all insns. */ 1435 int 1436 dep_cost_1 (dep_t link, dw_t dw) 1437 { 1438 rtx_insn *insn = DEP_PRO (link); 1439 rtx_insn *used = DEP_CON (link); 1440 int cost; 1441 1442 if (DEP_COST (link) != UNKNOWN_DEP_COST) 1443 return DEP_COST (link); 1444 1445 if (delay_htab) 1446 { 1447 struct delay_pair *delay_entry; 1448 delay_entry 1449 = delay_htab_i2->find_with_hash (used, htab_hash_pointer (used)); 1450 if (delay_entry) 1451 { 1452 if (delay_entry->i1 == insn) 1453 { 1454 DEP_COST (link) = pair_delay (delay_entry); 1455 return DEP_COST (link); 1456 } 1457 } 1458 } 1459 1460 /* A USE insn should never require the value used to be computed. 1461 This allows the computation of a function's result and parameter 1462 values to overlap the return and call. We don't care about the 1463 dependence cost when only decreasing register pressure. */ 1464 if (recog_memoized (used) < 0) 1465 { 1466 cost = 0; 1467 recog_memoized (insn); 1468 } 1469 else 1470 { 1471 enum reg_note dep_type = DEP_TYPE (link); 1472 1473 cost = insn_sched_cost (insn); 1474 1475 if (INSN_CODE (insn) >= 0) 1476 { 1477 if (dep_type == REG_DEP_ANTI) 1478 cost = 0; 1479 else if (dep_type == REG_DEP_OUTPUT) 1480 { 1481 cost = (insn_default_latency (insn) 1482 - insn_default_latency (used)); 1483 if (cost <= 0) 1484 cost = 1; 1485 } 1486 else if (bypass_p (insn)) 1487 cost = insn_latency (insn, used); 1488 } 1489 1490 1491 if (targetm.sched.adjust_cost) 1492 cost = targetm.sched.adjust_cost (used, (int) dep_type, insn, cost, 1493 dw); 1494 1495 if (cost < 0) 1496 cost = 0; 1497 } 1498 1499 DEP_COST (link) = cost; 1500 return cost; 1501 } 1502 1503 /* Compute cost of dependence LINK. 1504 This is the number of cycles between instruction issue and 1505 instruction results. */ 1506 int 1507 dep_cost (dep_t link) 1508 { 1509 return dep_cost_1 (link, 0); 1510 } 1511 1512 /* Use this sel-sched.c friendly function in reorder2 instead of increasing 1513 INSN_PRIORITY explicitly. */ 1514 void 1515 increase_insn_priority (rtx_insn *insn, int amount) 1516 { 1517 if (!sel_sched_p ()) 1518 { 1519 /* We're dealing with haifa-sched.c INSN_PRIORITY. */ 1520 if (INSN_PRIORITY_KNOWN (insn)) 1521 INSN_PRIORITY (insn) += amount; 1522 } 1523 else 1524 { 1525 /* In sel-sched.c INSN_PRIORITY is not kept up to date. 1526 Use EXPR_PRIORITY instead. */ 1527 sel_add_to_insn_priority (insn, amount); 1528 } 1529 } 1530 1531 /* Return 'true' if DEP should be included in priority calculations. */ 1532 static bool 1533 contributes_to_priority_p (dep_t dep) 1534 { 1535 if (DEBUG_INSN_P (DEP_CON (dep)) 1536 || DEBUG_INSN_P (DEP_PRO (dep))) 1537 return false; 1538 1539 /* Critical path is meaningful in block boundaries only. */ 1540 if (!current_sched_info->contributes_to_priority (DEP_CON (dep), 1541 DEP_PRO (dep))) 1542 return false; 1543 1544 if (DEP_REPLACE (dep) != NULL) 1545 return false; 1546 1547 /* If flag COUNT_SPEC_IN_CRITICAL_PATH is set, 1548 then speculative instructions will less likely be 1549 scheduled. That is because the priority of 1550 their producers will increase, and, thus, the 1551 producers will more likely be scheduled, thus, 1552 resolving the dependence. */ 1553 if (sched_deps_info->generate_spec_deps 1554 && !(spec_info->flags & COUNT_SPEC_IN_CRITICAL_PATH) 1555 && (DEP_STATUS (dep) & SPECULATIVE)) 1556 return false; 1557 1558 return true; 1559 } 1560 1561 /* Compute the number of nondebug deps in list LIST for INSN. */ 1562 1563 static int 1564 dep_list_size (rtx_insn *insn, sd_list_types_def list) 1565 { 1566 sd_iterator_def sd_it; 1567 dep_t dep; 1568 int dbgcount = 0, nodbgcount = 0; 1569 1570 if (!MAY_HAVE_DEBUG_INSNS) 1571 return sd_lists_size (insn, list); 1572 1573 FOR_EACH_DEP (insn, list, sd_it, dep) 1574 { 1575 if (DEBUG_INSN_P (DEP_CON (dep))) 1576 dbgcount++; 1577 else if (!DEBUG_INSN_P (DEP_PRO (dep))) 1578 nodbgcount++; 1579 } 1580 1581 gcc_assert (dbgcount + nodbgcount == sd_lists_size (insn, list)); 1582 1583 return nodbgcount; 1584 } 1585 1586 bool sched_fusion; 1587 1588 /* Compute the priority number for INSN. */ 1589 static int 1590 priority (rtx_insn *insn) 1591 { 1592 if (! INSN_P (insn)) 1593 return 0; 1594 1595 /* We should not be interested in priority of an already scheduled insn. */ 1596 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED); 1597 1598 if (!INSN_PRIORITY_KNOWN (insn)) 1599 { 1600 int this_priority = -1; 1601 1602 if (sched_fusion) 1603 { 1604 int this_fusion_priority; 1605 1606 targetm.sched.fusion_priority (insn, FUSION_MAX_PRIORITY, 1607 &this_fusion_priority, &this_priority); 1608 INSN_FUSION_PRIORITY (insn) = this_fusion_priority; 1609 } 1610 else if (dep_list_size (insn, SD_LIST_FORW) == 0) 1611 /* ??? We should set INSN_PRIORITY to insn_sched_cost when and insn 1612 has some forward deps but all of them are ignored by 1613 contributes_to_priority hook. At the moment we set priority of 1614 such insn to 0. */ 1615 this_priority = insn_sched_cost (insn); 1616 else 1617 { 1618 rtx_insn *prev_first, *twin; 1619 basic_block rec; 1620 1621 /* For recovery check instructions we calculate priority slightly 1622 different than that of normal instructions. Instead of walking 1623 through INSN_FORW_DEPS (check) list, we walk through 1624 INSN_FORW_DEPS list of each instruction in the corresponding 1625 recovery block. */ 1626 1627 /* Selective scheduling does not define RECOVERY_BLOCK macro. */ 1628 rec = sel_sched_p () ? NULL : RECOVERY_BLOCK (insn); 1629 if (!rec || rec == EXIT_BLOCK_PTR_FOR_FN (cfun)) 1630 { 1631 prev_first = PREV_INSN (insn); 1632 twin = insn; 1633 } 1634 else 1635 { 1636 prev_first = NEXT_INSN (BB_HEAD (rec)); 1637 twin = PREV_INSN (BB_END (rec)); 1638 } 1639 1640 do 1641 { 1642 sd_iterator_def sd_it; 1643 dep_t dep; 1644 1645 FOR_EACH_DEP (twin, SD_LIST_FORW, sd_it, dep) 1646 { 1647 rtx_insn *next; 1648 int next_priority; 1649 1650 next = DEP_CON (dep); 1651 1652 if (BLOCK_FOR_INSN (next) != rec) 1653 { 1654 int cost; 1655 1656 if (!contributes_to_priority_p (dep)) 1657 continue; 1658 1659 if (twin == insn) 1660 cost = dep_cost (dep); 1661 else 1662 { 1663 struct _dep _dep1, *dep1 = &_dep1; 1664 1665 init_dep (dep1, insn, next, REG_DEP_ANTI); 1666 1667 cost = dep_cost (dep1); 1668 } 1669 1670 next_priority = cost + priority (next); 1671 1672 if (next_priority > this_priority) 1673 this_priority = next_priority; 1674 } 1675 } 1676 1677 twin = PREV_INSN (twin); 1678 } 1679 while (twin != prev_first); 1680 } 1681 1682 if (this_priority < 0) 1683 { 1684 gcc_assert (this_priority == -1); 1685 1686 this_priority = insn_sched_cost (insn); 1687 } 1688 1689 INSN_PRIORITY (insn) = this_priority; 1690 INSN_PRIORITY_STATUS (insn) = 1; 1691 } 1692 1693 return INSN_PRIORITY (insn); 1694 } 1695 1696 /* Macros and functions for keeping the priority queue sorted, and 1697 dealing with queuing and dequeuing of instructions. */ 1698 1699 /* For each pressure class CL, set DEATH[CL] to the number of registers 1700 in that class that die in INSN. */ 1701 1702 static void 1703 calculate_reg_deaths (rtx_insn *insn, int *death) 1704 { 1705 int i; 1706 struct reg_use_data *use; 1707 1708 for (i = 0; i < ira_pressure_classes_num; i++) 1709 death[ira_pressure_classes[i]] = 0; 1710 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use) 1711 if (dying_use_p (use)) 1712 mark_regno_birth_or_death (0, death, use->regno, true); 1713 } 1714 1715 /* Setup info about the current register pressure impact of scheduling 1716 INSN at the current scheduling point. */ 1717 static void 1718 setup_insn_reg_pressure_info (rtx_insn *insn) 1719 { 1720 int i, change, before, after, hard_regno; 1721 int excess_cost_change; 1722 machine_mode mode; 1723 enum reg_class cl; 1724 struct reg_pressure_data *pressure_info; 1725 int *max_reg_pressure; 1726 static int death[N_REG_CLASSES]; 1727 1728 gcc_checking_assert (!DEBUG_INSN_P (insn)); 1729 1730 excess_cost_change = 0; 1731 calculate_reg_deaths (insn, death); 1732 pressure_info = INSN_REG_PRESSURE (insn); 1733 max_reg_pressure = INSN_MAX_REG_PRESSURE (insn); 1734 gcc_assert (pressure_info != NULL && max_reg_pressure != NULL); 1735 for (i = 0; i < ira_pressure_classes_num; i++) 1736 { 1737 cl = ira_pressure_classes[i]; 1738 gcc_assert (curr_reg_pressure[cl] >= 0); 1739 change = (int) pressure_info[i].set_increase - death[cl]; 1740 before = MAX (0, max_reg_pressure[i] - sched_class_regs_num[cl]); 1741 after = MAX (0, max_reg_pressure[i] + change 1742 - sched_class_regs_num[cl]); 1743 hard_regno = ira_class_hard_regs[cl][0]; 1744 gcc_assert (hard_regno >= 0); 1745 mode = reg_raw_mode[hard_regno]; 1746 excess_cost_change += ((after - before) 1747 * (ira_memory_move_cost[mode][cl][0] 1748 + ira_memory_move_cost[mode][cl][1])); 1749 } 1750 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insn) = excess_cost_change; 1751 } 1752 1753 /* This is the first page of code related to SCHED_PRESSURE_MODEL. 1754 It tries to make the scheduler take register pressure into account 1755 without introducing too many unnecessary stalls. It hooks into the 1756 main scheduling algorithm at several points: 1757 1758 - Before scheduling starts, model_start_schedule constructs a 1759 "model schedule" for the current block. This model schedule is 1760 chosen solely to keep register pressure down. It does not take the 1761 target's pipeline or the original instruction order into account, 1762 except as a tie-breaker. It also doesn't work to a particular 1763 pressure limit. 1764 1765 This model schedule gives us an idea of what pressure can be 1766 achieved for the block and gives us an example of a schedule that 1767 keeps to that pressure. It also makes the final schedule less 1768 dependent on the original instruction order. This is important 1769 because the original order can either be "wide" (many values live 1770 at once, such as in user-scheduled code) or "narrow" (few values 1771 live at once, such as after loop unrolling, where several 1772 iterations are executed sequentially). 1773 1774 We do not apply this model schedule to the rtx stream. We simply 1775 record it in model_schedule. We also compute the maximum pressure, 1776 MP, that was seen during this schedule. 1777 1778 - Instructions are added to the ready queue even if they require 1779 a stall. The length of the stall is instead computed as: 1780 1781 MAX (INSN_TICK (INSN) - clock_var, 0) 1782 1783 (= insn_delay). This allows rank_for_schedule to choose between 1784 introducing a deliberate stall or increasing pressure. 1785 1786 - Before sorting the ready queue, model_set_excess_costs assigns 1787 a pressure-based cost to each ready instruction in the queue. 1788 This is the instruction's INSN_REG_PRESSURE_EXCESS_COST_CHANGE 1789 (ECC for short) and is effectively measured in cycles. 1790 1791 - rank_for_schedule ranks instructions based on: 1792 1793 ECC (insn) + insn_delay (insn) 1794 1795 then as: 1796 1797 insn_delay (insn) 1798 1799 So, for example, an instruction X1 with an ECC of 1 that can issue 1800 now will win over an instruction X0 with an ECC of zero that would 1801 introduce a stall of one cycle. However, an instruction X2 with an 1802 ECC of 2 that can issue now will lose to both X0 and X1. 1803 1804 - When an instruction is scheduled, model_recompute updates the model 1805 schedule with the new pressures (some of which might now exceed the 1806 original maximum pressure MP). model_update_limit_points then searches 1807 for the new point of maximum pressure, if not already known. */ 1808 1809 /* Used to separate high-verbosity debug information for SCHED_PRESSURE_MODEL 1810 from surrounding debug information. */ 1811 #define MODEL_BAR \ 1812 ";;\t\t+------------------------------------------------------\n" 1813 1814 /* Information about the pressure on a particular register class at a 1815 particular point of the model schedule. */ 1816 struct model_pressure_data { 1817 /* The pressure at this point of the model schedule, or -1 if the 1818 point is associated with an instruction that has already been 1819 scheduled. */ 1820 int ref_pressure; 1821 1822 /* The maximum pressure during or after this point of the model schedule. */ 1823 int max_pressure; 1824 }; 1825 1826 /* Per-instruction information that is used while building the model 1827 schedule. Here, "schedule" refers to the model schedule rather 1828 than the main schedule. */ 1829 struct model_insn_info { 1830 /* The instruction itself. */ 1831 rtx_insn *insn; 1832 1833 /* If this instruction is in model_worklist, these fields link to the 1834 previous (higher-priority) and next (lower-priority) instructions 1835 in the list. */ 1836 struct model_insn_info *prev; 1837 struct model_insn_info *next; 1838 1839 /* While constructing the schedule, QUEUE_INDEX describes whether an 1840 instruction has already been added to the schedule (QUEUE_SCHEDULED), 1841 is in model_worklist (QUEUE_READY), or neither (QUEUE_NOWHERE). 1842 old_queue records the value that QUEUE_INDEX had before scheduling 1843 started, so that we can restore it once the schedule is complete. */ 1844 int old_queue; 1845 1846 /* The relative importance of an unscheduled instruction. Higher 1847 values indicate greater importance. */ 1848 unsigned int model_priority; 1849 1850 /* The length of the longest path of satisfied true dependencies 1851 that leads to this instruction. */ 1852 unsigned int depth; 1853 1854 /* The length of the longest path of dependencies of any kind 1855 that leads from this instruction. */ 1856 unsigned int alap; 1857 1858 /* The number of predecessor nodes that must still be scheduled. */ 1859 int unscheduled_preds; 1860 }; 1861 1862 /* Information about the pressure limit for a particular register class. 1863 This structure is used when applying a model schedule to the main 1864 schedule. */ 1865 struct model_pressure_limit { 1866 /* The maximum register pressure seen in the original model schedule. */ 1867 int orig_pressure; 1868 1869 /* The maximum register pressure seen in the current model schedule 1870 (which excludes instructions that have already been scheduled). */ 1871 int pressure; 1872 1873 /* The point of the current model schedule at which PRESSURE is first 1874 reached. It is set to -1 if the value needs to be recomputed. */ 1875 int point; 1876 }; 1877 1878 /* Describes a particular way of measuring register pressure. */ 1879 struct model_pressure_group { 1880 /* Index PCI describes the maximum pressure on ira_pressure_classes[PCI]. */ 1881 struct model_pressure_limit limits[N_REG_CLASSES]; 1882 1883 /* Index (POINT * ira_num_pressure_classes + PCI) describes the pressure 1884 on register class ira_pressure_classes[PCI] at point POINT of the 1885 current model schedule. A POINT of model_num_insns describes the 1886 pressure at the end of the schedule. */ 1887 struct model_pressure_data *model; 1888 }; 1889 1890 /* Index POINT gives the instruction at point POINT of the model schedule. 1891 This array doesn't change during main scheduling. */ 1892 static vec<rtx_insn *> model_schedule; 1893 1894 /* The list of instructions in the model worklist, sorted in order of 1895 decreasing priority. */ 1896 static struct model_insn_info *model_worklist; 1897 1898 /* Index I describes the instruction with INSN_LUID I. */ 1899 static struct model_insn_info *model_insns; 1900 1901 /* The number of instructions in the model schedule. */ 1902 static int model_num_insns; 1903 1904 /* The index of the first instruction in model_schedule that hasn't yet been 1905 added to the main schedule, or model_num_insns if all of them have. */ 1906 static int model_curr_point; 1907 1908 /* Describes the pressure before each instruction in the model schedule. */ 1909 static struct model_pressure_group model_before_pressure; 1910 1911 /* The first unused model_priority value (as used in model_insn_info). */ 1912 static unsigned int model_next_priority; 1913 1914 1915 /* The model_pressure_data for ira_pressure_classes[PCI] in GROUP 1916 at point POINT of the model schedule. */ 1917 #define MODEL_PRESSURE_DATA(GROUP, POINT, PCI) \ 1918 (&(GROUP)->model[(POINT) * ira_pressure_classes_num + (PCI)]) 1919 1920 /* The maximum pressure on ira_pressure_classes[PCI] in GROUP at or 1921 after point POINT of the model schedule. */ 1922 #define MODEL_MAX_PRESSURE(GROUP, POINT, PCI) \ 1923 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->max_pressure) 1924 1925 /* The pressure on ira_pressure_classes[PCI] in GROUP at point POINT 1926 of the model schedule. */ 1927 #define MODEL_REF_PRESSURE(GROUP, POINT, PCI) \ 1928 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->ref_pressure) 1929 1930 /* Information about INSN that is used when creating the model schedule. */ 1931 #define MODEL_INSN_INFO(INSN) \ 1932 (&model_insns[INSN_LUID (INSN)]) 1933 1934 /* The instruction at point POINT of the model schedule. */ 1935 #define MODEL_INSN(POINT) \ 1936 (model_schedule[POINT]) 1937 1938 1939 /* Return INSN's index in the model schedule, or model_num_insns if it 1940 doesn't belong to that schedule. */ 1941 1942 static int 1943 model_index (rtx_insn *insn) 1944 { 1945 if (INSN_MODEL_INDEX (insn) == 0) 1946 return model_num_insns; 1947 return INSN_MODEL_INDEX (insn) - 1; 1948 } 1949 1950 /* Make sure that GROUP->limits is up-to-date for the current point 1951 of the model schedule. */ 1952 1953 static void 1954 model_update_limit_points_in_group (struct model_pressure_group *group) 1955 { 1956 int pci, max_pressure, point; 1957 1958 for (pci = 0; pci < ira_pressure_classes_num; pci++) 1959 { 1960 /* We may have passed the final point at which the pressure in 1961 group->limits[pci].pressure was reached. Update the limit if so. */ 1962 max_pressure = MODEL_MAX_PRESSURE (group, model_curr_point, pci); 1963 group->limits[pci].pressure = max_pressure; 1964 1965 /* Find the point at which MAX_PRESSURE is first reached. We need 1966 to search in three cases: 1967 1968 - We've already moved past the previous pressure point. 1969 In this case we search forward from model_curr_point. 1970 1971 - We scheduled the previous point of maximum pressure ahead of 1972 its position in the model schedule, but doing so didn't bring 1973 the pressure point earlier. In this case we search forward 1974 from that previous pressure point. 1975 1976 - Scheduling an instruction early caused the maximum pressure 1977 to decrease. In this case we will have set the pressure 1978 point to -1, and we search forward from model_curr_point. */ 1979 point = MAX (group->limits[pci].point, model_curr_point); 1980 while (point < model_num_insns 1981 && MODEL_REF_PRESSURE (group, point, pci) < max_pressure) 1982 point++; 1983 group->limits[pci].point = point; 1984 1985 gcc_assert (MODEL_REF_PRESSURE (group, point, pci) == max_pressure); 1986 gcc_assert (MODEL_MAX_PRESSURE (group, point, pci) == max_pressure); 1987 } 1988 } 1989 1990 /* Make sure that all register-pressure limits are up-to-date for the 1991 current position in the model schedule. */ 1992 1993 static void 1994 model_update_limit_points (void) 1995 { 1996 model_update_limit_points_in_group (&model_before_pressure); 1997 } 1998 1999 /* Return the model_index of the last unscheduled use in chain USE 2000 outside of USE's instruction. Return -1 if there are no other uses, 2001 or model_num_insns if the register is live at the end of the block. */ 2002 2003 static int 2004 model_last_use_except (struct reg_use_data *use) 2005 { 2006 struct reg_use_data *next; 2007 int last, index; 2008 2009 last = -1; 2010 for (next = use->next_regno_use; next != use; next = next->next_regno_use) 2011 if (NONDEBUG_INSN_P (next->insn) 2012 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED) 2013 { 2014 index = model_index (next->insn); 2015 if (index == model_num_insns) 2016 return model_num_insns; 2017 if (last < index) 2018 last = index; 2019 } 2020 return last; 2021 } 2022 2023 /* An instruction with model_index POINT has just been scheduled, and it 2024 adds DELTA to the pressure on ira_pressure_classes[PCI] after POINT - 1. 2025 Update MODEL_REF_PRESSURE (GROUP, POINT, PCI) and 2026 MODEL_MAX_PRESSURE (GROUP, POINT, PCI) accordingly. */ 2027 2028 static void 2029 model_start_update_pressure (struct model_pressure_group *group, 2030 int point, int pci, int delta) 2031 { 2032 int next_max_pressure; 2033 2034 if (point == model_num_insns) 2035 { 2036 /* The instruction wasn't part of the model schedule; it was moved 2037 from a different block. Update the pressure for the end of 2038 the model schedule. */ 2039 MODEL_REF_PRESSURE (group, point, pci) += delta; 2040 MODEL_MAX_PRESSURE (group, point, pci) += delta; 2041 } 2042 else 2043 { 2044 /* Record that this instruction has been scheduled. Nothing now 2045 changes between POINT and POINT + 1, so get the maximum pressure 2046 from the latter. If the maximum pressure decreases, the new 2047 pressure point may be before POINT. */ 2048 MODEL_REF_PRESSURE (group, point, pci) = -1; 2049 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci); 2050 if (MODEL_MAX_PRESSURE (group, point, pci) > next_max_pressure) 2051 { 2052 MODEL_MAX_PRESSURE (group, point, pci) = next_max_pressure; 2053 if (group->limits[pci].point == point) 2054 group->limits[pci].point = -1; 2055 } 2056 } 2057 } 2058 2059 /* Record that scheduling a later instruction has changed the pressure 2060 at point POINT of the model schedule by DELTA (which might be 0). 2061 Update GROUP accordingly. Return nonzero if these changes might 2062 trigger changes to previous points as well. */ 2063 2064 static int 2065 model_update_pressure (struct model_pressure_group *group, 2066 int point, int pci, int delta) 2067 { 2068 int ref_pressure, max_pressure, next_max_pressure; 2069 2070 /* If POINT hasn't yet been scheduled, update its pressure. */ 2071 ref_pressure = MODEL_REF_PRESSURE (group, point, pci); 2072 if (ref_pressure >= 0 && delta != 0) 2073 { 2074 ref_pressure += delta; 2075 MODEL_REF_PRESSURE (group, point, pci) = ref_pressure; 2076 2077 /* Check whether the maximum pressure in the overall schedule 2078 has increased. (This means that the MODEL_MAX_PRESSURE of 2079 every point <= POINT will need to increase too; see below.) */ 2080 if (group->limits[pci].pressure < ref_pressure) 2081 group->limits[pci].pressure = ref_pressure; 2082 2083 /* If we are at maximum pressure, and the maximum pressure 2084 point was previously unknown or later than POINT, 2085 bring it forward. */ 2086 if (group->limits[pci].pressure == ref_pressure 2087 && !IN_RANGE (group->limits[pci].point, 0, point)) 2088 group->limits[pci].point = point; 2089 2090 /* If POINT used to be the point of maximum pressure, but isn't 2091 any longer, we need to recalculate it using a forward walk. */ 2092 if (group->limits[pci].pressure > ref_pressure 2093 && group->limits[pci].point == point) 2094 group->limits[pci].point = -1; 2095 } 2096 2097 /* Update the maximum pressure at POINT. Changes here might also 2098 affect the maximum pressure at POINT - 1. */ 2099 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci); 2100 max_pressure = MAX (ref_pressure, next_max_pressure); 2101 if (MODEL_MAX_PRESSURE (group, point, pci) != max_pressure) 2102 { 2103 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure; 2104 return 1; 2105 } 2106 return 0; 2107 } 2108 2109 /* INSN has just been scheduled. Update the model schedule accordingly. */ 2110 2111 static void 2112 model_recompute (rtx_insn *insn) 2113 { 2114 struct { 2115 int last_use; 2116 int regno; 2117 } uses[FIRST_PSEUDO_REGISTER + MAX_RECOG_OPERANDS]; 2118 struct reg_use_data *use; 2119 struct reg_pressure_data *reg_pressure; 2120 int delta[N_REG_CLASSES]; 2121 int pci, point, mix, new_last, cl, ref_pressure, queue; 2122 unsigned int i, num_uses, num_pending_births; 2123 bool print_p; 2124 2125 /* The destinations of INSN were previously live from POINT onwards, but are 2126 now live from model_curr_point onwards. Set up DELTA accordingly. */ 2127 point = model_index (insn); 2128 reg_pressure = INSN_REG_PRESSURE (insn); 2129 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2130 { 2131 cl = ira_pressure_classes[pci]; 2132 delta[cl] = reg_pressure[pci].set_increase; 2133 } 2134 2135 /* Record which registers previously died at POINT, but which now die 2136 before POINT. Adjust DELTA so that it represents the effect of 2137 this change after POINT - 1. Set NUM_PENDING_BIRTHS to the number of 2138 registers that will be born in the range [model_curr_point, POINT). */ 2139 num_uses = 0; 2140 num_pending_births = 0; 2141 bitmap_clear (tmp_bitmap); 2142 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use) 2143 { 2144 new_last = model_last_use_except (use); 2145 if (new_last < point && bitmap_set_bit (tmp_bitmap, use->regno)) 2146 { 2147 gcc_assert (num_uses < ARRAY_SIZE (uses)); 2148 uses[num_uses].last_use = new_last; 2149 uses[num_uses].regno = use->regno; 2150 /* This register is no longer live after POINT - 1. */ 2151 mark_regno_birth_or_death (NULL, delta, use->regno, false); 2152 num_uses++; 2153 if (new_last >= 0) 2154 num_pending_births++; 2155 } 2156 } 2157 2158 /* Update the MODEL_REF_PRESSURE and MODEL_MAX_PRESSURE for POINT. 2159 Also set each group pressure limit for POINT. */ 2160 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2161 { 2162 cl = ira_pressure_classes[pci]; 2163 model_start_update_pressure (&model_before_pressure, 2164 point, pci, delta[cl]); 2165 } 2166 2167 /* Walk the model schedule backwards, starting immediately before POINT. */ 2168 print_p = false; 2169 if (point != model_curr_point) 2170 do 2171 { 2172 point--; 2173 insn = MODEL_INSN (point); 2174 queue = QUEUE_INDEX (insn); 2175 2176 if (queue != QUEUE_SCHEDULED) 2177 { 2178 /* DELTA describes the effect of the move on the register pressure 2179 after POINT. Make it describe the effect on the pressure 2180 before POINT. */ 2181 i = 0; 2182 while (i < num_uses) 2183 { 2184 if (uses[i].last_use == point) 2185 { 2186 /* This register is now live again. */ 2187 mark_regno_birth_or_death (NULL, delta, 2188 uses[i].regno, true); 2189 2190 /* Remove this use from the array. */ 2191 uses[i] = uses[num_uses - 1]; 2192 num_uses--; 2193 num_pending_births--; 2194 } 2195 else 2196 i++; 2197 } 2198 2199 if (sched_verbose >= 5) 2200 { 2201 if (!print_p) 2202 { 2203 fprintf (sched_dump, MODEL_BAR); 2204 fprintf (sched_dump, ";;\t\t| New pressure for model" 2205 " schedule\n"); 2206 fprintf (sched_dump, MODEL_BAR); 2207 print_p = true; 2208 } 2209 2210 fprintf (sched_dump, ";;\t\t| %3d %4d %-30s ", 2211 point, INSN_UID (insn), 2212 str_pattern_slim (PATTERN (insn))); 2213 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2214 { 2215 cl = ira_pressure_classes[pci]; 2216 ref_pressure = MODEL_REF_PRESSURE (&model_before_pressure, 2217 point, pci); 2218 fprintf (sched_dump, " %s:[%d->%d]", 2219 reg_class_names[ira_pressure_classes[pci]], 2220 ref_pressure, ref_pressure + delta[cl]); 2221 } 2222 fprintf (sched_dump, "\n"); 2223 } 2224 } 2225 2226 /* Adjust the pressure at POINT. Set MIX to nonzero if POINT - 1 2227 might have changed as well. */ 2228 mix = num_pending_births; 2229 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2230 { 2231 cl = ira_pressure_classes[pci]; 2232 mix |= delta[cl]; 2233 mix |= model_update_pressure (&model_before_pressure, 2234 point, pci, delta[cl]); 2235 } 2236 } 2237 while (mix && point > model_curr_point); 2238 2239 if (print_p) 2240 fprintf (sched_dump, MODEL_BAR); 2241 } 2242 2243 /* After DEP, which was cancelled, has been resolved for insn NEXT, 2244 check whether the insn's pattern needs restoring. */ 2245 static bool 2246 must_restore_pattern_p (rtx_insn *next, dep_t dep) 2247 { 2248 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED) 2249 return false; 2250 2251 if (DEP_TYPE (dep) == REG_DEP_CONTROL) 2252 { 2253 gcc_assert (ORIG_PAT (next) != NULL_RTX); 2254 gcc_assert (next == DEP_CON (dep)); 2255 } 2256 else 2257 { 2258 struct dep_replacement *desc = DEP_REPLACE (dep); 2259 if (desc->insn != next) 2260 { 2261 gcc_assert (*desc->loc == desc->orig); 2262 return false; 2263 } 2264 } 2265 return true; 2266 } 2267 2268 /* model_spill_cost (CL, P, P') returns the cost of increasing the 2269 pressure on CL from P to P'. We use this to calculate a "base ECC", 2270 baseECC (CL, X), for each pressure class CL and each instruction X. 2271 Supposing X changes the pressure on CL from P to P', and that the 2272 maximum pressure on CL in the current model schedule is MP', then: 2273 2274 * if X occurs before or at the next point of maximum pressure in 2275 the model schedule and P' > MP', then: 2276 2277 baseECC (CL, X) = model_spill_cost (CL, MP, P') 2278 2279 The idea is that the pressure after scheduling a fixed set of 2280 instructions -- in this case, the set up to and including the 2281 next maximum pressure point -- is going to be the same regardless 2282 of the order; we simply want to keep the intermediate pressure 2283 under control. Thus X has a cost of zero unless scheduling it 2284 now would exceed MP'. 2285 2286 If all increases in the set are by the same amount, no zero-cost 2287 instruction will ever cause the pressure to exceed MP'. However, 2288 if X is instead moved past an instruction X' with pressure in the 2289 range (MP' - (P' - P), MP'), the pressure at X' will increase 2290 beyond MP'. Since baseECC is very much a heuristic anyway, 2291 it doesn't seem worth the overhead of tracking cases like these. 2292 2293 The cost of exceeding MP' is always based on the original maximum 2294 pressure MP. This is so that going 2 registers over the original 2295 limit has the same cost regardless of whether it comes from two 2296 separate +1 deltas or from a single +2 delta. 2297 2298 * if X occurs after the next point of maximum pressure in the model 2299 schedule and P' > P, then: 2300 2301 baseECC (CL, X) = model_spill_cost (CL, MP, MP' + (P' - P)) 2302 2303 That is, if we move X forward across a point of maximum pressure, 2304 and if X increases the pressure by P' - P, then we conservatively 2305 assume that scheduling X next would increase the maximum pressure 2306 by P' - P. Again, the cost of doing this is based on the original 2307 maximum pressure MP, for the same reason as above. 2308 2309 * if P' < P, P > MP, and X occurs at or after the next point of 2310 maximum pressure, then: 2311 2312 baseECC (CL, X) = -model_spill_cost (CL, MAX (MP, P'), P) 2313 2314 That is, if we have already exceeded the original maximum pressure MP, 2315 and if X might reduce the maximum pressure again -- or at least push 2316 it further back, and thus allow more scheduling freedom -- it is given 2317 a negative cost to reflect the improvement. 2318 2319 * otherwise, 2320 2321 baseECC (CL, X) = 0 2322 2323 In this case, X is not expected to affect the maximum pressure MP', 2324 so it has zero cost. 2325 2326 We then create a combined value baseECC (X) that is the sum of 2327 baseECC (CL, X) for each pressure class CL. 2328 2329 baseECC (X) could itself be used as the ECC value described above. 2330 However, this is often too conservative, in the sense that it 2331 tends to make high-priority instructions that increase pressure 2332 wait too long in cases where introducing a spill would be better. 2333 For this reason the final ECC is a priority-adjusted form of 2334 baseECC (X). Specifically, we calculate: 2335 2336 P (X) = INSN_PRIORITY (X) - insn_delay (X) - baseECC (X) 2337 baseP = MAX { P (X) | baseECC (X) <= 0 } 2338 2339 Then: 2340 2341 ECC (X) = MAX (MIN (baseP - P (X), baseECC (X)), 0) 2342 2343 Thus an instruction's effect on pressure is ignored if it has a high 2344 enough priority relative to the ones that don't increase pressure. 2345 Negative values of baseECC (X) do not increase the priority of X 2346 itself, but they do make it harder for other instructions to 2347 increase the pressure further. 2348 2349 This pressure cost is deliberately timid. The intention has been 2350 to choose a heuristic that rarely interferes with the normal list 2351 scheduler in cases where that scheduler would produce good code. 2352 We simply want to curb some of its worst excesses. */ 2353 2354 /* Return the cost of increasing the pressure in class CL from FROM to TO. 2355 2356 Here we use the very simplistic cost model that every register above 2357 sched_class_regs_num[CL] has a spill cost of 1. We could use other 2358 measures instead, such as one based on MEMORY_MOVE_COST. However: 2359 2360 (1) In order for an instruction to be scheduled, the higher cost 2361 would need to be justified in a single saving of that many stalls. 2362 This is overly pessimistic, because the benefit of spilling is 2363 often to avoid a sequence of several short stalls rather than 2364 a single long one. 2365 2366 (2) The cost is still arbitrary. Because we are not allocating 2367 registers during scheduling, we have no way of knowing for 2368 sure how many memory accesses will be required by each spill, 2369 where the spills will be placed within the block, or even 2370 which block(s) will contain the spills. 2371 2372 So a higher cost than 1 is often too conservative in practice, 2373 forcing blocks to contain unnecessary stalls instead of spill code. 2374 The simple cost below seems to be the best compromise. It reduces 2375 the interference with the normal list scheduler, which helps make 2376 it more suitable for a default-on option. */ 2377 2378 static int 2379 model_spill_cost (int cl, int from, int to) 2380 { 2381 from = MAX (from, sched_class_regs_num[cl]); 2382 return MAX (to, from) - from; 2383 } 2384 2385 /* Return baseECC (ira_pressure_classes[PCI], POINT), given that 2386 P = curr_reg_pressure[ira_pressure_classes[PCI]] and that 2387 P' = P + DELTA. */ 2388 2389 static int 2390 model_excess_group_cost (struct model_pressure_group *group, 2391 int point, int pci, int delta) 2392 { 2393 int pressure, cl; 2394 2395 cl = ira_pressure_classes[pci]; 2396 if (delta < 0 && point >= group->limits[pci].point) 2397 { 2398 pressure = MAX (group->limits[pci].orig_pressure, 2399 curr_reg_pressure[cl] + delta); 2400 return -model_spill_cost (cl, pressure, curr_reg_pressure[cl]); 2401 } 2402 2403 if (delta > 0) 2404 { 2405 if (point > group->limits[pci].point) 2406 pressure = group->limits[pci].pressure + delta; 2407 else 2408 pressure = curr_reg_pressure[cl] + delta; 2409 2410 if (pressure > group->limits[pci].pressure) 2411 return model_spill_cost (cl, group->limits[pci].orig_pressure, 2412 pressure); 2413 } 2414 2415 return 0; 2416 } 2417 2418 /* Return baseECC (MODEL_INSN (INSN)). Dump the costs to sched_dump 2419 if PRINT_P. */ 2420 2421 static int 2422 model_excess_cost (rtx_insn *insn, bool print_p) 2423 { 2424 int point, pci, cl, cost, this_cost, delta; 2425 struct reg_pressure_data *insn_reg_pressure; 2426 int insn_death[N_REG_CLASSES]; 2427 2428 calculate_reg_deaths (insn, insn_death); 2429 point = model_index (insn); 2430 insn_reg_pressure = INSN_REG_PRESSURE (insn); 2431 cost = 0; 2432 2433 if (print_p) 2434 fprintf (sched_dump, ";;\t\t| %3d %4d | %4d %+3d |", point, 2435 INSN_UID (insn), INSN_PRIORITY (insn), insn_delay (insn)); 2436 2437 /* Sum up the individual costs for each register class. */ 2438 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2439 { 2440 cl = ira_pressure_classes[pci]; 2441 delta = insn_reg_pressure[pci].set_increase - insn_death[cl]; 2442 this_cost = model_excess_group_cost (&model_before_pressure, 2443 point, pci, delta); 2444 cost += this_cost; 2445 if (print_p) 2446 fprintf (sched_dump, " %s:[%d base cost %d]", 2447 reg_class_names[cl], delta, this_cost); 2448 } 2449 2450 if (print_p) 2451 fprintf (sched_dump, "\n"); 2452 2453 return cost; 2454 } 2455 2456 /* Dump the next points of maximum pressure for GROUP. */ 2457 2458 static void 2459 model_dump_pressure_points (struct model_pressure_group *group) 2460 { 2461 int pci, cl; 2462 2463 fprintf (sched_dump, ";;\t\t| pressure points"); 2464 for (pci = 0; pci < ira_pressure_classes_num; pci++) 2465 { 2466 cl = ira_pressure_classes[pci]; 2467 fprintf (sched_dump, " %s:[%d->%d at ", reg_class_names[cl], 2468 curr_reg_pressure[cl], group->limits[pci].pressure); 2469 if (group->limits[pci].point < model_num_insns) 2470 fprintf (sched_dump, "%d:%d]", group->limits[pci].point, 2471 INSN_UID (MODEL_INSN (group->limits[pci].point))); 2472 else 2473 fprintf (sched_dump, "end]"); 2474 } 2475 fprintf (sched_dump, "\n"); 2476 } 2477 2478 /* Set INSN_REG_PRESSURE_EXCESS_COST_CHANGE for INSNS[0...COUNT-1]. */ 2479 2480 static void 2481 model_set_excess_costs (rtx_insn **insns, int count) 2482 { 2483 int i, cost, priority_base, priority; 2484 bool print_p; 2485 2486 /* Record the baseECC value for each instruction in the model schedule, 2487 except that negative costs are converted to zero ones now rather than 2488 later. Do not assign a cost to debug instructions, since they must 2489 not change code-generation decisions. Experiments suggest we also 2490 get better results by not assigning a cost to instructions from 2491 a different block. 2492 2493 Set PRIORITY_BASE to baseP in the block comment above. This is the 2494 maximum priority of the "cheap" instructions, which should always 2495 include the next model instruction. */ 2496 priority_base = 0; 2497 print_p = false; 2498 for (i = 0; i < count; i++) 2499 if (INSN_MODEL_INDEX (insns[i])) 2500 { 2501 if (sched_verbose >= 6 && !print_p) 2502 { 2503 fprintf (sched_dump, MODEL_BAR); 2504 fprintf (sched_dump, ";;\t\t| Pressure costs for ready queue\n"); 2505 model_dump_pressure_points (&model_before_pressure); 2506 fprintf (sched_dump, MODEL_BAR); 2507 print_p = true; 2508 } 2509 cost = model_excess_cost (insns[i], print_p); 2510 if (cost <= 0) 2511 { 2512 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]) - cost; 2513 priority_base = MAX (priority_base, priority); 2514 cost = 0; 2515 } 2516 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = cost; 2517 } 2518 if (print_p) 2519 fprintf (sched_dump, MODEL_BAR); 2520 2521 /* Use MAX (baseECC, 0) and baseP to calculcate ECC for each 2522 instruction. */ 2523 for (i = 0; i < count; i++) 2524 { 2525 cost = INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]); 2526 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]); 2527 if (cost > 0 && priority > priority_base) 2528 { 2529 cost += priority_base - priority; 2530 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = MAX (cost, 0); 2531 } 2532 } 2533 } 2534 2535 2536 /* Enum of rank_for_schedule heuristic decisions. */ 2537 enum rfs_decision { 2538 RFS_LIVE_RANGE_SHRINK1, RFS_LIVE_RANGE_SHRINK2, 2539 RFS_SCHED_GROUP, RFS_PRESSURE_DELAY, RFS_PRESSURE_TICK, 2540 RFS_FEEDS_BACKTRACK_INSN, RFS_PRIORITY, RFS_SPECULATION, 2541 RFS_SCHED_RANK, RFS_LAST_INSN, RFS_PRESSURE_INDEX, 2542 RFS_DEP_COUNT, RFS_TIE, RFS_FUSION, RFS_N }; 2543 2544 /* Corresponding strings for print outs. */ 2545 static const char *rfs_str[RFS_N] = { 2546 "RFS_LIVE_RANGE_SHRINK1", "RFS_LIVE_RANGE_SHRINK2", 2547 "RFS_SCHED_GROUP", "RFS_PRESSURE_DELAY", "RFS_PRESSURE_TICK", 2548 "RFS_FEEDS_BACKTRACK_INSN", "RFS_PRIORITY", "RFS_SPECULATION", 2549 "RFS_SCHED_RANK", "RFS_LAST_INSN", "RFS_PRESSURE_INDEX", 2550 "RFS_DEP_COUNT", "RFS_TIE", "RFS_FUSION" }; 2551 2552 /* Statistical breakdown of rank_for_schedule decisions. */ 2553 struct rank_for_schedule_stats_t { unsigned stats[RFS_N]; }; 2554 static rank_for_schedule_stats_t rank_for_schedule_stats; 2555 2556 /* Return the result of comparing insns TMP and TMP2 and update 2557 Rank_For_Schedule statistics. */ 2558 static int 2559 rfs_result (enum rfs_decision decision, int result, rtx tmp, rtx tmp2) 2560 { 2561 ++rank_for_schedule_stats.stats[decision]; 2562 if (result < 0) 2563 INSN_LAST_RFS_WIN (tmp) = decision; 2564 else if (result > 0) 2565 INSN_LAST_RFS_WIN (tmp2) = decision; 2566 else 2567 gcc_unreachable (); 2568 return result; 2569 } 2570 2571 /* Sorting predicate to move DEBUG_INSNs to the top of ready list, while 2572 keeping normal insns in original order. */ 2573 2574 static int 2575 rank_for_schedule_debug (const void *x, const void *y) 2576 { 2577 rtx_insn *tmp = *(rtx_insn * const *) y; 2578 rtx_insn *tmp2 = *(rtx_insn * const *) x; 2579 2580 /* Schedule debug insns as early as possible. */ 2581 if (DEBUG_INSN_P (tmp) && !DEBUG_INSN_P (tmp2)) 2582 return -1; 2583 else if (!DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2)) 2584 return 1; 2585 else if (DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2)) 2586 return INSN_LUID (tmp) - INSN_LUID (tmp2); 2587 else 2588 return INSN_RFS_DEBUG_ORIG_ORDER (tmp2) - INSN_RFS_DEBUG_ORIG_ORDER (tmp); 2589 } 2590 2591 /* Returns a positive value if x is preferred; returns a negative value if 2592 y is preferred. Should never return 0, since that will make the sort 2593 unstable. */ 2594 2595 static int 2596 rank_for_schedule (const void *x, const void *y) 2597 { 2598 rtx_insn *tmp = *(rtx_insn * const *) y; 2599 rtx_insn *tmp2 = *(rtx_insn * const *) x; 2600 int tmp_class, tmp2_class; 2601 int val, priority_val, info_val, diff; 2602 2603 if (live_range_shrinkage_p) 2604 { 2605 /* Don't use SCHED_PRESSURE_MODEL -- it results in much worse 2606 code. */ 2607 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED); 2608 if ((INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp) < 0 2609 || INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2) < 0) 2610 && (diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp) 2611 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2))) != 0) 2612 return rfs_result (RFS_LIVE_RANGE_SHRINK1, diff, tmp, tmp2); 2613 /* Sort by INSN_LUID (original insn order), so that we make the 2614 sort stable. This minimizes instruction movement, thus 2615 minimizing sched's effect on debugging and cross-jumping. */ 2616 return rfs_result (RFS_LIVE_RANGE_SHRINK2, 2617 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2); 2618 } 2619 2620 /* The insn in a schedule group should be issued the first. */ 2621 if (flag_sched_group_heuristic && 2622 SCHED_GROUP_P (tmp) != SCHED_GROUP_P (tmp2)) 2623 return rfs_result (RFS_SCHED_GROUP, SCHED_GROUP_P (tmp2) ? 1 : -1, 2624 tmp, tmp2); 2625 2626 /* Make sure that priority of TMP and TMP2 are initialized. */ 2627 gcc_assert (INSN_PRIORITY_KNOWN (tmp) && INSN_PRIORITY_KNOWN (tmp2)); 2628 2629 if (sched_fusion) 2630 { 2631 /* The instruction that has the same fusion priority as the last 2632 instruction is the instruction we picked next. If that is not 2633 the case, we sort ready list firstly by fusion priority, then 2634 by priority, and at last by INSN_LUID. */ 2635 int a = INSN_FUSION_PRIORITY (tmp); 2636 int b = INSN_FUSION_PRIORITY (tmp2); 2637 int last = -1; 2638 2639 if (last_nondebug_scheduled_insn 2640 && !NOTE_P (last_nondebug_scheduled_insn) 2641 && BLOCK_FOR_INSN (tmp) 2642 == BLOCK_FOR_INSN (last_nondebug_scheduled_insn)) 2643 last = INSN_FUSION_PRIORITY (last_nondebug_scheduled_insn); 2644 2645 if (a != last && b != last) 2646 { 2647 if (a == b) 2648 { 2649 a = INSN_PRIORITY (tmp); 2650 b = INSN_PRIORITY (tmp2); 2651 } 2652 if (a != b) 2653 return rfs_result (RFS_FUSION, b - a, tmp, tmp2); 2654 else 2655 return rfs_result (RFS_FUSION, 2656 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2); 2657 } 2658 else if (a == b) 2659 { 2660 gcc_assert (last_nondebug_scheduled_insn 2661 && !NOTE_P (last_nondebug_scheduled_insn)); 2662 last = INSN_PRIORITY (last_nondebug_scheduled_insn); 2663 2664 a = abs (INSN_PRIORITY (tmp) - last); 2665 b = abs (INSN_PRIORITY (tmp2) - last); 2666 if (a != b) 2667 return rfs_result (RFS_FUSION, a - b, tmp, tmp2); 2668 else 2669 return rfs_result (RFS_FUSION, 2670 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2); 2671 } 2672 else if (a == last) 2673 return rfs_result (RFS_FUSION, -1, tmp, tmp2); 2674 else 2675 return rfs_result (RFS_FUSION, 1, tmp, tmp2); 2676 } 2677 2678 if (sched_pressure != SCHED_PRESSURE_NONE) 2679 { 2680 /* Prefer insn whose scheduling results in the smallest register 2681 pressure excess. */ 2682 if ((diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp) 2683 + insn_delay (tmp) 2684 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2) 2685 - insn_delay (tmp2)))) 2686 return rfs_result (RFS_PRESSURE_DELAY, diff, tmp, tmp2); 2687 } 2688 2689 if (sched_pressure != SCHED_PRESSURE_NONE 2690 && (INSN_TICK (tmp2) > clock_var || INSN_TICK (tmp) > clock_var) 2691 && INSN_TICK (tmp2) != INSN_TICK (tmp)) 2692 { 2693 diff = INSN_TICK (tmp) - INSN_TICK (tmp2); 2694 return rfs_result (RFS_PRESSURE_TICK, diff, tmp, tmp2); 2695 } 2696 2697 /* If we are doing backtracking in this schedule, prefer insns that 2698 have forward dependencies with negative cost against an insn that 2699 was already scheduled. */ 2700 if (current_sched_info->flags & DO_BACKTRACKING) 2701 { 2702 priority_val = FEEDS_BACKTRACK_INSN (tmp2) - FEEDS_BACKTRACK_INSN (tmp); 2703 if (priority_val) 2704 return rfs_result (RFS_FEEDS_BACKTRACK_INSN, priority_val, tmp, tmp2); 2705 } 2706 2707 /* Prefer insn with higher priority. */ 2708 priority_val = INSN_PRIORITY (tmp2) - INSN_PRIORITY (tmp); 2709 2710 if (flag_sched_critical_path_heuristic && priority_val) 2711 return rfs_result (RFS_PRIORITY, priority_val, tmp, tmp2); 2712 2713 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) >= 0) 2714 { 2715 int autopref = autopref_rank_for_schedule (tmp, tmp2); 2716 if (autopref != 0) 2717 return autopref; 2718 } 2719 2720 /* Prefer speculative insn with greater dependencies weakness. */ 2721 if (flag_sched_spec_insn_heuristic && spec_info) 2722 { 2723 ds_t ds1, ds2; 2724 dw_t dw1, dw2; 2725 int dw; 2726 2727 ds1 = TODO_SPEC (tmp) & SPECULATIVE; 2728 if (ds1) 2729 dw1 = ds_weak (ds1); 2730 else 2731 dw1 = NO_DEP_WEAK; 2732 2733 ds2 = TODO_SPEC (tmp2) & SPECULATIVE; 2734 if (ds2) 2735 dw2 = ds_weak (ds2); 2736 else 2737 dw2 = NO_DEP_WEAK; 2738 2739 dw = dw2 - dw1; 2740 if (dw > (NO_DEP_WEAK / 8) || dw < -(NO_DEP_WEAK / 8)) 2741 return rfs_result (RFS_SPECULATION, dw, tmp, tmp2); 2742 } 2743 2744 info_val = (*current_sched_info->rank) (tmp, tmp2); 2745 if (flag_sched_rank_heuristic && info_val) 2746 return rfs_result (RFS_SCHED_RANK, info_val, tmp, tmp2); 2747 2748 /* Compare insns based on their relation to the last scheduled 2749 non-debug insn. */ 2750 if (flag_sched_last_insn_heuristic && last_nondebug_scheduled_insn) 2751 { 2752 dep_t dep1; 2753 dep_t dep2; 2754 rtx_insn *last = last_nondebug_scheduled_insn; 2755 2756 /* Classify the instructions into three classes: 2757 1) Data dependent on last schedule insn. 2758 2) Anti/Output dependent on last scheduled insn. 2759 3) Independent of last scheduled insn, or has latency of one. 2760 Choose the insn from the highest numbered class if different. */ 2761 dep1 = sd_find_dep_between (last, tmp, true); 2762 2763 if (dep1 == NULL || dep_cost (dep1) == 1) 2764 tmp_class = 3; 2765 else if (/* Data dependence. */ 2766 DEP_TYPE (dep1) == REG_DEP_TRUE) 2767 tmp_class = 1; 2768 else 2769 tmp_class = 2; 2770 2771 dep2 = sd_find_dep_between (last, tmp2, true); 2772 2773 if (dep2 == NULL || dep_cost (dep2) == 1) 2774 tmp2_class = 3; 2775 else if (/* Data dependence. */ 2776 DEP_TYPE (dep2) == REG_DEP_TRUE) 2777 tmp2_class = 1; 2778 else 2779 tmp2_class = 2; 2780 2781 if ((val = tmp2_class - tmp_class)) 2782 return rfs_result (RFS_LAST_INSN, val, tmp, tmp2); 2783 } 2784 2785 /* Prefer instructions that occur earlier in the model schedule. */ 2786 if (sched_pressure == SCHED_PRESSURE_MODEL) 2787 { 2788 diff = model_index (tmp) - model_index (tmp2); 2789 if (diff != 0) 2790 return rfs_result (RFS_PRESSURE_INDEX, diff, tmp, tmp2); 2791 } 2792 2793 /* Prefer the insn which has more later insns that depend on it. 2794 This gives the scheduler more freedom when scheduling later 2795 instructions at the expense of added register pressure. */ 2796 2797 val = (dep_list_size (tmp2, SD_LIST_FORW) 2798 - dep_list_size (tmp, SD_LIST_FORW)); 2799 2800 if (flag_sched_dep_count_heuristic && val != 0) 2801 return rfs_result (RFS_DEP_COUNT, val, tmp, tmp2); 2802 2803 /* If insns are equally good, sort by INSN_LUID (original insn order), 2804 so that we make the sort stable. This minimizes instruction movement, 2805 thus minimizing sched's effect on debugging and cross-jumping. */ 2806 return rfs_result (RFS_TIE, INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2); 2807 } 2808 2809 /* Resort the array A in which only element at index N may be out of order. */ 2810 2811 HAIFA_INLINE static void 2812 swap_sort (rtx_insn **a, int n) 2813 { 2814 rtx_insn *insn = a[n - 1]; 2815 int i = n - 2; 2816 2817 while (i >= 0 && rank_for_schedule (a + i, &insn) >= 0) 2818 { 2819 a[i + 1] = a[i]; 2820 i -= 1; 2821 } 2822 a[i + 1] = insn; 2823 } 2824 2825 /* Add INSN to the insn queue so that it can be executed at least 2826 N_CYCLES after the currently executing insn. Preserve insns 2827 chain for debugging purposes. REASON will be printed in debugging 2828 output. */ 2829 2830 HAIFA_INLINE static void 2831 queue_insn (rtx_insn *insn, int n_cycles, const char *reason) 2832 { 2833 int next_q = NEXT_Q_AFTER (q_ptr, n_cycles); 2834 rtx_insn_list *link = alloc_INSN_LIST (insn, insn_queue[next_q]); 2835 int new_tick; 2836 2837 gcc_assert (n_cycles <= max_insn_queue_index); 2838 gcc_assert (!DEBUG_INSN_P (insn)); 2839 2840 insn_queue[next_q] = link; 2841 q_size += 1; 2842 2843 if (sched_verbose >= 2) 2844 { 2845 fprintf (sched_dump, ";;\t\tReady-->Q: insn %s: ", 2846 (*current_sched_info->print_insn) (insn, 0)); 2847 2848 fprintf (sched_dump, "queued for %d cycles (%s).\n", n_cycles, reason); 2849 } 2850 2851 QUEUE_INDEX (insn) = next_q; 2852 2853 if (current_sched_info->flags & DO_BACKTRACKING) 2854 { 2855 new_tick = clock_var + n_cycles; 2856 if (INSN_TICK (insn) == INVALID_TICK || INSN_TICK (insn) < new_tick) 2857 INSN_TICK (insn) = new_tick; 2858 2859 if (INSN_EXACT_TICK (insn) != INVALID_TICK 2860 && INSN_EXACT_TICK (insn) < clock_var + n_cycles) 2861 { 2862 must_backtrack = true; 2863 if (sched_verbose >= 2) 2864 fprintf (sched_dump, ";;\t\tcausing a backtrack.\n"); 2865 } 2866 } 2867 } 2868 2869 /* Remove INSN from queue. */ 2870 static void 2871 queue_remove (rtx_insn *insn) 2872 { 2873 gcc_assert (QUEUE_INDEX (insn) >= 0); 2874 remove_free_INSN_LIST_elem (insn, &insn_queue[QUEUE_INDEX (insn)]); 2875 q_size--; 2876 QUEUE_INDEX (insn) = QUEUE_NOWHERE; 2877 } 2878 2879 /* Return a pointer to the bottom of the ready list, i.e. the insn 2880 with the lowest priority. */ 2881 2882 rtx_insn ** 2883 ready_lastpos (struct ready_list *ready) 2884 { 2885 gcc_assert (ready->n_ready >= 1); 2886 return ready->vec + ready->first - ready->n_ready + 1; 2887 } 2888 2889 /* Add an element INSN to the ready list so that it ends up with the 2890 lowest/highest priority depending on FIRST_P. */ 2891 2892 HAIFA_INLINE static void 2893 ready_add (struct ready_list *ready, rtx_insn *insn, bool first_p) 2894 { 2895 if (!first_p) 2896 { 2897 if (ready->first == ready->n_ready) 2898 { 2899 memmove (ready->vec + ready->veclen - ready->n_ready, 2900 ready_lastpos (ready), 2901 ready->n_ready * sizeof (rtx)); 2902 ready->first = ready->veclen - 1; 2903 } 2904 ready->vec[ready->first - ready->n_ready] = insn; 2905 } 2906 else 2907 { 2908 if (ready->first == ready->veclen - 1) 2909 { 2910 if (ready->n_ready) 2911 /* ready_lastpos() fails when called with (ready->n_ready == 0). */ 2912 memmove (ready->vec + ready->veclen - ready->n_ready - 1, 2913 ready_lastpos (ready), 2914 ready->n_ready * sizeof (rtx)); 2915 ready->first = ready->veclen - 2; 2916 } 2917 ready->vec[++(ready->first)] = insn; 2918 } 2919 2920 ready->n_ready++; 2921 if (DEBUG_INSN_P (insn)) 2922 ready->n_debug++; 2923 2924 gcc_assert (QUEUE_INDEX (insn) != QUEUE_READY); 2925 QUEUE_INDEX (insn) = QUEUE_READY; 2926 2927 if (INSN_EXACT_TICK (insn) != INVALID_TICK 2928 && INSN_EXACT_TICK (insn) < clock_var) 2929 { 2930 must_backtrack = true; 2931 } 2932 } 2933 2934 /* Remove the element with the highest priority from the ready list and 2935 return it. */ 2936 2937 HAIFA_INLINE static rtx_insn * 2938 ready_remove_first (struct ready_list *ready) 2939 { 2940 rtx_insn *t; 2941 2942 gcc_assert (ready->n_ready); 2943 t = ready->vec[ready->first--]; 2944 ready->n_ready--; 2945 if (DEBUG_INSN_P (t)) 2946 ready->n_debug--; 2947 /* If the queue becomes empty, reset it. */ 2948 if (ready->n_ready == 0) 2949 ready->first = ready->veclen - 1; 2950 2951 gcc_assert (QUEUE_INDEX (t) == QUEUE_READY); 2952 QUEUE_INDEX (t) = QUEUE_NOWHERE; 2953 2954 return t; 2955 } 2956 2957 /* The following code implements multi-pass scheduling for the first 2958 cycle. In other words, we will try to choose ready insn which 2959 permits to start maximum number of insns on the same cycle. */ 2960 2961 /* Return a pointer to the element INDEX from the ready. INDEX for 2962 insn with the highest priority is 0, and the lowest priority has 2963 N_READY - 1. */ 2964 2965 rtx_insn * 2966 ready_element (struct ready_list *ready, int index) 2967 { 2968 gcc_assert (ready->n_ready && index < ready->n_ready); 2969 2970 return ready->vec[ready->first - index]; 2971 } 2972 2973 /* Remove the element INDEX from the ready list and return it. INDEX 2974 for insn with the highest priority is 0, and the lowest priority 2975 has N_READY - 1. */ 2976 2977 HAIFA_INLINE static rtx_insn * 2978 ready_remove (struct ready_list *ready, int index) 2979 { 2980 rtx_insn *t; 2981 int i; 2982 2983 if (index == 0) 2984 return ready_remove_first (ready); 2985 gcc_assert (ready->n_ready && index < ready->n_ready); 2986 t = ready->vec[ready->first - index]; 2987 ready->n_ready--; 2988 if (DEBUG_INSN_P (t)) 2989 ready->n_debug--; 2990 for (i = index; i < ready->n_ready; i++) 2991 ready->vec[ready->first - i] = ready->vec[ready->first - i - 1]; 2992 QUEUE_INDEX (t) = QUEUE_NOWHERE; 2993 return t; 2994 } 2995 2996 /* Remove INSN from the ready list. */ 2997 static void 2998 ready_remove_insn (rtx_insn *insn) 2999 { 3000 int i; 3001 3002 for (i = 0; i < readyp->n_ready; i++) 3003 if (ready_element (readyp, i) == insn) 3004 { 3005 ready_remove (readyp, i); 3006 return; 3007 } 3008 gcc_unreachable (); 3009 } 3010 3011 /* Calculate difference of two statistics set WAS and NOW. 3012 Result returned in WAS. */ 3013 static void 3014 rank_for_schedule_stats_diff (rank_for_schedule_stats_t *was, 3015 const rank_for_schedule_stats_t *now) 3016 { 3017 for (int i = 0; i < RFS_N; ++i) 3018 was->stats[i] = now->stats[i] - was->stats[i]; 3019 } 3020 3021 /* Print rank_for_schedule statistics. */ 3022 static void 3023 print_rank_for_schedule_stats (const char *prefix, 3024 const rank_for_schedule_stats_t *stats, 3025 struct ready_list *ready) 3026 { 3027 for (int i = 0; i < RFS_N; ++i) 3028 if (stats->stats[i]) 3029 { 3030 fprintf (sched_dump, "%s%20s: %u", prefix, rfs_str[i], stats->stats[i]); 3031 3032 if (ready != NULL) 3033 /* Print out insns that won due to RFS_<I>. */ 3034 { 3035 rtx_insn **p = ready_lastpos (ready); 3036 3037 fprintf (sched_dump, ":"); 3038 /* Start with 1 since least-priority insn didn't have any wins. */ 3039 for (int j = 1; j < ready->n_ready; ++j) 3040 if (INSN_LAST_RFS_WIN (p[j]) == i) 3041 fprintf (sched_dump, " %s", 3042 (*current_sched_info->print_insn) (p[j], 0)); 3043 } 3044 fprintf (sched_dump, "\n"); 3045 } 3046 } 3047 3048 /* Separate DEBUG_INSNS from normal insns. DEBUG_INSNs go to the end 3049 of array. */ 3050 static void 3051 ready_sort_debug (struct ready_list *ready) 3052 { 3053 int i; 3054 rtx_insn **first = ready_lastpos (ready); 3055 3056 for (i = 0; i < ready->n_ready; ++i) 3057 if (!DEBUG_INSN_P (first[i])) 3058 INSN_RFS_DEBUG_ORIG_ORDER (first[i]) = i; 3059 3060 qsort (first, ready->n_ready, sizeof (rtx), rank_for_schedule_debug); 3061 } 3062 3063 /* Sort non-debug insns in the ready list READY by ascending priority. 3064 Assumes that all debug insns are separated from the real insns. */ 3065 static void 3066 ready_sort_real (struct ready_list *ready) 3067 { 3068 int i; 3069 rtx_insn **first = ready_lastpos (ready); 3070 int n_ready_real = ready->n_ready - ready->n_debug; 3071 3072 if (sched_pressure == SCHED_PRESSURE_WEIGHTED) 3073 for (i = 0; i < n_ready_real; ++i) 3074 setup_insn_reg_pressure_info (first[i]); 3075 else if (sched_pressure == SCHED_PRESSURE_MODEL 3076 && model_curr_point < model_num_insns) 3077 model_set_excess_costs (first, n_ready_real); 3078 3079 rank_for_schedule_stats_t stats1; 3080 if (sched_verbose >= 4) 3081 stats1 = rank_for_schedule_stats; 3082 3083 if (n_ready_real == 2) 3084 swap_sort (first, n_ready_real); 3085 else if (n_ready_real > 2) 3086 qsort (first, n_ready_real, sizeof (rtx), rank_for_schedule); 3087 3088 if (sched_verbose >= 4) 3089 { 3090 rank_for_schedule_stats_diff (&stats1, &rank_for_schedule_stats); 3091 print_rank_for_schedule_stats (";;\t\t", &stats1, ready); 3092 } 3093 } 3094 3095 /* Sort the ready list READY by ascending priority. */ 3096 static void 3097 ready_sort (struct ready_list *ready) 3098 { 3099 if (ready->n_debug > 0) 3100 ready_sort_debug (ready); 3101 else 3102 ready_sort_real (ready); 3103 } 3104 3105 /* PREV is an insn that is ready to execute. Adjust its priority if that 3106 will help shorten or lengthen register lifetimes as appropriate. Also 3107 provide a hook for the target to tweak itself. */ 3108 3109 HAIFA_INLINE static void 3110 adjust_priority (rtx_insn *prev) 3111 { 3112 /* ??? There used to be code here to try and estimate how an insn 3113 affected register lifetimes, but it did it by looking at REG_DEAD 3114 notes, which we removed in schedule_region. Nor did it try to 3115 take into account register pressure or anything useful like that. 3116 3117 Revisit when we have a machine model to work with and not before. */ 3118 3119 if (targetm.sched.adjust_priority) 3120 INSN_PRIORITY (prev) = 3121 targetm.sched.adjust_priority (prev, INSN_PRIORITY (prev)); 3122 } 3123 3124 /* Advance DFA state STATE on one cycle. */ 3125 void 3126 advance_state (state_t state) 3127 { 3128 if (targetm.sched.dfa_pre_advance_cycle) 3129 targetm.sched.dfa_pre_advance_cycle (); 3130 3131 if (targetm.sched.dfa_pre_cycle_insn) 3132 state_transition (state, 3133 targetm.sched.dfa_pre_cycle_insn ()); 3134 3135 state_transition (state, NULL); 3136 3137 if (targetm.sched.dfa_post_cycle_insn) 3138 state_transition (state, 3139 targetm.sched.dfa_post_cycle_insn ()); 3140 3141 if (targetm.sched.dfa_post_advance_cycle) 3142 targetm.sched.dfa_post_advance_cycle (); 3143 } 3144 3145 /* Advance time on one cycle. */ 3146 HAIFA_INLINE static void 3147 advance_one_cycle (void) 3148 { 3149 advance_state (curr_state); 3150 if (sched_verbose >= 4) 3151 fprintf (sched_dump, ";;\tAdvance the current state.\n"); 3152 } 3153 3154 /* Update register pressure after scheduling INSN. */ 3155 static void 3156 update_register_pressure (rtx_insn *insn) 3157 { 3158 struct reg_use_data *use; 3159 struct reg_set_data *set; 3160 3161 gcc_checking_assert (!DEBUG_INSN_P (insn)); 3162 3163 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use) 3164 if (dying_use_p (use)) 3165 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, 3166 use->regno, false); 3167 for (set = INSN_REG_SET_LIST (insn); set != NULL; set = set->next_insn_set) 3168 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, 3169 set->regno, true); 3170 } 3171 3172 /* Set up or update (if UPDATE_P) max register pressure (see its 3173 meaning in sched-int.h::_haifa_insn_data) for all current BB insns 3174 after insn AFTER. */ 3175 static void 3176 setup_insn_max_reg_pressure (rtx_insn *after, bool update_p) 3177 { 3178 int i, p; 3179 bool eq_p; 3180 rtx_insn *insn; 3181 static int max_reg_pressure[N_REG_CLASSES]; 3182 3183 save_reg_pressure (); 3184 for (i = 0; i < ira_pressure_classes_num; i++) 3185 max_reg_pressure[ira_pressure_classes[i]] 3186 = curr_reg_pressure[ira_pressure_classes[i]]; 3187 for (insn = NEXT_INSN (after); 3188 insn != NULL_RTX && ! BARRIER_P (insn) 3189 && BLOCK_FOR_INSN (insn) == BLOCK_FOR_INSN (after); 3190 insn = NEXT_INSN (insn)) 3191 if (NONDEBUG_INSN_P (insn)) 3192 { 3193 eq_p = true; 3194 for (i = 0; i < ira_pressure_classes_num; i++) 3195 { 3196 p = max_reg_pressure[ira_pressure_classes[i]]; 3197 if (INSN_MAX_REG_PRESSURE (insn)[i] != p) 3198 { 3199 eq_p = false; 3200 INSN_MAX_REG_PRESSURE (insn)[i] 3201 = max_reg_pressure[ira_pressure_classes[i]]; 3202 } 3203 } 3204 if (update_p && eq_p) 3205 break; 3206 update_register_pressure (insn); 3207 for (i = 0; i < ira_pressure_classes_num; i++) 3208 if (max_reg_pressure[ira_pressure_classes[i]] 3209 < curr_reg_pressure[ira_pressure_classes[i]]) 3210 max_reg_pressure[ira_pressure_classes[i]] 3211 = curr_reg_pressure[ira_pressure_classes[i]]; 3212 } 3213 restore_reg_pressure (); 3214 } 3215 3216 /* Update the current register pressure after scheduling INSN. Update 3217 also max register pressure for unscheduled insns of the current 3218 BB. */ 3219 static void 3220 update_reg_and_insn_max_reg_pressure (rtx_insn *insn) 3221 { 3222 int i; 3223 int before[N_REG_CLASSES]; 3224 3225 for (i = 0; i < ira_pressure_classes_num; i++) 3226 before[i] = curr_reg_pressure[ira_pressure_classes[i]]; 3227 update_register_pressure (insn); 3228 for (i = 0; i < ira_pressure_classes_num; i++) 3229 if (curr_reg_pressure[ira_pressure_classes[i]] != before[i]) 3230 break; 3231 if (i < ira_pressure_classes_num) 3232 setup_insn_max_reg_pressure (insn, true); 3233 } 3234 3235 /* Set up register pressure at the beginning of basic block BB whose 3236 insns starting after insn AFTER. Set up also max register pressure 3237 for all insns of the basic block. */ 3238 void 3239 sched_setup_bb_reg_pressure_info (basic_block bb, rtx_insn *after) 3240 { 3241 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED); 3242 initiate_bb_reg_pressure_info (bb); 3243 setup_insn_max_reg_pressure (after, false); 3244 } 3245 3246 /* If doing predication while scheduling, verify whether INSN, which 3247 has just been scheduled, clobbers the conditions of any 3248 instructions that must be predicated in order to break their 3249 dependencies. If so, remove them from the queues so that they will 3250 only be scheduled once their control dependency is resolved. */ 3251 3252 static void 3253 check_clobbered_conditions (rtx_insn *insn) 3254 { 3255 HARD_REG_SET t; 3256 int i; 3257 3258 if ((current_sched_info->flags & DO_PREDICATION) == 0) 3259 return; 3260 3261 find_all_hard_reg_sets (insn, &t, true); 3262 3263 restart: 3264 for (i = 0; i < ready.n_ready; i++) 3265 { 3266 rtx_insn *x = ready_element (&ready, i); 3267 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t)) 3268 { 3269 ready_remove_insn (x); 3270 goto restart; 3271 } 3272 } 3273 for (i = 0; i <= max_insn_queue_index; i++) 3274 { 3275 rtx_insn_list *link; 3276 int q = NEXT_Q_AFTER (q_ptr, i); 3277 3278 restart_queue: 3279 for (link = insn_queue[q]; link; link = link->next ()) 3280 { 3281 rtx_insn *x = link->insn (); 3282 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t)) 3283 { 3284 queue_remove (x); 3285 goto restart_queue; 3286 } 3287 } 3288 } 3289 } 3290 3291 /* Return (in order): 3292 3293 - positive if INSN adversely affects the pressure on one 3294 register class 3295 3296 - negative if INSN reduces the pressure on one register class 3297 3298 - 0 if INSN doesn't affect the pressure on any register class. */ 3299 3300 static int 3301 model_classify_pressure (struct model_insn_info *insn) 3302 { 3303 struct reg_pressure_data *reg_pressure; 3304 int death[N_REG_CLASSES]; 3305 int pci, cl, sum; 3306 3307 calculate_reg_deaths (insn->insn, death); 3308 reg_pressure = INSN_REG_PRESSURE (insn->insn); 3309 sum = 0; 3310 for (pci = 0; pci < ira_pressure_classes_num; pci++) 3311 { 3312 cl = ira_pressure_classes[pci]; 3313 if (death[cl] < reg_pressure[pci].set_increase) 3314 return 1; 3315 sum += reg_pressure[pci].set_increase - death[cl]; 3316 } 3317 return sum; 3318 } 3319 3320 /* Return true if INSN1 should come before INSN2 in the model schedule. */ 3321 3322 static int 3323 model_order_p (struct model_insn_info *insn1, struct model_insn_info *insn2) 3324 { 3325 unsigned int height1, height2; 3326 unsigned int priority1, priority2; 3327 3328 /* Prefer instructions with a higher model priority. */ 3329 if (insn1->model_priority != insn2->model_priority) 3330 return insn1->model_priority > insn2->model_priority; 3331 3332 /* Combine the length of the longest path of satisfied true dependencies 3333 that leads to each instruction (depth) with the length of the longest 3334 path of any dependencies that leads from the instruction (alap). 3335 Prefer instructions with the greatest combined length. If the combined 3336 lengths are equal, prefer instructions with the greatest depth. 3337 3338 The idea is that, if we have a set S of "equal" instructions that each 3339 have ALAP value X, and we pick one such instruction I, any true-dependent 3340 successors of I that have ALAP value X - 1 should be preferred over S. 3341 This encourages the schedule to be "narrow" rather than "wide". 3342 However, if I is a low-priority instruction that we decided to 3343 schedule because of its model_classify_pressure, and if there 3344 is a set of higher-priority instructions T, the aforementioned 3345 successors of I should not have the edge over T. */ 3346 height1 = insn1->depth + insn1->alap; 3347 height2 = insn2->depth + insn2->alap; 3348 if (height1 != height2) 3349 return height1 > height2; 3350 if (insn1->depth != insn2->depth) 3351 return insn1->depth > insn2->depth; 3352 3353 /* We have no real preference between INSN1 an INSN2 as far as attempts 3354 to reduce pressure go. Prefer instructions with higher priorities. */ 3355 priority1 = INSN_PRIORITY (insn1->insn); 3356 priority2 = INSN_PRIORITY (insn2->insn); 3357 if (priority1 != priority2) 3358 return priority1 > priority2; 3359 3360 /* Use the original rtl sequence as a tie-breaker. */ 3361 return insn1 < insn2; 3362 } 3363 3364 /* Add INSN to the model worklist immediately after PREV. Add it to the 3365 beginning of the list if PREV is null. */ 3366 3367 static void 3368 model_add_to_worklist_at (struct model_insn_info *insn, 3369 struct model_insn_info *prev) 3370 { 3371 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_NOWHERE); 3372 QUEUE_INDEX (insn->insn) = QUEUE_READY; 3373 3374 insn->prev = prev; 3375 if (prev) 3376 { 3377 insn->next = prev->next; 3378 prev->next = insn; 3379 } 3380 else 3381 { 3382 insn->next = model_worklist; 3383 model_worklist = insn; 3384 } 3385 if (insn->next) 3386 insn->next->prev = insn; 3387 } 3388 3389 /* Remove INSN from the model worklist. */ 3390 3391 static void 3392 model_remove_from_worklist (struct model_insn_info *insn) 3393 { 3394 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_READY); 3395 QUEUE_INDEX (insn->insn) = QUEUE_NOWHERE; 3396 3397 if (insn->prev) 3398 insn->prev->next = insn->next; 3399 else 3400 model_worklist = insn->next; 3401 if (insn->next) 3402 insn->next->prev = insn->prev; 3403 } 3404 3405 /* Add INSN to the model worklist. Start looking for a suitable position 3406 between neighbors PREV and NEXT, testing at most MAX_SCHED_READY_INSNS 3407 insns either side. A null PREV indicates the beginning of the list and 3408 a null NEXT indicates the end. */ 3409 3410 static void 3411 model_add_to_worklist (struct model_insn_info *insn, 3412 struct model_insn_info *prev, 3413 struct model_insn_info *next) 3414 { 3415 int count; 3416 3417 count = MAX_SCHED_READY_INSNS; 3418 if (count > 0 && prev && model_order_p (insn, prev)) 3419 do 3420 { 3421 count--; 3422 prev = prev->prev; 3423 } 3424 while (count > 0 && prev && model_order_p (insn, prev)); 3425 else 3426 while (count > 0 && next && model_order_p (next, insn)) 3427 { 3428 count--; 3429 prev = next; 3430 next = next->next; 3431 } 3432 model_add_to_worklist_at (insn, prev); 3433 } 3434 3435 /* INSN may now have a higher priority (in the model_order_p sense) 3436 than before. Move it up the worklist if necessary. */ 3437 3438 static void 3439 model_promote_insn (struct model_insn_info *insn) 3440 { 3441 struct model_insn_info *prev; 3442 int count; 3443 3444 prev = insn->prev; 3445 count = MAX_SCHED_READY_INSNS; 3446 while (count > 0 && prev && model_order_p (insn, prev)) 3447 { 3448 count--; 3449 prev = prev->prev; 3450 } 3451 if (prev != insn->prev) 3452 { 3453 model_remove_from_worklist (insn); 3454 model_add_to_worklist_at (insn, prev); 3455 } 3456 } 3457 3458 /* Add INSN to the end of the model schedule. */ 3459 3460 static void 3461 model_add_to_schedule (rtx_insn *insn) 3462 { 3463 unsigned int point; 3464 3465 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE); 3466 QUEUE_INDEX (insn) = QUEUE_SCHEDULED; 3467 3468 point = model_schedule.length (); 3469 model_schedule.quick_push (insn); 3470 INSN_MODEL_INDEX (insn) = point + 1; 3471 } 3472 3473 /* Analyze the instructions that are to be scheduled, setting up 3474 MODEL_INSN_INFO (...) and model_num_insns accordingly. Add ready 3475 instructions to model_worklist. */ 3476 3477 static void 3478 model_analyze_insns (void) 3479 { 3480 rtx_insn *start, *end, *iter; 3481 sd_iterator_def sd_it; 3482 dep_t dep; 3483 struct model_insn_info *insn, *con; 3484 3485 model_num_insns = 0; 3486 start = PREV_INSN (current_sched_info->next_tail); 3487 end = current_sched_info->prev_head; 3488 for (iter = start; iter != end; iter = PREV_INSN (iter)) 3489 if (NONDEBUG_INSN_P (iter)) 3490 { 3491 insn = MODEL_INSN_INFO (iter); 3492 insn->insn = iter; 3493 FOR_EACH_DEP (iter, SD_LIST_FORW, sd_it, dep) 3494 { 3495 con = MODEL_INSN_INFO (DEP_CON (dep)); 3496 if (con->insn && insn->alap < con->alap + 1) 3497 insn->alap = con->alap + 1; 3498 } 3499 3500 insn->old_queue = QUEUE_INDEX (iter); 3501 QUEUE_INDEX (iter) = QUEUE_NOWHERE; 3502 3503 insn->unscheduled_preds = dep_list_size (iter, SD_LIST_HARD_BACK); 3504 if (insn->unscheduled_preds == 0) 3505 model_add_to_worklist (insn, NULL, model_worklist); 3506 3507 model_num_insns++; 3508 } 3509 } 3510 3511 /* The global state describes the register pressure at the start of the 3512 model schedule. Initialize GROUP accordingly. */ 3513 3514 static void 3515 model_init_pressure_group (struct model_pressure_group *group) 3516 { 3517 int pci, cl; 3518 3519 for (pci = 0; pci < ira_pressure_classes_num; pci++) 3520 { 3521 cl = ira_pressure_classes[pci]; 3522 group->limits[pci].pressure = curr_reg_pressure[cl]; 3523 group->limits[pci].point = 0; 3524 } 3525 /* Use index model_num_insns to record the state after the last 3526 instruction in the model schedule. */ 3527 group->model = XNEWVEC (struct model_pressure_data, 3528 (model_num_insns + 1) * ira_pressure_classes_num); 3529 } 3530 3531 /* Record that MODEL_REF_PRESSURE (GROUP, POINT, PCI) is PRESSURE. 3532 Update the maximum pressure for the whole schedule. */ 3533 3534 static void 3535 model_record_pressure (struct model_pressure_group *group, 3536 int point, int pci, int pressure) 3537 { 3538 MODEL_REF_PRESSURE (group, point, pci) = pressure; 3539 if (group->limits[pci].pressure < pressure) 3540 { 3541 group->limits[pci].pressure = pressure; 3542 group->limits[pci].point = point; 3543 } 3544 } 3545 3546 /* INSN has just been added to the end of the model schedule. Record its 3547 register-pressure information. */ 3548 3549 static void 3550 model_record_pressures (struct model_insn_info *insn) 3551 { 3552 struct reg_pressure_data *reg_pressure; 3553 int point, pci, cl, delta; 3554 int death[N_REG_CLASSES]; 3555 3556 point = model_index (insn->insn); 3557 if (sched_verbose >= 2) 3558 { 3559 if (point == 0) 3560 { 3561 fprintf (sched_dump, "\n;;\tModel schedule:\n;;\n"); 3562 fprintf (sched_dump, ";;\t| idx insn | mpri hght dpth prio |\n"); 3563 } 3564 fprintf (sched_dump, ";;\t| %3d %4d | %4d %4d %4d %4d | %-30s ", 3565 point, INSN_UID (insn->insn), insn->model_priority, 3566 insn->depth + insn->alap, insn->depth, 3567 INSN_PRIORITY (insn->insn), 3568 str_pattern_slim (PATTERN (insn->insn))); 3569 } 3570 calculate_reg_deaths (insn->insn, death); 3571 reg_pressure = INSN_REG_PRESSURE (insn->insn); 3572 for (pci = 0; pci < ira_pressure_classes_num; pci++) 3573 { 3574 cl = ira_pressure_classes[pci]; 3575 delta = reg_pressure[pci].set_increase - death[cl]; 3576 if (sched_verbose >= 2) 3577 fprintf (sched_dump, " %s:[%d,%+d]", reg_class_names[cl], 3578 curr_reg_pressure[cl], delta); 3579 model_record_pressure (&model_before_pressure, point, pci, 3580 curr_reg_pressure[cl]); 3581 } 3582 if (sched_verbose >= 2) 3583 fprintf (sched_dump, "\n"); 3584 } 3585 3586 /* All instructions have been added to the model schedule. Record the 3587 final register pressure in GROUP and set up all MODEL_MAX_PRESSUREs. */ 3588 3589 static void 3590 model_record_final_pressures (struct model_pressure_group *group) 3591 { 3592 int point, pci, max_pressure, ref_pressure, cl; 3593 3594 for (pci = 0; pci < ira_pressure_classes_num; pci++) 3595 { 3596 /* Record the final pressure for this class. */ 3597 cl = ira_pressure_classes[pci]; 3598 point = model_num_insns; 3599 ref_pressure = curr_reg_pressure[cl]; 3600 model_record_pressure (group, point, pci, ref_pressure); 3601 3602 /* Record the original maximum pressure. */ 3603 group->limits[pci].orig_pressure = group->limits[pci].pressure; 3604 3605 /* Update the MODEL_MAX_PRESSURE for every point of the schedule. */ 3606 max_pressure = ref_pressure; 3607 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure; 3608 while (point > 0) 3609 { 3610 point--; 3611 ref_pressure = MODEL_REF_PRESSURE (group, point, pci); 3612 max_pressure = MAX (max_pressure, ref_pressure); 3613 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure; 3614 } 3615 } 3616 } 3617 3618 /* Update all successors of INSN, given that INSN has just been scheduled. */ 3619 3620 static void 3621 model_add_successors_to_worklist (struct model_insn_info *insn) 3622 { 3623 sd_iterator_def sd_it; 3624 struct model_insn_info *con; 3625 dep_t dep; 3626 3627 FOR_EACH_DEP (insn->insn, SD_LIST_FORW, sd_it, dep) 3628 { 3629 con = MODEL_INSN_INFO (DEP_CON (dep)); 3630 /* Ignore debug instructions, and instructions from other blocks. */ 3631 if (con->insn) 3632 { 3633 con->unscheduled_preds--; 3634 3635 /* Update the depth field of each true-dependent successor. 3636 Increasing the depth gives them a higher priority than 3637 before. */ 3638 if (DEP_TYPE (dep) == REG_DEP_TRUE && con->depth < insn->depth + 1) 3639 { 3640 con->depth = insn->depth + 1; 3641 if (QUEUE_INDEX (con->insn) == QUEUE_READY) 3642 model_promote_insn (con); 3643 } 3644 3645 /* If this is a true dependency, or if there are no remaining 3646 dependencies for CON (meaning that CON only had non-true 3647 dependencies), make sure that CON is on the worklist. 3648 We don't bother otherwise because it would tend to fill the 3649 worklist with a lot of low-priority instructions that are not 3650 yet ready to issue. */ 3651 if ((con->depth > 0 || con->unscheduled_preds == 0) 3652 && QUEUE_INDEX (con->insn) == QUEUE_NOWHERE) 3653 model_add_to_worklist (con, insn, insn->next); 3654 } 3655 } 3656 } 3657 3658 /* Give INSN a higher priority than any current instruction, then give 3659 unscheduled predecessors of INSN a higher priority still. If any of 3660 those predecessors are not on the model worklist, do the same for its 3661 predecessors, and so on. */ 3662 3663 static void 3664 model_promote_predecessors (struct model_insn_info *insn) 3665 { 3666 struct model_insn_info *pro, *first; 3667 sd_iterator_def sd_it; 3668 dep_t dep; 3669 3670 if (sched_verbose >= 7) 3671 fprintf (sched_dump, ";;\t+--- priority of %d = %d, priority of", 3672 INSN_UID (insn->insn), model_next_priority); 3673 insn->model_priority = model_next_priority++; 3674 model_remove_from_worklist (insn); 3675 model_add_to_worklist_at (insn, NULL); 3676 3677 first = NULL; 3678 for (;;) 3679 { 3680 FOR_EACH_DEP (insn->insn, SD_LIST_HARD_BACK, sd_it, dep) 3681 { 3682 pro = MODEL_INSN_INFO (DEP_PRO (dep)); 3683 /* The first test is to ignore debug instructions, and instructions 3684 from other blocks. */ 3685 if (pro->insn 3686 && pro->model_priority != model_next_priority 3687 && QUEUE_INDEX (pro->insn) != QUEUE_SCHEDULED) 3688 { 3689 pro->model_priority = model_next_priority; 3690 if (sched_verbose >= 7) 3691 fprintf (sched_dump, " %d", INSN_UID (pro->insn)); 3692 if (QUEUE_INDEX (pro->insn) == QUEUE_READY) 3693 { 3694 /* PRO is already in the worklist, but it now has 3695 a higher priority than before. Move it at the 3696 appropriate place. */ 3697 model_remove_from_worklist (pro); 3698 model_add_to_worklist (pro, NULL, model_worklist); 3699 } 3700 else 3701 { 3702 /* PRO isn't in the worklist. Recursively process 3703 its predecessors until we find one that is. */ 3704 pro->next = first; 3705 first = pro; 3706 } 3707 } 3708 } 3709 if (!first) 3710 break; 3711 insn = first; 3712 first = insn->next; 3713 } 3714 if (sched_verbose >= 7) 3715 fprintf (sched_dump, " = %d\n", model_next_priority); 3716 model_next_priority++; 3717 } 3718 3719 /* Pick one instruction from model_worklist and process it. */ 3720 3721 static void 3722 model_choose_insn (void) 3723 { 3724 struct model_insn_info *insn, *fallback; 3725 int count; 3726 3727 if (sched_verbose >= 7) 3728 { 3729 fprintf (sched_dump, ";;\t+--- worklist:\n"); 3730 insn = model_worklist; 3731 count = MAX_SCHED_READY_INSNS; 3732 while (count > 0 && insn) 3733 { 3734 fprintf (sched_dump, ";;\t+--- %d [%d, %d, %d, %d]\n", 3735 INSN_UID (insn->insn), insn->model_priority, 3736 insn->depth + insn->alap, insn->depth, 3737 INSN_PRIORITY (insn->insn)); 3738 count--; 3739 insn = insn->next; 3740 } 3741 } 3742 3743 /* Look for a ready instruction whose model_classify_priority is zero 3744 or negative, picking the highest-priority one. Adding such an 3745 instruction to the schedule now should do no harm, and may actually 3746 do some good. 3747 3748 Failing that, see whether there is an instruction with the highest 3749 extant model_priority that is not yet ready, but which would reduce 3750 pressure if it became ready. This is designed to catch cases like: 3751 3752 (set (mem (reg R1)) (reg R2)) 3753 3754 where the instruction is the last remaining use of R1 and where the 3755 value of R2 is not yet available (or vice versa). The death of R1 3756 means that this instruction already reduces pressure. It is of 3757 course possible that the computation of R2 involves other registers 3758 that are hard to kill, but such cases are rare enough for this 3759 heuristic to be a win in general. 3760 3761 Failing that, just pick the highest-priority instruction in the 3762 worklist. */ 3763 count = MAX_SCHED_READY_INSNS; 3764 insn = model_worklist; 3765 fallback = 0; 3766 for (;;) 3767 { 3768 if (count == 0 || !insn) 3769 { 3770 insn = fallback ? fallback : model_worklist; 3771 break; 3772 } 3773 if (insn->unscheduled_preds) 3774 { 3775 if (model_worklist->model_priority == insn->model_priority 3776 && !fallback 3777 && model_classify_pressure (insn) < 0) 3778 fallback = insn; 3779 } 3780 else 3781 { 3782 if (model_classify_pressure (insn) <= 0) 3783 break; 3784 } 3785 count--; 3786 insn = insn->next; 3787 } 3788 3789 if (sched_verbose >= 7 && insn != model_worklist) 3790 { 3791 if (insn->unscheduled_preds) 3792 fprintf (sched_dump, ";;\t+--- promoting insn %d, with dependencies\n", 3793 INSN_UID (insn->insn)); 3794 else 3795 fprintf (sched_dump, ";;\t+--- promoting insn %d, which is ready\n", 3796 INSN_UID (insn->insn)); 3797 } 3798 if (insn->unscheduled_preds) 3799 /* INSN isn't yet ready to issue. Give all its predecessors the 3800 highest priority. */ 3801 model_promote_predecessors (insn); 3802 else 3803 { 3804 /* INSN is ready. Add it to the end of model_schedule and 3805 process its successors. */ 3806 model_add_successors_to_worklist (insn); 3807 model_remove_from_worklist (insn); 3808 model_add_to_schedule (insn->insn); 3809 model_record_pressures (insn); 3810 update_register_pressure (insn->insn); 3811 } 3812 } 3813 3814 /* Restore all QUEUE_INDEXs to the values that they had before 3815 model_start_schedule was called. */ 3816 3817 static void 3818 model_reset_queue_indices (void) 3819 { 3820 unsigned int i; 3821 rtx_insn *insn; 3822 3823 FOR_EACH_VEC_ELT (model_schedule, i, insn) 3824 QUEUE_INDEX (insn) = MODEL_INSN_INFO (insn)->old_queue; 3825 } 3826 3827 /* We have calculated the model schedule and spill costs. Print a summary 3828 to sched_dump. */ 3829 3830 static void 3831 model_dump_pressure_summary (void) 3832 { 3833 int pci, cl; 3834 3835 fprintf (sched_dump, ";; Pressure summary:"); 3836 for (pci = 0; pci < ira_pressure_classes_num; pci++) 3837 { 3838 cl = ira_pressure_classes[pci]; 3839 fprintf (sched_dump, " %s:%d", reg_class_names[cl], 3840 model_before_pressure.limits[pci].pressure); 3841 } 3842 fprintf (sched_dump, "\n\n"); 3843 } 3844 3845 /* Initialize the SCHED_PRESSURE_MODEL information for the current 3846 scheduling region. */ 3847 3848 static void 3849 model_start_schedule (basic_block bb) 3850 { 3851 model_next_priority = 1; 3852 model_schedule.create (sched_max_luid); 3853 model_insns = XCNEWVEC (struct model_insn_info, sched_max_luid); 3854 3855 gcc_assert (bb == BLOCK_FOR_INSN (NEXT_INSN (current_sched_info->prev_head))); 3856 initiate_reg_pressure_info (df_get_live_in (bb)); 3857 3858 model_analyze_insns (); 3859 model_init_pressure_group (&model_before_pressure); 3860 while (model_worklist) 3861 model_choose_insn (); 3862 gcc_assert (model_num_insns == (int) model_schedule.length ()); 3863 if (sched_verbose >= 2) 3864 fprintf (sched_dump, "\n"); 3865 3866 model_record_final_pressures (&model_before_pressure); 3867 model_reset_queue_indices (); 3868 3869 XDELETEVEC (model_insns); 3870 3871 model_curr_point = 0; 3872 initiate_reg_pressure_info (df_get_live_in (bb)); 3873 if (sched_verbose >= 1) 3874 model_dump_pressure_summary (); 3875 } 3876 3877 /* Free the information associated with GROUP. */ 3878 3879 static void 3880 model_finalize_pressure_group (struct model_pressure_group *group) 3881 { 3882 XDELETEVEC (group->model); 3883 } 3884 3885 /* Free the information created by model_start_schedule. */ 3886 3887 static void 3888 model_end_schedule (void) 3889 { 3890 model_finalize_pressure_group (&model_before_pressure); 3891 model_schedule.release (); 3892 } 3893 3894 /* Prepare reg pressure scheduling for basic block BB. */ 3895 static void 3896 sched_pressure_start_bb (basic_block bb) 3897 { 3898 /* Set the number of available registers for each class taking into account 3899 relative probability of current basic block versus function prologue and 3900 epilogue. 3901 * If the basic block executes much more often than the prologue/epilogue 3902 (e.g., inside a hot loop), then cost of spill in the prologue is close to 3903 nil, so the effective number of available registers is 3904 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl] - 0). 3905 * If the basic block executes as often as the prologue/epilogue, 3906 then spill in the block is as costly as in the prologue, so the effective 3907 number of available registers is 3908 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl] 3909 - call_saved_regs_num[cl]). 3910 Note that all-else-equal, we prefer to spill in the prologue, since that 3911 allows "extra" registers for other basic blocks of the function. 3912 * If the basic block is on the cold path of the function and executes 3913 rarely, then we should always prefer to spill in the block, rather than 3914 in the prologue/epilogue. The effective number of available register is 3915 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl] 3916 - call_saved_regs_num[cl]). */ 3917 { 3918 int i; 3919 int entry_freq = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count.to_frequency (cfun); 3920 int bb_freq = bb->count.to_frequency (cfun); 3921 3922 if (bb_freq == 0) 3923 { 3924 if (entry_freq == 0) 3925 entry_freq = bb_freq = 1; 3926 } 3927 if (bb_freq < entry_freq) 3928 bb_freq = entry_freq; 3929 3930 for (i = 0; i < ira_pressure_classes_num; ++i) 3931 { 3932 enum reg_class cl = ira_pressure_classes[i]; 3933 sched_class_regs_num[cl] = ira_class_hard_regs_num[cl] 3934 - fixed_regs_num[cl]; 3935 sched_class_regs_num[cl] 3936 -= (call_saved_regs_num[cl] * entry_freq) / bb_freq; 3937 } 3938 } 3939 3940 if (sched_pressure == SCHED_PRESSURE_MODEL) 3941 model_start_schedule (bb); 3942 } 3943 3944 /* A structure that holds local state for the loop in schedule_block. */ 3945 struct sched_block_state 3946 { 3947 /* True if no real insns have been scheduled in the current cycle. */ 3948 bool first_cycle_insn_p; 3949 /* True if a shadow insn has been scheduled in the current cycle, which 3950 means that no more normal insns can be issued. */ 3951 bool shadows_only_p; 3952 /* True if we're winding down a modulo schedule, which means that we only 3953 issue insns with INSN_EXACT_TICK set. */ 3954 bool modulo_epilogue; 3955 /* Initialized with the machine's issue rate every cycle, and updated 3956 by calls to the variable_issue hook. */ 3957 int can_issue_more; 3958 }; 3959 3960 /* INSN is the "currently executing insn". Launch each insn which was 3961 waiting on INSN. READY is the ready list which contains the insns 3962 that are ready to fire. CLOCK is the current cycle. The function 3963 returns necessary cycle advance after issuing the insn (it is not 3964 zero for insns in a schedule group). */ 3965 3966 static int 3967 schedule_insn (rtx_insn *insn) 3968 { 3969 sd_iterator_def sd_it; 3970 dep_t dep; 3971 int i; 3972 int advance = 0; 3973 3974 if (sched_verbose >= 1) 3975 { 3976 struct reg_pressure_data *pressure_info; 3977 fprintf (sched_dump, ";;\t%3i--> %s %-40s:", 3978 clock_var, (*current_sched_info->print_insn) (insn, 1), 3979 str_pattern_slim (PATTERN (insn))); 3980 3981 if (recog_memoized (insn) < 0) 3982 fprintf (sched_dump, "nothing"); 3983 else 3984 print_reservation (sched_dump, insn); 3985 pressure_info = INSN_REG_PRESSURE (insn); 3986 if (pressure_info != NULL) 3987 { 3988 fputc (':', sched_dump); 3989 for (i = 0; i < ira_pressure_classes_num; i++) 3990 fprintf (sched_dump, "%s%s%+d(%d)", 3991 scheduled_insns.length () > 1 3992 && INSN_LUID (insn) 3993 < INSN_LUID (scheduled_insns[scheduled_insns.length () - 2]) ? "@" : "", 3994 reg_class_names[ira_pressure_classes[i]], 3995 pressure_info[i].set_increase, pressure_info[i].change); 3996 } 3997 if (sched_pressure == SCHED_PRESSURE_MODEL 3998 && model_curr_point < model_num_insns 3999 && model_index (insn) == model_curr_point) 4000 fprintf (sched_dump, ":model %d", model_curr_point); 4001 fputc ('\n', sched_dump); 4002 } 4003 4004 if (sched_pressure == SCHED_PRESSURE_WEIGHTED && !DEBUG_INSN_P (insn)) 4005 update_reg_and_insn_max_reg_pressure (insn); 4006 4007 /* Scheduling instruction should have all its dependencies resolved and 4008 should have been removed from the ready list. */ 4009 gcc_assert (sd_lists_empty_p (insn, SD_LIST_HARD_BACK)); 4010 4011 /* Reset debug insns invalidated by moving this insn. */ 4012 if (MAY_HAVE_DEBUG_BIND_INSNS && !DEBUG_INSN_P (insn)) 4013 for (sd_it = sd_iterator_start (insn, SD_LIST_BACK); 4014 sd_iterator_cond (&sd_it, &dep);) 4015 { 4016 rtx_insn *dbg = DEP_PRO (dep); 4017 struct reg_use_data *use, *next; 4018 4019 if (DEP_STATUS (dep) & DEP_CANCELLED) 4020 { 4021 sd_iterator_next (&sd_it); 4022 continue; 4023 } 4024 4025 gcc_assert (DEBUG_BIND_INSN_P (dbg)); 4026 4027 if (sched_verbose >= 6) 4028 fprintf (sched_dump, ";;\t\tresetting: debug insn %d\n", 4029 INSN_UID (dbg)); 4030 4031 /* ??? Rather than resetting the debug insn, we might be able 4032 to emit a debug temp before the just-scheduled insn, but 4033 this would involve checking that the expression at the 4034 point of the debug insn is equivalent to the expression 4035 before the just-scheduled insn. They might not be: the 4036 expression in the debug insn may depend on other insns not 4037 yet scheduled that set MEMs, REGs or even other debug 4038 insns. It's not clear that attempting to preserve debug 4039 information in these cases is worth the effort, given how 4040 uncommon these resets are and the likelihood that the debug 4041 temps introduced won't survive the schedule change. */ 4042 INSN_VAR_LOCATION_LOC (dbg) = gen_rtx_UNKNOWN_VAR_LOC (); 4043 df_insn_rescan (dbg); 4044 4045 /* Unknown location doesn't use any registers. */ 4046 for (use = INSN_REG_USE_LIST (dbg); use != NULL; use = next) 4047 { 4048 struct reg_use_data *prev = use; 4049 4050 /* Remove use from the cyclic next_regno_use chain first. */ 4051 while (prev->next_regno_use != use) 4052 prev = prev->next_regno_use; 4053 prev->next_regno_use = use->next_regno_use; 4054 next = use->next_insn_use; 4055 free (use); 4056 } 4057 INSN_REG_USE_LIST (dbg) = NULL; 4058 4059 /* We delete rather than resolve these deps, otherwise we 4060 crash in sched_free_deps(), because forward deps are 4061 expected to be released before backward deps. */ 4062 sd_delete_dep (sd_it); 4063 } 4064 4065 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE); 4066 QUEUE_INDEX (insn) = QUEUE_SCHEDULED; 4067 4068 if (sched_pressure == SCHED_PRESSURE_MODEL 4069 && model_curr_point < model_num_insns 4070 && NONDEBUG_INSN_P (insn)) 4071 { 4072 if (model_index (insn) == model_curr_point) 4073 do 4074 model_curr_point++; 4075 while (model_curr_point < model_num_insns 4076 && (QUEUE_INDEX (MODEL_INSN (model_curr_point)) 4077 == QUEUE_SCHEDULED)); 4078 else 4079 model_recompute (insn); 4080 model_update_limit_points (); 4081 update_register_pressure (insn); 4082 if (sched_verbose >= 2) 4083 print_curr_reg_pressure (); 4084 } 4085 4086 gcc_assert (INSN_TICK (insn) >= MIN_TICK); 4087 if (INSN_TICK (insn) > clock_var) 4088 /* INSN has been prematurely moved from the queue to the ready list. 4089 This is possible only if following flags are set. */ 4090 gcc_assert (flag_sched_stalled_insns || sched_fusion); 4091 4092 /* ??? Probably, if INSN is scheduled prematurely, we should leave 4093 INSN_TICK untouched. This is a machine-dependent issue, actually. */ 4094 INSN_TICK (insn) = clock_var; 4095 4096 check_clobbered_conditions (insn); 4097 4098 /* Update dependent instructions. First, see if by scheduling this insn 4099 now we broke a dependence in a way that requires us to change another 4100 insn. */ 4101 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 4102 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it)) 4103 { 4104 struct dep_replacement *desc = DEP_REPLACE (dep); 4105 rtx_insn *pro = DEP_PRO (dep); 4106 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED 4107 && desc != NULL && desc->insn == pro) 4108 apply_replacement (dep, false); 4109 } 4110 4111 /* Go through and resolve forward dependencies. */ 4112 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW); 4113 sd_iterator_cond (&sd_it, &dep);) 4114 { 4115 rtx_insn *next = DEP_CON (dep); 4116 bool cancelled = (DEP_STATUS (dep) & DEP_CANCELLED) != 0; 4117 4118 /* Resolve the dependence between INSN and NEXT. 4119 sd_resolve_dep () moves current dep to another list thus 4120 advancing the iterator. */ 4121 sd_resolve_dep (sd_it); 4122 4123 if (cancelled) 4124 { 4125 if (must_restore_pattern_p (next, dep)) 4126 restore_pattern (dep, false); 4127 continue; 4128 } 4129 4130 /* Don't bother trying to mark next as ready if insn is a debug 4131 insn. If insn is the last hard dependency, it will have 4132 already been discounted. */ 4133 if (DEBUG_INSN_P (insn) && !DEBUG_INSN_P (next)) 4134 continue; 4135 4136 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn)) 4137 { 4138 int effective_cost; 4139 4140 effective_cost = try_ready (next); 4141 4142 if (effective_cost >= 0 4143 && SCHED_GROUP_P (next) 4144 && advance < effective_cost) 4145 advance = effective_cost; 4146 } 4147 else 4148 /* Check always has only one forward dependence (to the first insn in 4149 the recovery block), therefore, this will be executed only once. */ 4150 { 4151 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW)); 4152 fix_recovery_deps (RECOVERY_BLOCK (insn)); 4153 } 4154 } 4155 4156 /* Annotate the instruction with issue information -- TImode 4157 indicates that the instruction is expected not to be able 4158 to issue on the same cycle as the previous insn. A machine 4159 may use this information to decide how the instruction should 4160 be aligned. */ 4161 if (issue_rate > 1 4162 && GET_CODE (PATTERN (insn)) != USE 4163 && GET_CODE (PATTERN (insn)) != CLOBBER 4164 && !DEBUG_INSN_P (insn)) 4165 { 4166 if (reload_completed) 4167 PUT_MODE (insn, clock_var > last_clock_var ? TImode : VOIDmode); 4168 last_clock_var = clock_var; 4169 } 4170 4171 if (nonscheduled_insns_begin != NULL_RTX) 4172 /* Indicate to debug counters that INSN is scheduled. */ 4173 nonscheduled_insns_begin = insn; 4174 4175 return advance; 4176 } 4177 4178 /* Functions for handling of notes. */ 4179 4180 /* Add note list that ends on FROM_END to the end of TO_ENDP. */ 4181 void 4182 concat_note_lists (rtx_insn *from_end, rtx_insn **to_endp) 4183 { 4184 rtx_insn *from_start; 4185 4186 /* It's easy when have nothing to concat. */ 4187 if (from_end == NULL) 4188 return; 4189 4190 /* It's also easy when destination is empty. */ 4191 if (*to_endp == NULL) 4192 { 4193 *to_endp = from_end; 4194 return; 4195 } 4196 4197 from_start = from_end; 4198 while (PREV_INSN (from_start) != NULL) 4199 from_start = PREV_INSN (from_start); 4200 4201 SET_PREV_INSN (from_start) = *to_endp; 4202 SET_NEXT_INSN (*to_endp) = from_start; 4203 *to_endp = from_end; 4204 } 4205 4206 /* Delete notes between HEAD and TAIL and put them in the chain 4207 of notes ended by NOTE_LIST. */ 4208 void 4209 remove_notes (rtx_insn *head, rtx_insn *tail) 4210 { 4211 rtx_insn *next_tail, *insn, *next; 4212 4213 note_list = 0; 4214 if (head == tail && !INSN_P (head)) 4215 return; 4216 4217 next_tail = NEXT_INSN (tail); 4218 for (insn = head; insn != next_tail; insn = next) 4219 { 4220 next = NEXT_INSN (insn); 4221 if (!NOTE_P (insn)) 4222 continue; 4223 4224 switch (NOTE_KIND (insn)) 4225 { 4226 case NOTE_INSN_BASIC_BLOCK: 4227 continue; 4228 4229 case NOTE_INSN_EPILOGUE_BEG: 4230 if (insn != tail) 4231 { 4232 remove_insn (insn); 4233 add_reg_note (next, REG_SAVE_NOTE, 4234 GEN_INT (NOTE_INSN_EPILOGUE_BEG)); 4235 break; 4236 } 4237 /* FALLTHRU */ 4238 4239 default: 4240 remove_insn (insn); 4241 4242 /* Add the note to list that ends at NOTE_LIST. */ 4243 SET_PREV_INSN (insn) = note_list; 4244 SET_NEXT_INSN (insn) = NULL_RTX; 4245 if (note_list) 4246 SET_NEXT_INSN (note_list) = insn; 4247 note_list = insn; 4248 break; 4249 } 4250 4251 gcc_assert ((sel_sched_p () || insn != tail) && insn != head); 4252 } 4253 } 4254 4255 /* A structure to record enough data to allow us to backtrack the scheduler to 4256 a previous state. */ 4257 struct haifa_saved_data 4258 { 4259 /* Next entry on the list. */ 4260 struct haifa_saved_data *next; 4261 4262 /* Backtracking is associated with scheduling insns that have delay slots. 4263 DELAY_PAIR points to the structure that contains the insns involved, and 4264 the number of cycles between them. */ 4265 struct delay_pair *delay_pair; 4266 4267 /* Data used by the frontend (e.g. sched-ebb or sched-rgn). */ 4268 void *fe_saved_data; 4269 /* Data used by the backend. */ 4270 void *be_saved_data; 4271 4272 /* Copies of global state. */ 4273 int clock_var, last_clock_var; 4274 struct ready_list ready; 4275 state_t curr_state; 4276 4277 rtx_insn *last_scheduled_insn; 4278 rtx_insn *last_nondebug_scheduled_insn; 4279 rtx_insn *nonscheduled_insns_begin; 4280 int cycle_issued_insns; 4281 4282 /* Copies of state used in the inner loop of schedule_block. */ 4283 struct sched_block_state sched_block; 4284 4285 /* We don't need to save q_ptr, as its value is arbitrary and we can set it 4286 to 0 when restoring. */ 4287 int q_size; 4288 rtx_insn_list **insn_queue; 4289 4290 /* Describe pattern replacements that occurred since this backtrack point 4291 was queued. */ 4292 vec<dep_t> replacement_deps; 4293 vec<int> replace_apply; 4294 4295 /* A copy of the next-cycle replacement vectors at the time of the backtrack 4296 point. */ 4297 vec<dep_t> next_cycle_deps; 4298 vec<int> next_cycle_apply; 4299 }; 4300 4301 /* A record, in reverse order, of all scheduled insns which have delay slots 4302 and may require backtracking. */ 4303 static struct haifa_saved_data *backtrack_queue; 4304 4305 /* For every dependency of INSN, set the FEEDS_BACKTRACK_INSN bit according 4306 to SET_P. */ 4307 static void 4308 mark_backtrack_feeds (rtx_insn *insn, int set_p) 4309 { 4310 sd_iterator_def sd_it; 4311 dep_t dep; 4312 FOR_EACH_DEP (insn, SD_LIST_HARD_BACK, sd_it, dep) 4313 { 4314 FEEDS_BACKTRACK_INSN (DEP_PRO (dep)) = set_p; 4315 } 4316 } 4317 4318 /* Save the current scheduler state so that we can backtrack to it 4319 later if necessary. PAIR gives the insns that make it necessary to 4320 save this point. SCHED_BLOCK is the local state of schedule_block 4321 that need to be saved. */ 4322 static void 4323 save_backtrack_point (struct delay_pair *pair, 4324 struct sched_block_state sched_block) 4325 { 4326 int i; 4327 struct haifa_saved_data *save = XNEW (struct haifa_saved_data); 4328 4329 save->curr_state = xmalloc (dfa_state_size); 4330 memcpy (save->curr_state, curr_state, dfa_state_size); 4331 4332 save->ready.first = ready.first; 4333 save->ready.n_ready = ready.n_ready; 4334 save->ready.n_debug = ready.n_debug; 4335 save->ready.veclen = ready.veclen; 4336 save->ready.vec = XNEWVEC (rtx_insn *, ready.veclen); 4337 memcpy (save->ready.vec, ready.vec, ready.veclen * sizeof (rtx)); 4338 4339 save->insn_queue = XNEWVEC (rtx_insn_list *, max_insn_queue_index + 1); 4340 save->q_size = q_size; 4341 for (i = 0; i <= max_insn_queue_index; i++) 4342 { 4343 int q = NEXT_Q_AFTER (q_ptr, i); 4344 save->insn_queue[i] = copy_INSN_LIST (insn_queue[q]); 4345 } 4346 4347 save->clock_var = clock_var; 4348 save->last_clock_var = last_clock_var; 4349 save->cycle_issued_insns = cycle_issued_insns; 4350 save->last_scheduled_insn = last_scheduled_insn; 4351 save->last_nondebug_scheduled_insn = last_nondebug_scheduled_insn; 4352 save->nonscheduled_insns_begin = nonscheduled_insns_begin; 4353 4354 save->sched_block = sched_block; 4355 4356 save->replacement_deps.create (0); 4357 save->replace_apply.create (0); 4358 save->next_cycle_deps = next_cycle_replace_deps.copy (); 4359 save->next_cycle_apply = next_cycle_apply.copy (); 4360 4361 if (current_sched_info->save_state) 4362 save->fe_saved_data = (*current_sched_info->save_state) (); 4363 4364 if (targetm.sched.alloc_sched_context) 4365 { 4366 save->be_saved_data = targetm.sched.alloc_sched_context (); 4367 targetm.sched.init_sched_context (save->be_saved_data, false); 4368 } 4369 else 4370 save->be_saved_data = NULL; 4371 4372 save->delay_pair = pair; 4373 4374 save->next = backtrack_queue; 4375 backtrack_queue = save; 4376 4377 while (pair) 4378 { 4379 mark_backtrack_feeds (pair->i2, 1); 4380 INSN_TICK (pair->i2) = INVALID_TICK; 4381 INSN_EXACT_TICK (pair->i2) = clock_var + pair_delay (pair); 4382 SHADOW_P (pair->i2) = pair->stages == 0; 4383 pair = pair->next_same_i1; 4384 } 4385 } 4386 4387 /* Walk the ready list and all queues. If any insns have unresolved backwards 4388 dependencies, these must be cancelled deps, broken by predication. Set or 4389 clear (depending on SET) the DEP_CANCELLED bit in DEP_STATUS. */ 4390 4391 static void 4392 toggle_cancelled_flags (bool set) 4393 { 4394 int i; 4395 sd_iterator_def sd_it; 4396 dep_t dep; 4397 4398 if (ready.n_ready > 0) 4399 { 4400 rtx_insn **first = ready_lastpos (&ready); 4401 for (i = 0; i < ready.n_ready; i++) 4402 FOR_EACH_DEP (first[i], SD_LIST_BACK, sd_it, dep) 4403 if (!DEBUG_INSN_P (DEP_PRO (dep))) 4404 { 4405 if (set) 4406 DEP_STATUS (dep) |= DEP_CANCELLED; 4407 else 4408 DEP_STATUS (dep) &= ~DEP_CANCELLED; 4409 } 4410 } 4411 for (i = 0; i <= max_insn_queue_index; i++) 4412 { 4413 int q = NEXT_Q_AFTER (q_ptr, i); 4414 rtx_insn_list *link; 4415 for (link = insn_queue[q]; link; link = link->next ()) 4416 { 4417 rtx_insn *insn = link->insn (); 4418 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep) 4419 if (!DEBUG_INSN_P (DEP_PRO (dep))) 4420 { 4421 if (set) 4422 DEP_STATUS (dep) |= DEP_CANCELLED; 4423 else 4424 DEP_STATUS (dep) &= ~DEP_CANCELLED; 4425 } 4426 } 4427 } 4428 } 4429 4430 /* Undo the replacements that have occurred after backtrack point SAVE 4431 was placed. */ 4432 static void 4433 undo_replacements_for_backtrack (struct haifa_saved_data *save) 4434 { 4435 while (!save->replacement_deps.is_empty ()) 4436 { 4437 dep_t dep = save->replacement_deps.pop (); 4438 int apply_p = save->replace_apply.pop (); 4439 4440 if (apply_p) 4441 restore_pattern (dep, true); 4442 else 4443 apply_replacement (dep, true); 4444 } 4445 save->replacement_deps.release (); 4446 save->replace_apply.release (); 4447 } 4448 4449 /* Pop entries from the SCHEDULED_INSNS vector up to and including INSN. 4450 Restore their dependencies to an unresolved state, and mark them as 4451 queued nowhere. */ 4452 4453 static void 4454 unschedule_insns_until (rtx_insn *insn) 4455 { 4456 auto_vec<rtx_insn *> recompute_vec; 4457 4458 /* Make two passes over the insns to be unscheduled. First, we clear out 4459 dependencies and other trivial bookkeeping. */ 4460 for (;;) 4461 { 4462 rtx_insn *last; 4463 sd_iterator_def sd_it; 4464 dep_t dep; 4465 4466 last = scheduled_insns.pop (); 4467 4468 /* This will be changed by restore_backtrack_point if the insn is in 4469 any queue. */ 4470 QUEUE_INDEX (last) = QUEUE_NOWHERE; 4471 if (last != insn) 4472 INSN_TICK (last) = INVALID_TICK; 4473 4474 if (modulo_ii > 0 && INSN_UID (last) < modulo_iter0_max_uid) 4475 modulo_insns_scheduled--; 4476 4477 for (sd_it = sd_iterator_start (last, SD_LIST_RES_FORW); 4478 sd_iterator_cond (&sd_it, &dep);) 4479 { 4480 rtx_insn *con = DEP_CON (dep); 4481 sd_unresolve_dep (sd_it); 4482 if (!MUST_RECOMPUTE_SPEC_P (con)) 4483 { 4484 MUST_RECOMPUTE_SPEC_P (con) = 1; 4485 recompute_vec.safe_push (con); 4486 } 4487 } 4488 4489 if (last == insn) 4490 break; 4491 } 4492 4493 /* A second pass, to update ready and speculation status for insns 4494 depending on the unscheduled ones. The first pass must have 4495 popped the scheduled_insns vector up to the point where we 4496 restart scheduling, as recompute_todo_spec requires it to be 4497 up-to-date. */ 4498 while (!recompute_vec.is_empty ()) 4499 { 4500 rtx_insn *con; 4501 4502 con = recompute_vec.pop (); 4503 MUST_RECOMPUTE_SPEC_P (con) = 0; 4504 if (!sd_lists_empty_p (con, SD_LIST_HARD_BACK)) 4505 { 4506 TODO_SPEC (con) = HARD_DEP; 4507 INSN_TICK (con) = INVALID_TICK; 4508 if (PREDICATED_PAT (con) != NULL_RTX) 4509 haifa_change_pattern (con, ORIG_PAT (con)); 4510 } 4511 else if (QUEUE_INDEX (con) != QUEUE_SCHEDULED) 4512 TODO_SPEC (con) = recompute_todo_spec (con, true); 4513 } 4514 } 4515 4516 /* Restore scheduler state from the topmost entry on the backtracking queue. 4517 PSCHED_BLOCK_P points to the local data of schedule_block that we must 4518 overwrite with the saved data. 4519 The caller must already have called unschedule_insns_until. */ 4520 4521 static void 4522 restore_last_backtrack_point (struct sched_block_state *psched_block) 4523 { 4524 int i; 4525 struct haifa_saved_data *save = backtrack_queue; 4526 4527 backtrack_queue = save->next; 4528 4529 if (current_sched_info->restore_state) 4530 (*current_sched_info->restore_state) (save->fe_saved_data); 4531 4532 if (targetm.sched.alloc_sched_context) 4533 { 4534 targetm.sched.set_sched_context (save->be_saved_data); 4535 targetm.sched.free_sched_context (save->be_saved_data); 4536 } 4537 4538 /* Do this first since it clobbers INSN_TICK of the involved 4539 instructions. */ 4540 undo_replacements_for_backtrack (save); 4541 4542 /* Clear the QUEUE_INDEX of everything in the ready list or one 4543 of the queues. */ 4544 if (ready.n_ready > 0) 4545 { 4546 rtx_insn **first = ready_lastpos (&ready); 4547 for (i = 0; i < ready.n_ready; i++) 4548 { 4549 rtx_insn *insn = first[i]; 4550 QUEUE_INDEX (insn) = QUEUE_NOWHERE; 4551 INSN_TICK (insn) = INVALID_TICK; 4552 } 4553 } 4554 for (i = 0; i <= max_insn_queue_index; i++) 4555 { 4556 int q = NEXT_Q_AFTER (q_ptr, i); 4557 4558 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ()) 4559 { 4560 rtx_insn *x = link->insn (); 4561 QUEUE_INDEX (x) = QUEUE_NOWHERE; 4562 INSN_TICK (x) = INVALID_TICK; 4563 } 4564 free_INSN_LIST_list (&insn_queue[q]); 4565 } 4566 4567 free (ready.vec); 4568 ready = save->ready; 4569 4570 if (ready.n_ready > 0) 4571 { 4572 rtx_insn **first = ready_lastpos (&ready); 4573 for (i = 0; i < ready.n_ready; i++) 4574 { 4575 rtx_insn *insn = first[i]; 4576 QUEUE_INDEX (insn) = QUEUE_READY; 4577 TODO_SPEC (insn) = recompute_todo_spec (insn, true); 4578 INSN_TICK (insn) = save->clock_var; 4579 } 4580 } 4581 4582 q_ptr = 0; 4583 q_size = save->q_size; 4584 for (i = 0; i <= max_insn_queue_index; i++) 4585 { 4586 int q = NEXT_Q_AFTER (q_ptr, i); 4587 4588 insn_queue[q] = save->insn_queue[q]; 4589 4590 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ()) 4591 { 4592 rtx_insn *x = link->insn (); 4593 QUEUE_INDEX (x) = i; 4594 TODO_SPEC (x) = recompute_todo_spec (x, true); 4595 INSN_TICK (x) = save->clock_var + i; 4596 } 4597 } 4598 free (save->insn_queue); 4599 4600 toggle_cancelled_flags (true); 4601 4602 clock_var = save->clock_var; 4603 last_clock_var = save->last_clock_var; 4604 cycle_issued_insns = save->cycle_issued_insns; 4605 last_scheduled_insn = save->last_scheduled_insn; 4606 last_nondebug_scheduled_insn = save->last_nondebug_scheduled_insn; 4607 nonscheduled_insns_begin = save->nonscheduled_insns_begin; 4608 4609 *psched_block = save->sched_block; 4610 4611 memcpy (curr_state, save->curr_state, dfa_state_size); 4612 free (save->curr_state); 4613 4614 mark_backtrack_feeds (save->delay_pair->i2, 0); 4615 4616 gcc_assert (next_cycle_replace_deps.is_empty ()); 4617 next_cycle_replace_deps = save->next_cycle_deps.copy (); 4618 next_cycle_apply = save->next_cycle_apply.copy (); 4619 4620 free (save); 4621 4622 for (save = backtrack_queue; save; save = save->next) 4623 { 4624 mark_backtrack_feeds (save->delay_pair->i2, 1); 4625 } 4626 } 4627 4628 /* Discard all data associated with the topmost entry in the backtrack 4629 queue. If RESET_TICK is false, we just want to free the data. If true, 4630 we are doing this because we discovered a reason to backtrack. In the 4631 latter case, also reset the INSN_TICK for the shadow insn. */ 4632 static void 4633 free_topmost_backtrack_point (bool reset_tick) 4634 { 4635 struct haifa_saved_data *save = backtrack_queue; 4636 int i; 4637 4638 backtrack_queue = save->next; 4639 4640 if (reset_tick) 4641 { 4642 struct delay_pair *pair = save->delay_pair; 4643 while (pair) 4644 { 4645 INSN_TICK (pair->i2) = INVALID_TICK; 4646 INSN_EXACT_TICK (pair->i2) = INVALID_TICK; 4647 pair = pair->next_same_i1; 4648 } 4649 undo_replacements_for_backtrack (save); 4650 } 4651 else 4652 { 4653 save->replacement_deps.release (); 4654 save->replace_apply.release (); 4655 } 4656 4657 if (targetm.sched.free_sched_context) 4658 targetm.sched.free_sched_context (save->be_saved_data); 4659 if (current_sched_info->restore_state) 4660 free (save->fe_saved_data); 4661 for (i = 0; i <= max_insn_queue_index; i++) 4662 free_INSN_LIST_list (&save->insn_queue[i]); 4663 free (save->insn_queue); 4664 free (save->curr_state); 4665 free (save->ready.vec); 4666 free (save); 4667 } 4668 4669 /* Free the entire backtrack queue. */ 4670 static void 4671 free_backtrack_queue (void) 4672 { 4673 while (backtrack_queue) 4674 free_topmost_backtrack_point (false); 4675 } 4676 4677 /* Apply a replacement described by DESC. If IMMEDIATELY is false, we 4678 may have to postpone the replacement until the start of the next cycle, 4679 at which point we will be called again with IMMEDIATELY true. This is 4680 only done for machines which have instruction packets with explicit 4681 parallelism however. */ 4682 static void 4683 apply_replacement (dep_t dep, bool immediately) 4684 { 4685 struct dep_replacement *desc = DEP_REPLACE (dep); 4686 if (!immediately && targetm.sched.exposed_pipeline && reload_completed) 4687 { 4688 next_cycle_replace_deps.safe_push (dep); 4689 next_cycle_apply.safe_push (1); 4690 } 4691 else 4692 { 4693 bool success; 4694 4695 if (QUEUE_INDEX (desc->insn) == QUEUE_SCHEDULED) 4696 return; 4697 4698 if (sched_verbose >= 5) 4699 fprintf (sched_dump, "applying replacement for insn %d\n", 4700 INSN_UID (desc->insn)); 4701 4702 success = validate_change (desc->insn, desc->loc, desc->newval, 0); 4703 gcc_assert (success); 4704 4705 update_insn_after_change (desc->insn); 4706 if ((TODO_SPEC (desc->insn) & (HARD_DEP | DEP_POSTPONED)) == 0) 4707 fix_tick_ready (desc->insn); 4708 4709 if (backtrack_queue != NULL) 4710 { 4711 backtrack_queue->replacement_deps.safe_push (dep); 4712 backtrack_queue->replace_apply.safe_push (1); 4713 } 4714 } 4715 } 4716 4717 /* We have determined that a pattern involved in DEP must be restored. 4718 If IMMEDIATELY is false, we may have to postpone the replacement 4719 until the start of the next cycle, at which point we will be called 4720 again with IMMEDIATELY true. */ 4721 static void 4722 restore_pattern (dep_t dep, bool immediately) 4723 { 4724 rtx_insn *next = DEP_CON (dep); 4725 int tick = INSN_TICK (next); 4726 4727 /* If we already scheduled the insn, the modified version is 4728 correct. */ 4729 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED) 4730 return; 4731 4732 if (!immediately && targetm.sched.exposed_pipeline && reload_completed) 4733 { 4734 next_cycle_replace_deps.safe_push (dep); 4735 next_cycle_apply.safe_push (0); 4736 return; 4737 } 4738 4739 4740 if (DEP_TYPE (dep) == REG_DEP_CONTROL) 4741 { 4742 if (sched_verbose >= 5) 4743 fprintf (sched_dump, "restoring pattern for insn %d\n", 4744 INSN_UID (next)); 4745 haifa_change_pattern (next, ORIG_PAT (next)); 4746 } 4747 else 4748 { 4749 struct dep_replacement *desc = DEP_REPLACE (dep); 4750 bool success; 4751 4752 if (sched_verbose >= 5) 4753 fprintf (sched_dump, "restoring pattern for insn %d\n", 4754 INSN_UID (desc->insn)); 4755 tick = INSN_TICK (desc->insn); 4756 4757 success = validate_change (desc->insn, desc->loc, desc->orig, 0); 4758 gcc_assert (success); 4759 update_insn_after_change (desc->insn); 4760 if (backtrack_queue != NULL) 4761 { 4762 backtrack_queue->replacement_deps.safe_push (dep); 4763 backtrack_queue->replace_apply.safe_push (0); 4764 } 4765 } 4766 INSN_TICK (next) = tick; 4767 if (TODO_SPEC (next) == DEP_POSTPONED) 4768 return; 4769 4770 if (sd_lists_empty_p (next, SD_LIST_BACK)) 4771 TODO_SPEC (next) = 0; 4772 else if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK)) 4773 TODO_SPEC (next) = HARD_DEP; 4774 } 4775 4776 /* Perform pattern replacements that were queued up until the next 4777 cycle. */ 4778 static void 4779 perform_replacements_new_cycle (void) 4780 { 4781 int i; 4782 dep_t dep; 4783 FOR_EACH_VEC_ELT (next_cycle_replace_deps, i, dep) 4784 { 4785 int apply_p = next_cycle_apply[i]; 4786 if (apply_p) 4787 apply_replacement (dep, true); 4788 else 4789 restore_pattern (dep, true); 4790 } 4791 next_cycle_replace_deps.truncate (0); 4792 next_cycle_apply.truncate (0); 4793 } 4794 4795 /* Compute INSN_TICK_ESTIMATE for INSN. PROCESSED is a bitmap of 4796 instructions we've previously encountered, a set bit prevents 4797 recursion. BUDGET is a limit on how far ahead we look, it is 4798 reduced on recursive calls. Return true if we produced a good 4799 estimate, or false if we exceeded the budget. */ 4800 static bool 4801 estimate_insn_tick (bitmap processed, rtx_insn *insn, int budget) 4802 { 4803 sd_iterator_def sd_it; 4804 dep_t dep; 4805 int earliest = INSN_TICK (insn); 4806 4807 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep) 4808 { 4809 rtx_insn *pro = DEP_PRO (dep); 4810 int t; 4811 4812 if (DEP_STATUS (dep) & DEP_CANCELLED) 4813 continue; 4814 4815 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED) 4816 gcc_assert (INSN_TICK (pro) + dep_cost (dep) <= INSN_TICK (insn)); 4817 else 4818 { 4819 int cost = dep_cost (dep); 4820 if (cost >= budget) 4821 return false; 4822 if (!bitmap_bit_p (processed, INSN_LUID (pro))) 4823 { 4824 if (!estimate_insn_tick (processed, pro, budget - cost)) 4825 return false; 4826 } 4827 gcc_assert (INSN_TICK_ESTIMATE (pro) != INVALID_TICK); 4828 t = INSN_TICK_ESTIMATE (pro) + cost; 4829 if (earliest == INVALID_TICK || t > earliest) 4830 earliest = t; 4831 } 4832 } 4833 bitmap_set_bit (processed, INSN_LUID (insn)); 4834 INSN_TICK_ESTIMATE (insn) = earliest; 4835 return true; 4836 } 4837 4838 /* Examine the pair of insns in P, and estimate (optimistically, assuming 4839 infinite resources) the cycle in which the delayed shadow can be issued. 4840 Return the number of cycles that must pass before the real insn can be 4841 issued in order to meet this constraint. */ 4842 static int 4843 estimate_shadow_tick (struct delay_pair *p) 4844 { 4845 auto_bitmap processed; 4846 int t; 4847 bool cutoff; 4848 4849 cutoff = !estimate_insn_tick (processed, p->i2, 4850 max_insn_queue_index + pair_delay (p)); 4851 if (cutoff) 4852 return max_insn_queue_index; 4853 t = INSN_TICK_ESTIMATE (p->i2) - (clock_var + pair_delay (p) + 1); 4854 if (t > 0) 4855 return t; 4856 return 0; 4857 } 4858 4859 /* If INSN has no unresolved backwards dependencies, add it to the schedule and 4860 recursively resolve all its forward dependencies. */ 4861 static void 4862 resolve_dependencies (rtx_insn *insn) 4863 { 4864 sd_iterator_def sd_it; 4865 dep_t dep; 4866 4867 /* Don't use sd_lists_empty_p; it ignores debug insns. */ 4868 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (insn)) != NULL 4869 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (insn)) != NULL) 4870 return; 4871 4872 if (sched_verbose >= 4) 4873 fprintf (sched_dump, ";;\tquickly resolving %d\n", INSN_UID (insn)); 4874 4875 if (QUEUE_INDEX (insn) >= 0) 4876 queue_remove (insn); 4877 4878 scheduled_insns.safe_push (insn); 4879 4880 /* Update dependent instructions. */ 4881 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW); 4882 sd_iterator_cond (&sd_it, &dep);) 4883 { 4884 rtx_insn *next = DEP_CON (dep); 4885 4886 if (sched_verbose >= 4) 4887 fprintf (sched_dump, ";;\t\tdep %d against %d\n", INSN_UID (insn), 4888 INSN_UID (next)); 4889 4890 /* Resolve the dependence between INSN and NEXT. 4891 sd_resolve_dep () moves current dep to another list thus 4892 advancing the iterator. */ 4893 sd_resolve_dep (sd_it); 4894 4895 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn)) 4896 { 4897 resolve_dependencies (next); 4898 } 4899 else 4900 /* Check always has only one forward dependence (to the first insn in 4901 the recovery block), therefore, this will be executed only once. */ 4902 { 4903 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW)); 4904 } 4905 } 4906 } 4907 4908 4909 /* Return the head and tail pointers of ebb starting at BEG and ending 4910 at END. */ 4911 void 4912 get_ebb_head_tail (basic_block beg, basic_block end, 4913 rtx_insn **headp, rtx_insn **tailp) 4914 { 4915 rtx_insn *beg_head = BB_HEAD (beg); 4916 rtx_insn * beg_tail = BB_END (beg); 4917 rtx_insn * end_head = BB_HEAD (end); 4918 rtx_insn * end_tail = BB_END (end); 4919 4920 /* Don't include any notes or labels at the beginning of the BEG 4921 basic block, or notes at the end of the END basic blocks. */ 4922 4923 if (LABEL_P (beg_head)) 4924 beg_head = NEXT_INSN (beg_head); 4925 4926 while (beg_head != beg_tail) 4927 if (NOTE_P (beg_head)) 4928 beg_head = NEXT_INSN (beg_head); 4929 else if (DEBUG_INSN_P (beg_head)) 4930 { 4931 rtx_insn * note, *next; 4932 4933 for (note = NEXT_INSN (beg_head); 4934 note != beg_tail; 4935 note = next) 4936 { 4937 next = NEXT_INSN (note); 4938 if (NOTE_P (note)) 4939 { 4940 if (sched_verbose >= 9) 4941 fprintf (sched_dump, "reorder %i\n", INSN_UID (note)); 4942 4943 reorder_insns_nobb (note, note, PREV_INSN (beg_head)); 4944 4945 if (BLOCK_FOR_INSN (note) != beg) 4946 df_insn_change_bb (note, beg); 4947 } 4948 else if (!DEBUG_INSN_P (note)) 4949 break; 4950 } 4951 4952 break; 4953 } 4954 else 4955 break; 4956 4957 *headp = beg_head; 4958 4959 if (beg == end) 4960 end_head = beg_head; 4961 else if (LABEL_P (end_head)) 4962 end_head = NEXT_INSN (end_head); 4963 4964 while (end_head != end_tail) 4965 if (NOTE_P (end_tail)) 4966 end_tail = PREV_INSN (end_tail); 4967 else if (DEBUG_INSN_P (end_tail)) 4968 { 4969 rtx_insn * note, *prev; 4970 4971 for (note = PREV_INSN (end_tail); 4972 note != end_head; 4973 note = prev) 4974 { 4975 prev = PREV_INSN (note); 4976 if (NOTE_P (note)) 4977 { 4978 if (sched_verbose >= 9) 4979 fprintf (sched_dump, "reorder %i\n", INSN_UID (note)); 4980 4981 reorder_insns_nobb (note, note, end_tail); 4982 4983 if (end_tail == BB_END (end)) 4984 BB_END (end) = note; 4985 4986 if (BLOCK_FOR_INSN (note) != end) 4987 df_insn_change_bb (note, end); 4988 } 4989 else if (!DEBUG_INSN_P (note)) 4990 break; 4991 } 4992 4993 break; 4994 } 4995 else 4996 break; 4997 4998 *tailp = end_tail; 4999 } 5000 5001 /* Return nonzero if there are no real insns in the range [ HEAD, TAIL ]. */ 5002 5003 int 5004 no_real_insns_p (const rtx_insn *head, const rtx_insn *tail) 5005 { 5006 while (head != NEXT_INSN (tail)) 5007 { 5008 if (!NOTE_P (head) && !LABEL_P (head)) 5009 return 0; 5010 head = NEXT_INSN (head); 5011 } 5012 return 1; 5013 } 5014 5015 /* Restore-other-notes: NOTE_LIST is the end of a chain of notes 5016 previously found among the insns. Insert them just before HEAD. */ 5017 rtx_insn * 5018 restore_other_notes (rtx_insn *head, basic_block head_bb) 5019 { 5020 if (note_list != 0) 5021 { 5022 rtx_insn *note_head = note_list; 5023 5024 if (head) 5025 head_bb = BLOCK_FOR_INSN (head); 5026 else 5027 head = NEXT_INSN (bb_note (head_bb)); 5028 5029 while (PREV_INSN (note_head)) 5030 { 5031 set_block_for_insn (note_head, head_bb); 5032 note_head = PREV_INSN (note_head); 5033 } 5034 /* In the above cycle we've missed this note. */ 5035 set_block_for_insn (note_head, head_bb); 5036 5037 SET_PREV_INSN (note_head) = PREV_INSN (head); 5038 SET_NEXT_INSN (PREV_INSN (head)) = note_head; 5039 SET_PREV_INSN (head) = note_list; 5040 SET_NEXT_INSN (note_list) = head; 5041 5042 if (BLOCK_FOR_INSN (head) != head_bb) 5043 BB_END (head_bb) = note_list; 5044 5045 head = note_head; 5046 } 5047 5048 return head; 5049 } 5050 5051 /* When we know we are going to discard the schedule due to a failed attempt 5052 at modulo scheduling, undo all replacements. */ 5053 static void 5054 undo_all_replacements (void) 5055 { 5056 rtx_insn *insn; 5057 int i; 5058 5059 FOR_EACH_VEC_ELT (scheduled_insns, i, insn) 5060 { 5061 sd_iterator_def sd_it; 5062 dep_t dep; 5063 5064 /* See if we must undo a replacement. */ 5065 for (sd_it = sd_iterator_start (insn, SD_LIST_RES_FORW); 5066 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it)) 5067 { 5068 struct dep_replacement *desc = DEP_REPLACE (dep); 5069 if (desc != NULL) 5070 validate_change (desc->insn, desc->loc, desc->orig, 0); 5071 } 5072 } 5073 } 5074 5075 /* Return first non-scheduled insn in the current scheduling block. 5076 This is mostly used for debug-counter purposes. */ 5077 static rtx_insn * 5078 first_nonscheduled_insn (void) 5079 { 5080 rtx_insn *insn = (nonscheduled_insns_begin != NULL_RTX 5081 ? nonscheduled_insns_begin 5082 : current_sched_info->prev_head); 5083 5084 do 5085 { 5086 insn = next_nonnote_nondebug_insn (insn); 5087 } 5088 while (QUEUE_INDEX (insn) == QUEUE_SCHEDULED); 5089 5090 return insn; 5091 } 5092 5093 /* Move insns that became ready to fire from queue to ready list. */ 5094 5095 static void 5096 queue_to_ready (struct ready_list *ready) 5097 { 5098 rtx_insn *insn; 5099 rtx_insn_list *link; 5100 rtx_insn *skip_insn; 5101 5102 q_ptr = NEXT_Q (q_ptr); 5103 5104 if (dbg_cnt (sched_insn) == false) 5105 /* If debug counter is activated do not requeue the first 5106 nonscheduled insn. */ 5107 skip_insn = first_nonscheduled_insn (); 5108 else 5109 skip_insn = NULL; 5110 5111 /* Add all pending insns that can be scheduled without stalls to the 5112 ready list. */ 5113 for (link = insn_queue[q_ptr]; link; link = link->next ()) 5114 { 5115 insn = link->insn (); 5116 q_size -= 1; 5117 5118 if (sched_verbose >= 2) 5119 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ", 5120 (*current_sched_info->print_insn) (insn, 0)); 5121 5122 /* If the ready list is full, delay the insn for 1 cycle. 5123 See the comment in schedule_block for the rationale. */ 5124 if (!reload_completed 5125 && (ready->n_ready - ready->n_debug > MAX_SCHED_READY_INSNS 5126 || (sched_pressure == SCHED_PRESSURE_MODEL 5127 /* Limit pressure recalculations to MAX_SCHED_READY_INSNS 5128 instructions too. */ 5129 && model_index (insn) > (model_curr_point 5130 + MAX_SCHED_READY_INSNS))) 5131 && !(sched_pressure == SCHED_PRESSURE_MODEL 5132 && model_curr_point < model_num_insns 5133 /* Always allow the next model instruction to issue. */ 5134 && model_index (insn) == model_curr_point) 5135 && !SCHED_GROUP_P (insn) 5136 && insn != skip_insn) 5137 { 5138 if (sched_verbose >= 2) 5139 fprintf (sched_dump, "keeping in queue, ready full\n"); 5140 queue_insn (insn, 1, "ready full"); 5141 } 5142 else 5143 { 5144 ready_add (ready, insn, false); 5145 if (sched_verbose >= 2) 5146 fprintf (sched_dump, "moving to ready without stalls\n"); 5147 } 5148 } 5149 free_INSN_LIST_list (&insn_queue[q_ptr]); 5150 5151 /* If there are no ready insns, stall until one is ready and add all 5152 of the pending insns at that point to the ready list. */ 5153 if (ready->n_ready == 0) 5154 { 5155 int stalls; 5156 5157 for (stalls = 1; stalls <= max_insn_queue_index; stalls++) 5158 { 5159 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)])) 5160 { 5161 for (; link; link = link->next ()) 5162 { 5163 insn = link->insn (); 5164 q_size -= 1; 5165 5166 if (sched_verbose >= 2) 5167 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ", 5168 (*current_sched_info->print_insn) (insn, 0)); 5169 5170 ready_add (ready, insn, false); 5171 if (sched_verbose >= 2) 5172 fprintf (sched_dump, "moving to ready with %d stalls\n", stalls); 5173 } 5174 free_INSN_LIST_list (&insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]); 5175 5176 advance_one_cycle (); 5177 5178 break; 5179 } 5180 5181 advance_one_cycle (); 5182 } 5183 5184 q_ptr = NEXT_Q_AFTER (q_ptr, stalls); 5185 clock_var += stalls; 5186 if (sched_verbose >= 2) 5187 fprintf (sched_dump, ";;\tAdvancing clock by %d cycle[s] to %d\n", 5188 stalls, clock_var); 5189 } 5190 } 5191 5192 /* Used by early_queue_to_ready. Determines whether it is "ok" to 5193 prematurely move INSN from the queue to the ready list. Currently, 5194 if a target defines the hook 'is_costly_dependence', this function 5195 uses the hook to check whether there exist any dependences which are 5196 considered costly by the target, between INSN and other insns that 5197 have already been scheduled. Dependences are checked up to Y cycles 5198 back, with default Y=1; The flag -fsched-stalled-insns-dep=Y allows 5199 controlling this value. 5200 (Other considerations could be taken into account instead (or in 5201 addition) depending on user flags and target hooks. */ 5202 5203 static bool 5204 ok_for_early_queue_removal (rtx_insn *insn) 5205 { 5206 if (targetm.sched.is_costly_dependence) 5207 { 5208 int n_cycles; 5209 int i = scheduled_insns.length (); 5210 for (n_cycles = flag_sched_stalled_insns_dep; n_cycles; n_cycles--) 5211 { 5212 while (i-- > 0) 5213 { 5214 int cost; 5215 5216 rtx_insn *prev_insn = scheduled_insns[i]; 5217 5218 if (!NOTE_P (prev_insn)) 5219 { 5220 dep_t dep; 5221 5222 dep = sd_find_dep_between (prev_insn, insn, true); 5223 5224 if (dep != NULL) 5225 { 5226 cost = dep_cost (dep); 5227 5228 if (targetm.sched.is_costly_dependence (dep, cost, 5229 flag_sched_stalled_insns_dep - n_cycles)) 5230 return false; 5231 } 5232 } 5233 5234 if (GET_MODE (prev_insn) == TImode) /* end of dispatch group */ 5235 break; 5236 } 5237 5238 if (i == 0) 5239 break; 5240 } 5241 } 5242 5243 return true; 5244 } 5245 5246 5247 /* Remove insns from the queue, before they become "ready" with respect 5248 to FU latency considerations. */ 5249 5250 static int 5251 early_queue_to_ready (state_t state, struct ready_list *ready) 5252 { 5253 rtx_insn *insn; 5254 rtx_insn_list *link; 5255 rtx_insn_list *next_link; 5256 rtx_insn_list *prev_link; 5257 bool move_to_ready; 5258 int cost; 5259 state_t temp_state = alloca (dfa_state_size); 5260 int stalls; 5261 int insns_removed = 0; 5262 5263 /* 5264 Flag '-fsched-stalled-insns=X' determines the aggressiveness of this 5265 function: 5266 5267 X == 0: There is no limit on how many queued insns can be removed 5268 prematurely. (flag_sched_stalled_insns = -1). 5269 5270 X >= 1: Only X queued insns can be removed prematurely in each 5271 invocation. (flag_sched_stalled_insns = X). 5272 5273 Otherwise: Early queue removal is disabled. 5274 (flag_sched_stalled_insns = 0) 5275 */ 5276 5277 if (! flag_sched_stalled_insns) 5278 return 0; 5279 5280 for (stalls = 0; stalls <= max_insn_queue_index; stalls++) 5281 { 5282 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)])) 5283 { 5284 if (sched_verbose > 6) 5285 fprintf (sched_dump, ";; look at index %d + %d\n", q_ptr, stalls); 5286 5287 prev_link = 0; 5288 while (link) 5289 { 5290 next_link = link->next (); 5291 insn = link->insn (); 5292 if (insn && sched_verbose > 6) 5293 print_rtl_single (sched_dump, insn); 5294 5295 memcpy (temp_state, state, dfa_state_size); 5296 if (recog_memoized (insn) < 0) 5297 /* non-negative to indicate that it's not ready 5298 to avoid infinite Q->R->Q->R... */ 5299 cost = 0; 5300 else 5301 cost = state_transition (temp_state, insn); 5302 5303 if (sched_verbose >= 6) 5304 fprintf (sched_dump, "transition cost = %d\n", cost); 5305 5306 move_to_ready = false; 5307 if (cost < 0) 5308 { 5309 move_to_ready = ok_for_early_queue_removal (insn); 5310 if (move_to_ready == true) 5311 { 5312 /* move from Q to R */ 5313 q_size -= 1; 5314 ready_add (ready, insn, false); 5315 5316 if (prev_link) 5317 XEXP (prev_link, 1) = next_link; 5318 else 5319 insn_queue[NEXT_Q_AFTER (q_ptr, stalls)] = next_link; 5320 5321 free_INSN_LIST_node (link); 5322 5323 if (sched_verbose >= 2) 5324 fprintf (sched_dump, ";;\t\tEarly Q-->Ready: insn %s\n", 5325 (*current_sched_info->print_insn) (insn, 0)); 5326 5327 insns_removed++; 5328 if (insns_removed == flag_sched_stalled_insns) 5329 /* Remove no more than flag_sched_stalled_insns insns 5330 from Q at a time. */ 5331 return insns_removed; 5332 } 5333 } 5334 5335 if (move_to_ready == false) 5336 prev_link = link; 5337 5338 link = next_link; 5339 } /* while link */ 5340 } /* if link */ 5341 5342 } /* for stalls.. */ 5343 5344 return insns_removed; 5345 } 5346 5347 5348 /* Print the ready list for debugging purposes. 5349 If READY_TRY is non-zero then only print insns that max_issue 5350 will consider. */ 5351 static void 5352 debug_ready_list_1 (struct ready_list *ready, signed char *ready_try) 5353 { 5354 rtx_insn **p; 5355 int i; 5356 5357 if (ready->n_ready == 0) 5358 { 5359 fprintf (sched_dump, "\n"); 5360 return; 5361 } 5362 5363 p = ready_lastpos (ready); 5364 for (i = 0; i < ready->n_ready; i++) 5365 { 5366 if (ready_try != NULL && ready_try[ready->n_ready - i - 1]) 5367 continue; 5368 5369 fprintf (sched_dump, " %s:%d", 5370 (*current_sched_info->print_insn) (p[i], 0), 5371 INSN_LUID (p[i])); 5372 if (sched_pressure != SCHED_PRESSURE_NONE) 5373 fprintf (sched_dump, "(cost=%d", 5374 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (p[i])); 5375 fprintf (sched_dump, ":prio=%d", INSN_PRIORITY (p[i])); 5376 if (INSN_TICK (p[i]) > clock_var) 5377 fprintf (sched_dump, ":delay=%d", INSN_TICK (p[i]) - clock_var); 5378 if (sched_pressure == SCHED_PRESSURE_MODEL) 5379 fprintf (sched_dump, ":idx=%d", 5380 model_index (p[i])); 5381 if (sched_pressure != SCHED_PRESSURE_NONE) 5382 fprintf (sched_dump, ")"); 5383 } 5384 fprintf (sched_dump, "\n"); 5385 } 5386 5387 /* Print the ready list. Callable from debugger. */ 5388 static void 5389 debug_ready_list (struct ready_list *ready) 5390 { 5391 debug_ready_list_1 (ready, NULL); 5392 } 5393 5394 /* Search INSN for REG_SAVE_NOTE notes and convert them back into insn 5395 NOTEs. This is used for NOTE_INSN_EPILOGUE_BEG, so that sched-ebb 5396 replaces the epilogue note in the correct basic block. */ 5397 void 5398 reemit_notes (rtx_insn *insn) 5399 { 5400 rtx note; 5401 rtx_insn *last = insn; 5402 5403 for (note = REG_NOTES (insn); note; note = XEXP (note, 1)) 5404 { 5405 if (REG_NOTE_KIND (note) == REG_SAVE_NOTE) 5406 { 5407 enum insn_note note_type = (enum insn_note) INTVAL (XEXP (note, 0)); 5408 5409 last = emit_note_before (note_type, last); 5410 remove_note (insn, note); 5411 } 5412 } 5413 } 5414 5415 /* Move INSN. Reemit notes if needed. Update CFG, if needed. */ 5416 static void 5417 move_insn (rtx_insn *insn, rtx_insn *last, rtx nt) 5418 { 5419 if (PREV_INSN (insn) != last) 5420 { 5421 basic_block bb; 5422 rtx_insn *note; 5423 int jump_p = 0; 5424 5425 bb = BLOCK_FOR_INSN (insn); 5426 5427 /* BB_HEAD is either LABEL or NOTE. */ 5428 gcc_assert (BB_HEAD (bb) != insn); 5429 5430 if (BB_END (bb) == insn) 5431 /* If this is last instruction in BB, move end marker one 5432 instruction up. */ 5433 { 5434 /* Jumps are always placed at the end of basic block. */ 5435 jump_p = control_flow_insn_p (insn); 5436 5437 gcc_assert (!jump_p 5438 || ((common_sched_info->sched_pass_id == SCHED_RGN_PASS) 5439 && IS_SPECULATION_BRANCHY_CHECK_P (insn)) 5440 || (common_sched_info->sched_pass_id 5441 == SCHED_EBB_PASS)); 5442 5443 gcc_assert (BLOCK_FOR_INSN (PREV_INSN (insn)) == bb); 5444 5445 BB_END (bb) = PREV_INSN (insn); 5446 } 5447 5448 gcc_assert (BB_END (bb) != last); 5449 5450 if (jump_p) 5451 /* We move the block note along with jump. */ 5452 { 5453 gcc_assert (nt); 5454 5455 note = NEXT_INSN (insn); 5456 while (NOTE_NOT_BB_P (note) && note != nt) 5457 note = NEXT_INSN (note); 5458 5459 if (note != nt 5460 && (LABEL_P (note) 5461 || BARRIER_P (note))) 5462 note = NEXT_INSN (note); 5463 5464 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note)); 5465 } 5466 else 5467 note = insn; 5468 5469 SET_NEXT_INSN (PREV_INSN (insn)) = NEXT_INSN (note); 5470 SET_PREV_INSN (NEXT_INSN (note)) = PREV_INSN (insn); 5471 5472 SET_NEXT_INSN (note) = NEXT_INSN (last); 5473 SET_PREV_INSN (NEXT_INSN (last)) = note; 5474 5475 SET_NEXT_INSN (last) = insn; 5476 SET_PREV_INSN (insn) = last; 5477 5478 bb = BLOCK_FOR_INSN (last); 5479 5480 if (jump_p) 5481 { 5482 fix_jump_move (insn); 5483 5484 if (BLOCK_FOR_INSN (insn) != bb) 5485 move_block_after_check (insn); 5486 5487 gcc_assert (BB_END (bb) == last); 5488 } 5489 5490 df_insn_change_bb (insn, bb); 5491 5492 /* Update BB_END, if needed. */ 5493 if (BB_END (bb) == last) 5494 BB_END (bb) = insn; 5495 } 5496 5497 SCHED_GROUP_P (insn) = 0; 5498 } 5499 5500 /* Return true if scheduling INSN will finish current clock cycle. */ 5501 static bool 5502 insn_finishes_cycle_p (rtx_insn *insn) 5503 { 5504 if (SCHED_GROUP_P (insn)) 5505 /* After issuing INSN, rest of the sched_group will be forced to issue 5506 in order. Don't make any plans for the rest of cycle. */ 5507 return true; 5508 5509 /* Finishing the block will, apparently, finish the cycle. */ 5510 if (current_sched_info->insn_finishes_block_p 5511 && current_sched_info->insn_finishes_block_p (insn)) 5512 return true; 5513 5514 return false; 5515 } 5516 5517 /* Helper for autopref_multipass_init. Given a SET in PAT and whether 5518 we're expecting a memory WRITE or not, check that the insn is relevant to 5519 the autoprefetcher modelling code. Return true iff that is the case. 5520 If it is relevant, record the base register of the memory op in BASE and 5521 the offset in OFFSET. */ 5522 5523 static bool 5524 analyze_set_insn_for_autopref (rtx pat, bool write, rtx *base, int *offset) 5525 { 5526 if (GET_CODE (pat) != SET) 5527 return false; 5528 5529 rtx mem = write ? SET_DEST (pat) : SET_SRC (pat); 5530 if (!MEM_P (mem)) 5531 return false; 5532 5533 struct address_info info; 5534 decompose_mem_address (&info, mem); 5535 5536 /* TODO: Currently only (base+const) addressing is supported. */ 5537 if (info.base == NULL || !REG_P (*info.base) 5538 || (info.disp != NULL && !CONST_INT_P (*info.disp))) 5539 return false; 5540 5541 *base = *info.base; 5542 *offset = info.disp ? INTVAL (*info.disp) : 0; 5543 return true; 5544 } 5545 5546 /* Functions to model cache auto-prefetcher. 5547 5548 Some of the CPUs have cache auto-prefetcher, which /seems/ to initiate 5549 memory prefetches if it sees instructions with consequitive memory accesses 5550 in the instruction stream. Details of such hardware units are not published, 5551 so we can only guess what exactly is going on there. 5552 In the scheduler, we model abstract auto-prefetcher. If there are memory 5553 insns in the ready list (or the queue) that have same memory base, but 5554 different offsets, then we delay the insns with larger offsets until insns 5555 with smaller offsets get scheduled. If PARAM_SCHED_AUTOPREF_QUEUE_DEPTH 5556 is "1", then we look at the ready list; if it is N>1, then we also look 5557 through N-1 queue entries. 5558 If the param is N>=0, then rank_for_schedule will consider auto-prefetching 5559 among its heuristics. 5560 Param value of "-1" disables modelling of the auto-prefetcher. */ 5561 5562 /* Initialize autoprefetcher model data for INSN. */ 5563 static void 5564 autopref_multipass_init (const rtx_insn *insn, int write) 5565 { 5566 autopref_multipass_data_t data = &INSN_AUTOPREF_MULTIPASS_DATA (insn)[write]; 5567 5568 gcc_assert (data->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED); 5569 data->base = NULL_RTX; 5570 data->offset = 0; 5571 /* Set insn entry initialized, but not relevant for auto-prefetcher. */ 5572 data->status = AUTOPREF_MULTIPASS_DATA_IRRELEVANT; 5573 5574 rtx pat = PATTERN (insn); 5575 5576 /* We have a multi-set insn like a load-multiple or store-multiple. 5577 We care about these as long as all the memory ops inside the PARALLEL 5578 have the same base register. We care about the minimum and maximum 5579 offsets from that base but don't check for the order of those offsets 5580 within the PARALLEL insn itself. */ 5581 if (GET_CODE (pat) == PARALLEL) 5582 { 5583 int n_elems = XVECLEN (pat, 0); 5584 5585 int i, offset; 5586 rtx base, prev_base = NULL_RTX; 5587 int min_offset = INT_MAX; 5588 5589 for (i = 0; i < n_elems; i++) 5590 { 5591 rtx set = XVECEXP (pat, 0, i); 5592 if (GET_CODE (set) != SET) 5593 return; 5594 5595 if (!analyze_set_insn_for_autopref (set, write, &base, &offset)) 5596 return; 5597 5598 /* Ensure that all memory operations in the PARALLEL use the same 5599 base register. */ 5600 if (i > 0 && REGNO (base) != REGNO (prev_base)) 5601 return; 5602 prev_base = base; 5603 min_offset = MIN (min_offset, offset); 5604 } 5605 5606 /* If we reached here then we have a valid PARALLEL of multiple memory ops 5607 with prev_base as the base and min_offset containing the offset. */ 5608 gcc_assert (prev_base); 5609 data->base = prev_base; 5610 data->offset = min_offset; 5611 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL; 5612 return; 5613 } 5614 5615 /* Otherwise this is a single set memory operation. */ 5616 rtx set = single_set (insn); 5617 if (set == NULL_RTX) 5618 return; 5619 5620 if (!analyze_set_insn_for_autopref (set, write, &data->base, 5621 &data->offset)) 5622 return; 5623 5624 /* This insn is relevant for the auto-prefetcher. 5625 The base and offset fields will have been filled in the 5626 analyze_set_insn_for_autopref call above. */ 5627 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL; 5628 } 5629 5630 /* Helper function for rank_for_schedule sorting. */ 5631 static int 5632 autopref_rank_for_schedule (const rtx_insn *insn1, const rtx_insn *insn2) 5633 { 5634 int r = 0; 5635 for (int write = 0; write < 2 && !r; ++write) 5636 { 5637 autopref_multipass_data_t data1 5638 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write]; 5639 autopref_multipass_data_t data2 5640 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write]; 5641 5642 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED) 5643 autopref_multipass_init (insn1, write); 5644 5645 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED) 5646 autopref_multipass_init (insn2, write); 5647 5648 int irrel1 = data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT; 5649 int irrel2 = data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT; 5650 5651 if (!irrel1 && !irrel2) 5652 r = data1->offset - data2->offset; 5653 else 5654 r = irrel2 - irrel1; 5655 } 5656 5657 return r; 5658 } 5659 5660 /* True if header of debug dump was printed. */ 5661 static bool autopref_multipass_dfa_lookahead_guard_started_dump_p; 5662 5663 /* Helper for autopref_multipass_dfa_lookahead_guard. 5664 Return "1" if INSN1 should be delayed in favor of INSN2. */ 5665 static int 5666 autopref_multipass_dfa_lookahead_guard_1 (const rtx_insn *insn1, 5667 const rtx_insn *insn2, int write) 5668 { 5669 autopref_multipass_data_t data1 5670 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write]; 5671 autopref_multipass_data_t data2 5672 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write]; 5673 5674 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED) 5675 autopref_multipass_init (insn2, write); 5676 if (data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT) 5677 return 0; 5678 5679 if (rtx_equal_p (data1->base, data2->base) 5680 && data1->offset > data2->offset) 5681 { 5682 if (sched_verbose >= 2) 5683 { 5684 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p) 5685 { 5686 fprintf (sched_dump, 5687 ";;\t\tnot trying in max_issue due to autoprefetch " 5688 "model: "); 5689 autopref_multipass_dfa_lookahead_guard_started_dump_p = true; 5690 } 5691 5692 fprintf (sched_dump, " %d(%d)", INSN_UID (insn1), INSN_UID (insn2)); 5693 } 5694 5695 return 1; 5696 } 5697 5698 return 0; 5699 } 5700 5701 /* General note: 5702 5703 We could have also hooked autoprefetcher model into 5704 first_cycle_multipass_backtrack / first_cycle_multipass_issue hooks 5705 to enable intelligent selection of "[r1+0]=r2; [r1+4]=r3" on the same cycle 5706 (e.g., once "[r1+0]=r2" is issued in max_issue(), "[r1+4]=r3" gets 5707 unblocked). We don't bother about this yet because target of interest 5708 (ARM Cortex-A15) can issue only 1 memory operation per cycle. */ 5709 5710 /* Implementation of first_cycle_multipass_dfa_lookahead_guard hook. 5711 Return "1" if INSN1 should not be considered in max_issue due to 5712 auto-prefetcher considerations. */ 5713 int 5714 autopref_multipass_dfa_lookahead_guard (rtx_insn *insn1, int ready_index) 5715 { 5716 int r = 0; 5717 5718 /* Exit early if the param forbids this or if we're not entering here through 5719 normal haifa scheduling. This can happen if selective scheduling is 5720 explicitly enabled. */ 5721 if (!insn_queue || PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) <= 0) 5722 return 0; 5723 5724 if (sched_verbose >= 2 && ready_index == 0) 5725 autopref_multipass_dfa_lookahead_guard_started_dump_p = false; 5726 5727 for (int write = 0; write < 2; ++write) 5728 { 5729 autopref_multipass_data_t data1 5730 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write]; 5731 5732 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED) 5733 autopref_multipass_init (insn1, write); 5734 if (data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT) 5735 continue; 5736 5737 if (ready_index == 0 5738 && data1->status == AUTOPREF_MULTIPASS_DATA_DONT_DELAY) 5739 /* We allow only a single delay on priviledged instructions. 5740 Doing otherwise would cause infinite loop. */ 5741 { 5742 if (sched_verbose >= 2) 5743 { 5744 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p) 5745 { 5746 fprintf (sched_dump, 5747 ";;\t\tnot trying in max_issue due to autoprefetch " 5748 "model: "); 5749 autopref_multipass_dfa_lookahead_guard_started_dump_p = true; 5750 } 5751 5752 fprintf (sched_dump, " *%d*", INSN_UID (insn1)); 5753 } 5754 continue; 5755 } 5756 5757 for (int i2 = 0; i2 < ready.n_ready; ++i2) 5758 { 5759 rtx_insn *insn2 = get_ready_element (i2); 5760 if (insn1 == insn2) 5761 continue; 5762 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2, write); 5763 if (r) 5764 { 5765 if (ready_index == 0) 5766 { 5767 r = -1; 5768 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY; 5769 } 5770 goto finish; 5771 } 5772 } 5773 5774 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) == 1) 5775 continue; 5776 5777 /* Everything from the current queue slot should have been moved to 5778 the ready list. */ 5779 gcc_assert (insn_queue[NEXT_Q_AFTER (q_ptr, 0)] == NULL_RTX); 5780 5781 int n_stalls = PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) - 1; 5782 if (n_stalls > max_insn_queue_index) 5783 n_stalls = max_insn_queue_index; 5784 5785 for (int stalls = 1; stalls <= n_stalls; ++stalls) 5786 { 5787 for (rtx_insn_list *link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]; 5788 link != NULL_RTX; 5789 link = link->next ()) 5790 { 5791 rtx_insn *insn2 = link->insn (); 5792 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2, 5793 write); 5794 if (r) 5795 { 5796 /* Queue INSN1 until INSN2 can issue. */ 5797 r = -stalls; 5798 if (ready_index == 0) 5799 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY; 5800 goto finish; 5801 } 5802 } 5803 } 5804 } 5805 5806 finish: 5807 if (sched_verbose >= 2 5808 && autopref_multipass_dfa_lookahead_guard_started_dump_p 5809 && (ready_index == ready.n_ready - 1 || r < 0)) 5810 /* This does not /always/ trigger. We don't output EOL if the last 5811 insn is not recognized (INSN_CODE < 0) and lookahead_guard is not 5812 called. We can live with this. */ 5813 fprintf (sched_dump, "\n"); 5814 5815 return r; 5816 } 5817 5818 /* Define type for target data used in multipass scheduling. */ 5819 #ifndef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T 5820 # define TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T int 5821 #endif 5822 typedef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T first_cycle_multipass_data_t; 5823 5824 /* The following structure describe an entry of the stack of choices. */ 5825 struct choice_entry 5826 { 5827 /* Ordinal number of the issued insn in the ready queue. */ 5828 int index; 5829 /* The number of the rest insns whose issues we should try. */ 5830 int rest; 5831 /* The number of issued essential insns. */ 5832 int n; 5833 /* State after issuing the insn. */ 5834 state_t state; 5835 /* Target-specific data. */ 5836 first_cycle_multipass_data_t target_data; 5837 }; 5838 5839 /* The following array is used to implement a stack of choices used in 5840 function max_issue. */ 5841 static struct choice_entry *choice_stack; 5842 5843 /* This holds the value of the target dfa_lookahead hook. */ 5844 int dfa_lookahead; 5845 5846 /* The following variable value is maximal number of tries of issuing 5847 insns for the first cycle multipass insn scheduling. We define 5848 this value as constant*(DFA_LOOKAHEAD**ISSUE_RATE). We would not 5849 need this constraint if all real insns (with non-negative codes) 5850 had reservations because in this case the algorithm complexity is 5851 O(DFA_LOOKAHEAD**ISSUE_RATE). Unfortunately, the dfa descriptions 5852 might be incomplete and such insn might occur. For such 5853 descriptions, the complexity of algorithm (without the constraint) 5854 could achieve DFA_LOOKAHEAD ** N , where N is the queue length. */ 5855 static int max_lookahead_tries; 5856 5857 /* The following function returns maximal (or close to maximal) number 5858 of insns which can be issued on the same cycle and one of which 5859 insns is insns with the best rank (the first insn in READY). To 5860 make this function tries different samples of ready insns. READY 5861 is current queue `ready'. Global array READY_TRY reflects what 5862 insns are already issued in this try. The function stops immediately, 5863 if it reached the such a solution, that all instruction can be issued. 5864 INDEX will contain index of the best insn in READY. The following 5865 function is used only for first cycle multipass scheduling. 5866 5867 PRIVILEGED_N >= 0 5868 5869 This function expects recognized insns only. All USEs, 5870 CLOBBERs, etc must be filtered elsewhere. */ 5871 int 5872 max_issue (struct ready_list *ready, int privileged_n, state_t state, 5873 bool first_cycle_insn_p, int *index) 5874 { 5875 int n, i, all, n_ready, best, delay, tries_num; 5876 int more_issue; 5877 struct choice_entry *top; 5878 rtx_insn *insn; 5879 5880 if (sched_fusion) 5881 return 0; 5882 5883 n_ready = ready->n_ready; 5884 gcc_assert (dfa_lookahead >= 1 && privileged_n >= 0 5885 && privileged_n <= n_ready); 5886 5887 /* Init MAX_LOOKAHEAD_TRIES. */ 5888 if (max_lookahead_tries == 0) 5889 { 5890 max_lookahead_tries = 100; 5891 for (i = 0; i < issue_rate; i++) 5892 max_lookahead_tries *= dfa_lookahead; 5893 } 5894 5895 /* Init max_points. */ 5896 more_issue = issue_rate - cycle_issued_insns; 5897 gcc_assert (more_issue >= 0); 5898 5899 /* The number of the issued insns in the best solution. */ 5900 best = 0; 5901 5902 top = choice_stack; 5903 5904 /* Set initial state of the search. */ 5905 memcpy (top->state, state, dfa_state_size); 5906 top->rest = dfa_lookahead; 5907 top->n = 0; 5908 if (targetm.sched.first_cycle_multipass_begin) 5909 targetm.sched.first_cycle_multipass_begin (&top->target_data, 5910 ready_try, n_ready, 5911 first_cycle_insn_p); 5912 5913 /* Count the number of the insns to search among. */ 5914 for (all = i = 0; i < n_ready; i++) 5915 if (!ready_try [i]) 5916 all++; 5917 5918 if (sched_verbose >= 2) 5919 { 5920 fprintf (sched_dump, ";;\t\tmax_issue among %d insns:", all); 5921 debug_ready_list_1 (ready, ready_try); 5922 } 5923 5924 /* I is the index of the insn to try next. */ 5925 i = 0; 5926 tries_num = 0; 5927 for (;;) 5928 { 5929 if (/* If we've reached a dead end or searched enough of what we have 5930 been asked... */ 5931 top->rest == 0 5932 /* or have nothing else to try... */ 5933 || i >= n_ready 5934 /* or should not issue more. */ 5935 || top->n >= more_issue) 5936 { 5937 /* ??? (... || i == n_ready). */ 5938 gcc_assert (i <= n_ready); 5939 5940 /* We should not issue more than issue_rate instructions. */ 5941 gcc_assert (top->n <= more_issue); 5942 5943 if (top == choice_stack) 5944 break; 5945 5946 if (best < top - choice_stack) 5947 { 5948 if (privileged_n) 5949 { 5950 n = privileged_n; 5951 /* Try to find issued privileged insn. */ 5952 while (n && !ready_try[--n]) 5953 ; 5954 } 5955 5956 if (/* If all insns are equally good... */ 5957 privileged_n == 0 5958 /* Or a privileged insn will be issued. */ 5959 || ready_try[n]) 5960 /* Then we have a solution. */ 5961 { 5962 best = top - choice_stack; 5963 /* This is the index of the insn issued first in this 5964 solution. */ 5965 *index = choice_stack [1].index; 5966 if (top->n == more_issue || best == all) 5967 break; 5968 } 5969 } 5970 5971 /* Set ready-list index to point to the last insn 5972 ('i++' below will advance it to the next insn). */ 5973 i = top->index; 5974 5975 /* Backtrack. */ 5976 ready_try [i] = 0; 5977 5978 if (targetm.sched.first_cycle_multipass_backtrack) 5979 targetm.sched.first_cycle_multipass_backtrack (&top->target_data, 5980 ready_try, n_ready); 5981 5982 top--; 5983 memcpy (state, top->state, dfa_state_size); 5984 } 5985 else if (!ready_try [i]) 5986 { 5987 tries_num++; 5988 if (tries_num > max_lookahead_tries) 5989 break; 5990 insn = ready_element (ready, i); 5991 delay = state_transition (state, insn); 5992 if (delay < 0) 5993 { 5994 if (state_dead_lock_p (state) 5995 || insn_finishes_cycle_p (insn)) 5996 /* We won't issue any more instructions in the next 5997 choice_state. */ 5998 top->rest = 0; 5999 else 6000 top->rest--; 6001 6002 n = top->n; 6003 if (memcmp (top->state, state, dfa_state_size) != 0) 6004 n++; 6005 6006 /* Advance to the next choice_entry. */ 6007 top++; 6008 /* Initialize it. */ 6009 top->rest = dfa_lookahead; 6010 top->index = i; 6011 top->n = n; 6012 memcpy (top->state, state, dfa_state_size); 6013 ready_try [i] = 1; 6014 6015 if (targetm.sched.first_cycle_multipass_issue) 6016 targetm.sched.first_cycle_multipass_issue (&top->target_data, 6017 ready_try, n_ready, 6018 insn, 6019 &((top - 1) 6020 ->target_data)); 6021 6022 i = -1; 6023 } 6024 } 6025 6026 /* Increase ready-list index. */ 6027 i++; 6028 } 6029 6030 if (targetm.sched.first_cycle_multipass_end) 6031 targetm.sched.first_cycle_multipass_end (best != 0 6032 ? &choice_stack[1].target_data 6033 : NULL); 6034 6035 /* Restore the original state of the DFA. */ 6036 memcpy (state, choice_stack->state, dfa_state_size); 6037 6038 return best; 6039 } 6040 6041 /* The following function chooses insn from READY and modifies 6042 READY. The following function is used only for first 6043 cycle multipass scheduling. 6044 Return: 6045 -1 if cycle should be advanced, 6046 0 if INSN_PTR is set to point to the desirable insn, 6047 1 if choose_ready () should be restarted without advancing the cycle. */ 6048 static int 6049 choose_ready (struct ready_list *ready, bool first_cycle_insn_p, 6050 rtx_insn **insn_ptr) 6051 { 6052 if (dbg_cnt (sched_insn) == false) 6053 { 6054 if (nonscheduled_insns_begin == NULL_RTX) 6055 nonscheduled_insns_begin = current_sched_info->prev_head; 6056 6057 rtx_insn *insn = first_nonscheduled_insn (); 6058 6059 if (QUEUE_INDEX (insn) == QUEUE_READY) 6060 /* INSN is in the ready_list. */ 6061 { 6062 ready_remove_insn (insn); 6063 *insn_ptr = insn; 6064 return 0; 6065 } 6066 6067 /* INSN is in the queue. Advance cycle to move it to the ready list. */ 6068 gcc_assert (QUEUE_INDEX (insn) >= 0); 6069 return -1; 6070 } 6071 6072 if (dfa_lookahead <= 0 || SCHED_GROUP_P (ready_element (ready, 0)) 6073 || DEBUG_INSN_P (ready_element (ready, 0))) 6074 { 6075 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON)) 6076 *insn_ptr = ready_remove_first_dispatch (ready); 6077 else 6078 *insn_ptr = ready_remove_first (ready); 6079 6080 return 0; 6081 } 6082 else 6083 { 6084 /* Try to choose the best insn. */ 6085 int index = 0, i; 6086 rtx_insn *insn; 6087 6088 insn = ready_element (ready, 0); 6089 if (INSN_CODE (insn) < 0) 6090 { 6091 *insn_ptr = ready_remove_first (ready); 6092 return 0; 6093 } 6094 6095 /* Filter the search space. */ 6096 for (i = 0; i < ready->n_ready; i++) 6097 { 6098 ready_try[i] = 0; 6099 6100 insn = ready_element (ready, i); 6101 6102 /* If this insn is recognizable we should have already 6103 recognized it earlier. 6104 ??? Not very clear where this is supposed to be done. 6105 See dep_cost_1. */ 6106 gcc_checking_assert (INSN_CODE (insn) >= 0 6107 || recog_memoized (insn) < 0); 6108 if (INSN_CODE (insn) < 0) 6109 { 6110 /* Non-recognized insns at position 0 are handled above. */ 6111 gcc_assert (i > 0); 6112 ready_try[i] = 1; 6113 continue; 6114 } 6115 6116 if (targetm.sched.first_cycle_multipass_dfa_lookahead_guard) 6117 { 6118 ready_try[i] 6119 = (targetm.sched.first_cycle_multipass_dfa_lookahead_guard 6120 (insn, i)); 6121 6122 if (ready_try[i] < 0) 6123 /* Queue instruction for several cycles. 6124 We need to restart choose_ready as we have changed 6125 the ready list. */ 6126 { 6127 change_queue_index (insn, -ready_try[i]); 6128 return 1; 6129 } 6130 6131 /* Make sure that we didn't end up with 0'th insn filtered out. 6132 Don't be tempted to make life easier for backends and just 6133 requeue 0'th insn if (ready_try[0] == 0) and restart 6134 choose_ready. Backends should be very considerate about 6135 requeueing instructions -- especially the highest priority 6136 one at position 0. */ 6137 gcc_assert (ready_try[i] == 0 || i > 0); 6138 if (ready_try[i]) 6139 continue; 6140 } 6141 6142 gcc_assert (ready_try[i] == 0); 6143 /* INSN made it through the scrutiny of filters! */ 6144 } 6145 6146 if (max_issue (ready, 1, curr_state, first_cycle_insn_p, &index) == 0) 6147 { 6148 *insn_ptr = ready_remove_first (ready); 6149 if (sched_verbose >= 4) 6150 fprintf (sched_dump, ";;\t\tChosen insn (but can't issue) : %s \n", 6151 (*current_sched_info->print_insn) (*insn_ptr, 0)); 6152 return 0; 6153 } 6154 else 6155 { 6156 if (sched_verbose >= 4) 6157 fprintf (sched_dump, ";;\t\tChosen insn : %s\n", 6158 (*current_sched_info->print_insn) 6159 (ready_element (ready, index), 0)); 6160 6161 *insn_ptr = ready_remove (ready, index); 6162 return 0; 6163 } 6164 } 6165 } 6166 6167 /* This function is called when we have successfully scheduled a 6168 block. It uses the schedule stored in the scheduled_insns vector 6169 to rearrange the RTL. PREV_HEAD is used as the anchor to which we 6170 append the scheduled insns; TAIL is the insn after the scheduled 6171 block. TARGET_BB is the argument passed to schedule_block. */ 6172 6173 static void 6174 commit_schedule (rtx_insn *prev_head, rtx_insn *tail, basic_block *target_bb) 6175 { 6176 unsigned int i; 6177 rtx_insn *insn; 6178 6179 last_scheduled_insn = prev_head; 6180 for (i = 0; 6181 scheduled_insns.iterate (i, &insn); 6182 i++) 6183 { 6184 if (control_flow_insn_p (last_scheduled_insn) 6185 || current_sched_info->advance_target_bb (*target_bb, insn)) 6186 { 6187 *target_bb = current_sched_info->advance_target_bb (*target_bb, 0); 6188 6189 if (sched_verbose) 6190 { 6191 rtx_insn *x; 6192 6193 x = next_real_insn (last_scheduled_insn); 6194 gcc_assert (x); 6195 dump_new_block_header (1, *target_bb, x, tail); 6196 } 6197 6198 last_scheduled_insn = bb_note (*target_bb); 6199 } 6200 6201 if (current_sched_info->begin_move_insn) 6202 (*current_sched_info->begin_move_insn) (insn, last_scheduled_insn); 6203 move_insn (insn, last_scheduled_insn, 6204 current_sched_info->next_tail); 6205 if (!DEBUG_INSN_P (insn)) 6206 reemit_notes (insn); 6207 last_scheduled_insn = insn; 6208 } 6209 6210 scheduled_insns.truncate (0); 6211 } 6212 6213 /* Examine all insns on the ready list and queue those which can't be 6214 issued in this cycle. TEMP_STATE is temporary scheduler state we 6215 can use as scratch space. If FIRST_CYCLE_INSN_P is true, no insns 6216 have been issued for the current cycle, which means it is valid to 6217 issue an asm statement. 6218 6219 If SHADOWS_ONLY_P is true, we eliminate all real insns and only 6220 leave those for which SHADOW_P is true. If MODULO_EPILOGUE is true, 6221 we only leave insns which have an INSN_EXACT_TICK. */ 6222 6223 static void 6224 prune_ready_list (state_t temp_state, bool first_cycle_insn_p, 6225 bool shadows_only_p, bool modulo_epilogue_p) 6226 { 6227 int i, pass; 6228 bool sched_group_found = false; 6229 int min_cost_group = 0; 6230 6231 if (sched_fusion) 6232 return; 6233 6234 for (i = 0; i < ready.n_ready; i++) 6235 { 6236 rtx_insn *insn = ready_element (&ready, i); 6237 if (SCHED_GROUP_P (insn)) 6238 { 6239 sched_group_found = true; 6240 break; 6241 } 6242 } 6243 6244 /* Make two passes if there's a SCHED_GROUP_P insn; make sure to handle 6245 such an insn first and note its cost. If at least one SCHED_GROUP_P insn 6246 gets queued, then all other insns get queued for one cycle later. */ 6247 for (pass = sched_group_found ? 0 : 1; pass < 2; ) 6248 { 6249 int n = ready.n_ready; 6250 for (i = 0; i < n; i++) 6251 { 6252 rtx_insn *insn = ready_element (&ready, i); 6253 int cost = 0; 6254 const char *reason = "resource conflict"; 6255 6256 if (DEBUG_INSN_P (insn)) 6257 continue; 6258 6259 if (sched_group_found && !SCHED_GROUP_P (insn) 6260 && ((pass == 0) || (min_cost_group >= 1))) 6261 { 6262 if (pass == 0) 6263 continue; 6264 cost = min_cost_group; 6265 reason = "not in sched group"; 6266 } 6267 else if (modulo_epilogue_p 6268 && INSN_EXACT_TICK (insn) == INVALID_TICK) 6269 { 6270 cost = max_insn_queue_index; 6271 reason = "not an epilogue insn"; 6272 } 6273 else if (shadows_only_p && !SHADOW_P (insn)) 6274 { 6275 cost = 1; 6276 reason = "not a shadow"; 6277 } 6278 else if (recog_memoized (insn) < 0) 6279 { 6280 if (!first_cycle_insn_p 6281 && (GET_CODE (PATTERN (insn)) == ASM_INPUT 6282 || asm_noperands (PATTERN (insn)) >= 0)) 6283 cost = 1; 6284 reason = "asm"; 6285 } 6286 else if (sched_pressure != SCHED_PRESSURE_NONE) 6287 { 6288 if (sched_pressure == SCHED_PRESSURE_MODEL 6289 && INSN_TICK (insn) <= clock_var) 6290 { 6291 memcpy (temp_state, curr_state, dfa_state_size); 6292 if (state_transition (temp_state, insn) >= 0) 6293 INSN_TICK (insn) = clock_var + 1; 6294 } 6295 cost = 0; 6296 } 6297 else 6298 { 6299 int delay_cost = 0; 6300 6301 if (delay_htab) 6302 { 6303 struct delay_pair *delay_entry; 6304 delay_entry 6305 = delay_htab->find_with_hash (insn, 6306 htab_hash_pointer (insn)); 6307 while (delay_entry && delay_cost == 0) 6308 { 6309 delay_cost = estimate_shadow_tick (delay_entry); 6310 if (delay_cost > max_insn_queue_index) 6311 delay_cost = max_insn_queue_index; 6312 delay_entry = delay_entry->next_same_i1; 6313 } 6314 } 6315 6316 memcpy (temp_state, curr_state, dfa_state_size); 6317 cost = state_transition (temp_state, insn); 6318 if (cost < 0) 6319 cost = 0; 6320 else if (cost == 0) 6321 cost = 1; 6322 if (cost < delay_cost) 6323 { 6324 cost = delay_cost; 6325 reason = "shadow tick"; 6326 } 6327 } 6328 if (cost >= 1) 6329 { 6330 if (SCHED_GROUP_P (insn) && cost > min_cost_group) 6331 min_cost_group = cost; 6332 ready_remove (&ready, i); 6333 /* Normally we'd want to queue INSN for COST cycles. However, 6334 if SCHED_GROUP_P is set, then we must ensure that nothing 6335 else comes between INSN and its predecessor. If there is 6336 some other insn ready to fire on the next cycle, then that 6337 invariant would be broken. 6338 6339 So when SCHED_GROUP_P is set, just queue this insn for a 6340 single cycle. */ 6341 queue_insn (insn, SCHED_GROUP_P (insn) ? 1 : cost, reason); 6342 if (i + 1 < n) 6343 break; 6344 } 6345 } 6346 if (i == n) 6347 pass++; 6348 } 6349 } 6350 6351 /* Called when we detect that the schedule is impossible. We examine the 6352 backtrack queue to find the earliest insn that caused this condition. */ 6353 6354 static struct haifa_saved_data * 6355 verify_shadows (void) 6356 { 6357 struct haifa_saved_data *save, *earliest_fail = NULL; 6358 for (save = backtrack_queue; save; save = save->next) 6359 { 6360 int t; 6361 struct delay_pair *pair = save->delay_pair; 6362 rtx_insn *i1 = pair->i1; 6363 6364 for (; pair; pair = pair->next_same_i1) 6365 { 6366 rtx_insn *i2 = pair->i2; 6367 6368 if (QUEUE_INDEX (i2) == QUEUE_SCHEDULED) 6369 continue; 6370 6371 t = INSN_TICK (i1) + pair_delay (pair); 6372 if (t < clock_var) 6373 { 6374 if (sched_verbose >= 2) 6375 fprintf (sched_dump, 6376 ";;\t\tfailed delay requirements for %d/%d (%d->%d)" 6377 ", not ready\n", 6378 INSN_UID (pair->i1), INSN_UID (pair->i2), 6379 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2)); 6380 earliest_fail = save; 6381 break; 6382 } 6383 if (QUEUE_INDEX (i2) >= 0) 6384 { 6385 int queued_for = INSN_TICK (i2); 6386 6387 if (t < queued_for) 6388 { 6389 if (sched_verbose >= 2) 6390 fprintf (sched_dump, 6391 ";;\t\tfailed delay requirements for %d/%d" 6392 " (%d->%d), queued too late\n", 6393 INSN_UID (pair->i1), INSN_UID (pair->i2), 6394 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2)); 6395 earliest_fail = save; 6396 break; 6397 } 6398 } 6399 } 6400 } 6401 6402 return earliest_fail; 6403 } 6404 6405 /* Print instructions together with useful scheduling information between 6406 HEAD and TAIL (inclusive). */ 6407 static void 6408 dump_insn_stream (rtx_insn *head, rtx_insn *tail) 6409 { 6410 fprintf (sched_dump, ";;\t| insn | prio |\n"); 6411 6412 rtx_insn *next_tail = NEXT_INSN (tail); 6413 for (rtx_insn *insn = head; insn != next_tail; insn = NEXT_INSN (insn)) 6414 { 6415 int priority = NOTE_P (insn) ? 0 : INSN_PRIORITY (insn); 6416 const char *pattern = (NOTE_P (insn) 6417 ? "note" 6418 : str_pattern_slim (PATTERN (insn))); 6419 6420 fprintf (sched_dump, ";;\t| %4d | %4d | %-30s ", 6421 INSN_UID (insn), priority, pattern); 6422 6423 if (sched_verbose >= 4) 6424 { 6425 if (NOTE_P (insn) || LABEL_P (insn) || recog_memoized (insn) < 0) 6426 fprintf (sched_dump, "nothing"); 6427 else 6428 print_reservation (sched_dump, insn); 6429 } 6430 fprintf (sched_dump, "\n"); 6431 } 6432 } 6433 6434 /* Use forward list scheduling to rearrange insns of block pointed to by 6435 TARGET_BB, possibly bringing insns from subsequent blocks in the same 6436 region. */ 6437 6438 bool 6439 schedule_block (basic_block *target_bb, state_t init_state) 6440 { 6441 int i; 6442 bool success = modulo_ii == 0; 6443 struct sched_block_state ls; 6444 state_t temp_state = NULL; /* It is used for multipass scheduling. */ 6445 int sort_p, advance, start_clock_var; 6446 6447 /* Head/tail info for this block. */ 6448 rtx_insn *prev_head = current_sched_info->prev_head; 6449 rtx_insn *next_tail = current_sched_info->next_tail; 6450 rtx_insn *head = NEXT_INSN (prev_head); 6451 rtx_insn *tail = PREV_INSN (next_tail); 6452 6453 if ((current_sched_info->flags & DONT_BREAK_DEPENDENCIES) == 0 6454 && sched_pressure != SCHED_PRESSURE_MODEL && !sched_fusion) 6455 find_modifiable_mems (head, tail); 6456 6457 /* We used to have code to avoid getting parameters moved from hard 6458 argument registers into pseudos. 6459 6460 However, it was removed when it proved to be of marginal benefit 6461 and caused problems because schedule_block and compute_forward_dependences 6462 had different notions of what the "head" insn was. */ 6463 6464 gcc_assert (head != tail || INSN_P (head)); 6465 6466 haifa_recovery_bb_recently_added_p = false; 6467 6468 backtrack_queue = NULL; 6469 6470 /* Debug info. */ 6471 if (sched_verbose) 6472 { 6473 dump_new_block_header (0, *target_bb, head, tail); 6474 6475 if (sched_verbose >= 2) 6476 { 6477 dump_insn_stream (head, tail); 6478 memset (&rank_for_schedule_stats, 0, 6479 sizeof (rank_for_schedule_stats)); 6480 } 6481 } 6482 6483 if (init_state == NULL) 6484 state_reset (curr_state); 6485 else 6486 memcpy (curr_state, init_state, dfa_state_size); 6487 6488 /* Clear the ready list. */ 6489 ready.first = ready.veclen - 1; 6490 ready.n_ready = 0; 6491 ready.n_debug = 0; 6492 6493 /* It is used for first cycle multipass scheduling. */ 6494 temp_state = alloca (dfa_state_size); 6495 6496 if (targetm.sched.init) 6497 targetm.sched.init (sched_dump, sched_verbose, ready.veclen); 6498 6499 /* We start inserting insns after PREV_HEAD. */ 6500 last_scheduled_insn = prev_head; 6501 last_nondebug_scheduled_insn = NULL; 6502 nonscheduled_insns_begin = NULL; 6503 6504 gcc_assert ((NOTE_P (last_scheduled_insn) 6505 || DEBUG_INSN_P (last_scheduled_insn)) 6506 && BLOCK_FOR_INSN (last_scheduled_insn) == *target_bb); 6507 6508 /* Initialize INSN_QUEUE. Q_SIZE is the total number of insns in the 6509 queue. */ 6510 q_ptr = 0; 6511 q_size = 0; 6512 6513 insn_queue = XALLOCAVEC (rtx_insn_list *, max_insn_queue_index + 1); 6514 memset (insn_queue, 0, (max_insn_queue_index + 1) * sizeof (rtx)); 6515 6516 /* Start just before the beginning of time. */ 6517 clock_var = -1; 6518 6519 /* We need queue and ready lists and clock_var be initialized 6520 in try_ready () (which is called through init_ready_list ()). */ 6521 (*current_sched_info->init_ready_list) (); 6522 6523 if (sched_pressure) 6524 sched_pressure_start_bb (*target_bb); 6525 6526 /* The algorithm is O(n^2) in the number of ready insns at any given 6527 time in the worst case. Before reload we are more likely to have 6528 big lists so truncate them to a reasonable size. */ 6529 if (!reload_completed 6530 && ready.n_ready - ready.n_debug > MAX_SCHED_READY_INSNS) 6531 { 6532 ready_sort_debug (&ready); 6533 ready_sort_real (&ready); 6534 6535 /* Find first free-standing insn past MAX_SCHED_READY_INSNS. 6536 If there are debug insns, we know they're first. */ 6537 for (i = MAX_SCHED_READY_INSNS + ready.n_debug; i < ready.n_ready; i++) 6538 if (!SCHED_GROUP_P (ready_element (&ready, i))) 6539 break; 6540 6541 if (sched_verbose >= 2) 6542 { 6543 fprintf (sched_dump, 6544 ";;\t\tReady list on entry: %d insns: ", ready.n_ready); 6545 debug_ready_list (&ready); 6546 fprintf (sched_dump, 6547 ";;\t\t before reload => truncated to %d insns\n", i); 6548 } 6549 6550 /* Delay all insns past it for 1 cycle. If debug counter is 6551 activated make an exception for the insn right after 6552 nonscheduled_insns_begin. */ 6553 { 6554 rtx_insn *skip_insn; 6555 6556 if (dbg_cnt (sched_insn) == false) 6557 skip_insn = first_nonscheduled_insn (); 6558 else 6559 skip_insn = NULL; 6560 6561 while (i < ready.n_ready) 6562 { 6563 rtx_insn *insn; 6564 6565 insn = ready_remove (&ready, i); 6566 6567 if (insn != skip_insn) 6568 queue_insn (insn, 1, "list truncated"); 6569 } 6570 if (skip_insn) 6571 ready_add (&ready, skip_insn, true); 6572 } 6573 } 6574 6575 /* Now we can restore basic block notes and maintain precise cfg. */ 6576 restore_bb_notes (*target_bb); 6577 6578 last_clock_var = -1; 6579 6580 advance = 0; 6581 6582 gcc_assert (scheduled_insns.length () == 0); 6583 sort_p = TRUE; 6584 must_backtrack = false; 6585 modulo_insns_scheduled = 0; 6586 6587 ls.modulo_epilogue = false; 6588 ls.first_cycle_insn_p = true; 6589 6590 /* Loop until all the insns in BB are scheduled. */ 6591 while ((*current_sched_info->schedule_more_p) ()) 6592 { 6593 perform_replacements_new_cycle (); 6594 do 6595 { 6596 start_clock_var = clock_var; 6597 6598 clock_var++; 6599 6600 advance_one_cycle (); 6601 6602 /* Add to the ready list all pending insns that can be issued now. 6603 If there are no ready insns, increment clock until one 6604 is ready and add all pending insns at that point to the ready 6605 list. */ 6606 queue_to_ready (&ready); 6607 6608 gcc_assert (ready.n_ready); 6609 6610 if (sched_verbose >= 2) 6611 { 6612 fprintf (sched_dump, ";;\t\tReady list after queue_to_ready:"); 6613 debug_ready_list (&ready); 6614 } 6615 advance -= clock_var - start_clock_var; 6616 } 6617 while (advance > 0); 6618 6619 if (ls.modulo_epilogue) 6620 { 6621 int stage = clock_var / modulo_ii; 6622 if (stage > modulo_last_stage * 2 + 2) 6623 { 6624 if (sched_verbose >= 2) 6625 fprintf (sched_dump, 6626 ";;\t\tmodulo scheduled succeeded at II %d\n", 6627 modulo_ii); 6628 success = true; 6629 goto end_schedule; 6630 } 6631 } 6632 else if (modulo_ii > 0) 6633 { 6634 int stage = clock_var / modulo_ii; 6635 if (stage > modulo_max_stages) 6636 { 6637 if (sched_verbose >= 2) 6638 fprintf (sched_dump, 6639 ";;\t\tfailing schedule due to excessive stages\n"); 6640 goto end_schedule; 6641 } 6642 if (modulo_n_insns == modulo_insns_scheduled 6643 && stage > modulo_last_stage) 6644 { 6645 if (sched_verbose >= 2) 6646 fprintf (sched_dump, 6647 ";;\t\tfound kernel after %d stages, II %d\n", 6648 stage, modulo_ii); 6649 ls.modulo_epilogue = true; 6650 } 6651 } 6652 6653 prune_ready_list (temp_state, true, false, ls.modulo_epilogue); 6654 if (ready.n_ready == 0) 6655 continue; 6656 if (must_backtrack) 6657 goto do_backtrack; 6658 6659 ls.shadows_only_p = false; 6660 cycle_issued_insns = 0; 6661 ls.can_issue_more = issue_rate; 6662 for (;;) 6663 { 6664 rtx_insn *insn; 6665 int cost; 6666 bool asm_p; 6667 6668 if (sort_p && ready.n_ready > 0) 6669 { 6670 /* Sort the ready list based on priority. This must be 6671 done every iteration through the loop, as schedule_insn 6672 may have readied additional insns that will not be 6673 sorted correctly. */ 6674 ready_sort (&ready); 6675 6676 if (sched_verbose >= 2) 6677 { 6678 fprintf (sched_dump, 6679 ";;\t\tReady list after ready_sort: "); 6680 debug_ready_list (&ready); 6681 } 6682 } 6683 6684 /* We don't want md sched reorder to even see debug isns, so put 6685 them out right away. */ 6686 if (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0)) 6687 && (*current_sched_info->schedule_more_p) ()) 6688 { 6689 while (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0))) 6690 { 6691 rtx_insn *insn = ready_remove_first (&ready); 6692 gcc_assert (DEBUG_INSN_P (insn)); 6693 (*current_sched_info->begin_schedule_ready) (insn); 6694 scheduled_insns.safe_push (insn); 6695 last_scheduled_insn = insn; 6696 advance = schedule_insn (insn); 6697 gcc_assert (advance == 0); 6698 if (ready.n_ready > 0) 6699 ready_sort (&ready); 6700 } 6701 } 6702 6703 if (ls.first_cycle_insn_p && !ready.n_ready) 6704 break; 6705 6706 resume_after_backtrack: 6707 /* Allow the target to reorder the list, typically for 6708 better instruction bundling. */ 6709 if (sort_p 6710 && (ready.n_ready == 0 6711 || !SCHED_GROUP_P (ready_element (&ready, 0)))) 6712 { 6713 if (ls.first_cycle_insn_p && targetm.sched.reorder) 6714 ls.can_issue_more 6715 = targetm.sched.reorder (sched_dump, sched_verbose, 6716 ready_lastpos (&ready), 6717 &ready.n_ready, clock_var); 6718 else if (!ls.first_cycle_insn_p && targetm.sched.reorder2) 6719 ls.can_issue_more 6720 = targetm.sched.reorder2 (sched_dump, sched_verbose, 6721 ready.n_ready 6722 ? ready_lastpos (&ready) : NULL, 6723 &ready.n_ready, clock_var); 6724 } 6725 6726 restart_choose_ready: 6727 if (sched_verbose >= 2) 6728 { 6729 fprintf (sched_dump, ";;\tReady list (t = %3d): ", 6730 clock_var); 6731 debug_ready_list (&ready); 6732 if (sched_pressure == SCHED_PRESSURE_WEIGHTED) 6733 print_curr_reg_pressure (); 6734 } 6735 6736 if (ready.n_ready == 0 6737 && ls.can_issue_more 6738 && reload_completed) 6739 { 6740 /* Allow scheduling insns directly from the queue in case 6741 there's nothing better to do (ready list is empty) but 6742 there are still vacant dispatch slots in the current cycle. */ 6743 if (sched_verbose >= 6) 6744 fprintf (sched_dump,";;\t\tSecond chance\n"); 6745 memcpy (temp_state, curr_state, dfa_state_size); 6746 if (early_queue_to_ready (temp_state, &ready)) 6747 ready_sort (&ready); 6748 } 6749 6750 if (ready.n_ready == 0 6751 || !ls.can_issue_more 6752 || state_dead_lock_p (curr_state) 6753 || !(*current_sched_info->schedule_more_p) ()) 6754 break; 6755 6756 /* Select and remove the insn from the ready list. */ 6757 if (sort_p) 6758 { 6759 int res; 6760 6761 insn = NULL; 6762 res = choose_ready (&ready, ls.first_cycle_insn_p, &insn); 6763 6764 if (res < 0) 6765 /* Finish cycle. */ 6766 break; 6767 if (res > 0) 6768 goto restart_choose_ready; 6769 6770 gcc_assert (insn != NULL_RTX); 6771 } 6772 else 6773 insn = ready_remove_first (&ready); 6774 6775 if (sched_pressure != SCHED_PRESSURE_NONE 6776 && INSN_TICK (insn) > clock_var) 6777 { 6778 ready_add (&ready, insn, true); 6779 advance = 1; 6780 break; 6781 } 6782 6783 if (targetm.sched.dfa_new_cycle 6784 && targetm.sched.dfa_new_cycle (sched_dump, sched_verbose, 6785 insn, last_clock_var, 6786 clock_var, &sort_p)) 6787 /* SORT_P is used by the target to override sorting 6788 of the ready list. This is needed when the target 6789 has modified its internal structures expecting that 6790 the insn will be issued next. As we need the insn 6791 to have the highest priority (so it will be returned by 6792 the ready_remove_first call above), we invoke 6793 ready_add (&ready, insn, true). 6794 But, still, there is one issue: INSN can be later 6795 discarded by scheduler's front end through 6796 current_sched_info->can_schedule_ready_p, hence, won't 6797 be issued next. */ 6798 { 6799 ready_add (&ready, insn, true); 6800 break; 6801 } 6802 6803 sort_p = TRUE; 6804 6805 if (current_sched_info->can_schedule_ready_p 6806 && ! (*current_sched_info->can_schedule_ready_p) (insn)) 6807 /* We normally get here only if we don't want to move 6808 insn from the split block. */ 6809 { 6810 TODO_SPEC (insn) = DEP_POSTPONED; 6811 goto restart_choose_ready; 6812 } 6813 6814 if (delay_htab) 6815 { 6816 /* If this insn is the first part of a delay-slot pair, record a 6817 backtrack point. */ 6818 struct delay_pair *delay_entry; 6819 delay_entry 6820 = delay_htab->find_with_hash (insn, htab_hash_pointer (insn)); 6821 if (delay_entry) 6822 { 6823 save_backtrack_point (delay_entry, ls); 6824 if (sched_verbose >= 2) 6825 fprintf (sched_dump, ";;\t\tsaving backtrack point\n"); 6826 } 6827 } 6828 6829 /* DECISION is made. */ 6830 6831 if (modulo_ii > 0 && INSN_UID (insn) < modulo_iter0_max_uid) 6832 { 6833 modulo_insns_scheduled++; 6834 modulo_last_stage = clock_var / modulo_ii; 6835 } 6836 if (TODO_SPEC (insn) & SPECULATIVE) 6837 generate_recovery_code (insn); 6838 6839 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON)) 6840 targetm.sched.dispatch_do (insn, ADD_TO_DISPATCH_WINDOW); 6841 6842 /* Update counters, etc in the scheduler's front end. */ 6843 (*current_sched_info->begin_schedule_ready) (insn); 6844 scheduled_insns.safe_push (insn); 6845 gcc_assert (NONDEBUG_INSN_P (insn)); 6846 last_nondebug_scheduled_insn = last_scheduled_insn = insn; 6847 6848 if (recog_memoized (insn) >= 0) 6849 { 6850 memcpy (temp_state, curr_state, dfa_state_size); 6851 cost = state_transition (curr_state, insn); 6852 if (sched_pressure != SCHED_PRESSURE_WEIGHTED && !sched_fusion) 6853 gcc_assert (cost < 0); 6854 if (memcmp (temp_state, curr_state, dfa_state_size) != 0) 6855 cycle_issued_insns++; 6856 asm_p = false; 6857 } 6858 else 6859 asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT 6860 || asm_noperands (PATTERN (insn)) >= 0); 6861 6862 if (targetm.sched.variable_issue) 6863 ls.can_issue_more = 6864 targetm.sched.variable_issue (sched_dump, sched_verbose, 6865 insn, ls.can_issue_more); 6866 /* A naked CLOBBER or USE generates no instruction, so do 6867 not count them against the issue rate. */ 6868 else if (GET_CODE (PATTERN (insn)) != USE 6869 && GET_CODE (PATTERN (insn)) != CLOBBER) 6870 ls.can_issue_more--; 6871 advance = schedule_insn (insn); 6872 6873 if (SHADOW_P (insn)) 6874 ls.shadows_only_p = true; 6875 6876 /* After issuing an asm insn we should start a new cycle. */ 6877 if (advance == 0 && asm_p) 6878 advance = 1; 6879 6880 if (must_backtrack) 6881 break; 6882 6883 if (advance != 0) 6884 break; 6885 6886 ls.first_cycle_insn_p = false; 6887 if (ready.n_ready > 0) 6888 prune_ready_list (temp_state, false, ls.shadows_only_p, 6889 ls.modulo_epilogue); 6890 } 6891 6892 do_backtrack: 6893 if (!must_backtrack) 6894 for (i = 0; i < ready.n_ready; i++) 6895 { 6896 rtx_insn *insn = ready_element (&ready, i); 6897 if (INSN_EXACT_TICK (insn) == clock_var) 6898 { 6899 must_backtrack = true; 6900 clock_var++; 6901 break; 6902 } 6903 } 6904 if (must_backtrack && modulo_ii > 0) 6905 { 6906 if (modulo_backtracks_left == 0) 6907 goto end_schedule; 6908 modulo_backtracks_left--; 6909 } 6910 while (must_backtrack) 6911 { 6912 struct haifa_saved_data *failed; 6913 rtx_insn *failed_insn; 6914 6915 must_backtrack = false; 6916 failed = verify_shadows (); 6917 gcc_assert (failed); 6918 6919 failed_insn = failed->delay_pair->i1; 6920 /* Clear these queues. */ 6921 perform_replacements_new_cycle (); 6922 toggle_cancelled_flags (false); 6923 unschedule_insns_until (failed_insn); 6924 while (failed != backtrack_queue) 6925 free_topmost_backtrack_point (true); 6926 restore_last_backtrack_point (&ls); 6927 if (sched_verbose >= 2) 6928 fprintf (sched_dump, ";;\t\trewind to cycle %d\n", clock_var); 6929 /* Delay by at least a cycle. This could cause additional 6930 backtracking. */ 6931 queue_insn (failed_insn, 1, "backtracked"); 6932 advance = 0; 6933 if (must_backtrack) 6934 continue; 6935 if (ready.n_ready > 0) 6936 goto resume_after_backtrack; 6937 else 6938 { 6939 if (clock_var == 0 && ls.first_cycle_insn_p) 6940 goto end_schedule; 6941 advance = 1; 6942 break; 6943 } 6944 } 6945 ls.first_cycle_insn_p = true; 6946 } 6947 if (ls.modulo_epilogue) 6948 success = true; 6949 end_schedule: 6950 if (!ls.first_cycle_insn_p || advance) 6951 advance_one_cycle (); 6952 perform_replacements_new_cycle (); 6953 if (modulo_ii > 0) 6954 { 6955 /* Once again, debug insn suckiness: they can be on the ready list 6956 even if they have unresolved dependencies. To make our view 6957 of the world consistent, remove such "ready" insns. */ 6958 restart_debug_insn_loop: 6959 for (i = ready.n_ready - 1; i >= 0; i--) 6960 { 6961 rtx_insn *x; 6962 6963 x = ready_element (&ready, i); 6964 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (x)) != NULL 6965 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (x)) != NULL) 6966 { 6967 ready_remove (&ready, i); 6968 goto restart_debug_insn_loop; 6969 } 6970 } 6971 for (i = ready.n_ready - 1; i >= 0; i--) 6972 { 6973 rtx_insn *x; 6974 6975 x = ready_element (&ready, i); 6976 resolve_dependencies (x); 6977 } 6978 for (i = 0; i <= max_insn_queue_index; i++) 6979 { 6980 rtx_insn_list *link; 6981 while ((link = insn_queue[i]) != NULL) 6982 { 6983 rtx_insn *x = link->insn (); 6984 insn_queue[i] = link->next (); 6985 QUEUE_INDEX (x) = QUEUE_NOWHERE; 6986 free_INSN_LIST_node (link); 6987 resolve_dependencies (x); 6988 } 6989 } 6990 } 6991 6992 if (!success) 6993 undo_all_replacements (); 6994 6995 /* Debug info. */ 6996 if (sched_verbose) 6997 { 6998 fprintf (sched_dump, ";;\tReady list (final): "); 6999 debug_ready_list (&ready); 7000 } 7001 7002 if (modulo_ii == 0 && current_sched_info->queue_must_finish_empty) 7003 /* Sanity check -- queue must be empty now. Meaningless if region has 7004 multiple bbs. */ 7005 gcc_assert (!q_size && !ready.n_ready && !ready.n_debug); 7006 else if (modulo_ii == 0) 7007 { 7008 /* We must maintain QUEUE_INDEX between blocks in region. */ 7009 for (i = ready.n_ready - 1; i >= 0; i--) 7010 { 7011 rtx_insn *x; 7012 7013 x = ready_element (&ready, i); 7014 QUEUE_INDEX (x) = QUEUE_NOWHERE; 7015 TODO_SPEC (x) = HARD_DEP; 7016 } 7017 7018 if (q_size) 7019 for (i = 0; i <= max_insn_queue_index; i++) 7020 { 7021 rtx_insn_list *link; 7022 for (link = insn_queue[i]; link; link = link->next ()) 7023 { 7024 rtx_insn *x; 7025 7026 x = link->insn (); 7027 QUEUE_INDEX (x) = QUEUE_NOWHERE; 7028 TODO_SPEC (x) = HARD_DEP; 7029 } 7030 free_INSN_LIST_list (&insn_queue[i]); 7031 } 7032 } 7033 7034 if (sched_pressure == SCHED_PRESSURE_MODEL) 7035 model_end_schedule (); 7036 7037 if (success) 7038 { 7039 commit_schedule (prev_head, tail, target_bb); 7040 if (sched_verbose) 7041 fprintf (sched_dump, ";; total time = %d\n", clock_var); 7042 } 7043 else 7044 last_scheduled_insn = tail; 7045 7046 scheduled_insns.truncate (0); 7047 7048 if (!current_sched_info->queue_must_finish_empty 7049 || haifa_recovery_bb_recently_added_p) 7050 { 7051 /* INSN_TICK (minimum clock tick at which the insn becomes 7052 ready) may be not correct for the insn in the subsequent 7053 blocks of the region. We should use a correct value of 7054 `clock_var' or modify INSN_TICK. It is better to keep 7055 clock_var value equal to 0 at the start of a basic block. 7056 Therefore we modify INSN_TICK here. */ 7057 fix_inter_tick (NEXT_INSN (prev_head), last_scheduled_insn); 7058 } 7059 7060 if (targetm.sched.finish) 7061 { 7062 targetm.sched.finish (sched_dump, sched_verbose); 7063 /* Target might have added some instructions to the scheduled block 7064 in its md_finish () hook. These new insns don't have any data 7065 initialized and to identify them we extend h_i_d so that they'll 7066 get zero luids. */ 7067 sched_extend_luids (); 7068 } 7069 7070 /* Update head/tail boundaries. */ 7071 head = NEXT_INSN (prev_head); 7072 tail = last_scheduled_insn; 7073 7074 if (sched_verbose) 7075 { 7076 fprintf (sched_dump, ";; new head = %d\n;; new tail = %d\n", 7077 INSN_UID (head), INSN_UID (tail)); 7078 7079 if (sched_verbose >= 2) 7080 { 7081 dump_insn_stream (head, tail); 7082 print_rank_for_schedule_stats (";; TOTAL ", &rank_for_schedule_stats, 7083 NULL); 7084 } 7085 7086 fprintf (sched_dump, "\n"); 7087 } 7088 7089 head = restore_other_notes (head, NULL); 7090 7091 current_sched_info->head = head; 7092 current_sched_info->tail = tail; 7093 7094 free_backtrack_queue (); 7095 7096 return success; 7097 } 7098 7099 /* Set_priorities: compute priority of each insn in the block. */ 7100 7101 int 7102 set_priorities (rtx_insn *head, rtx_insn *tail) 7103 { 7104 rtx_insn *insn; 7105 int n_insn; 7106 int sched_max_insns_priority = 7107 current_sched_info->sched_max_insns_priority; 7108 rtx_insn *prev_head; 7109 7110 if (head == tail && ! INSN_P (head)) 7111 gcc_unreachable (); 7112 7113 n_insn = 0; 7114 7115 prev_head = PREV_INSN (head); 7116 for (insn = tail; insn != prev_head; insn = PREV_INSN (insn)) 7117 { 7118 if (!INSN_P (insn)) 7119 continue; 7120 7121 n_insn++; 7122 (void) priority (insn); 7123 7124 gcc_assert (INSN_PRIORITY_KNOWN (insn)); 7125 7126 sched_max_insns_priority = MAX (sched_max_insns_priority, 7127 INSN_PRIORITY (insn)); 7128 } 7129 7130 current_sched_info->sched_max_insns_priority = sched_max_insns_priority; 7131 7132 return n_insn; 7133 } 7134 7135 /* Set sched_dump and sched_verbose for the desired debugging output. */ 7136 void 7137 setup_sched_dump (void) 7138 { 7139 sched_verbose = sched_verbose_param; 7140 sched_dump = dump_file; 7141 if (!dump_file) 7142 sched_verbose = 0; 7143 } 7144 7145 /* Allocate data for register pressure sensitive scheduling. */ 7146 static void 7147 alloc_global_sched_pressure_data (void) 7148 { 7149 if (sched_pressure != SCHED_PRESSURE_NONE) 7150 { 7151 int i, max_regno = max_reg_num (); 7152 7153 if (sched_dump != NULL) 7154 /* We need info about pseudos for rtl dumps about pseudo 7155 classes and costs. */ 7156 regstat_init_n_sets_and_refs (); 7157 ira_set_pseudo_classes (true, sched_verbose ? sched_dump : NULL); 7158 sched_regno_pressure_class 7159 = (enum reg_class *) xmalloc (max_regno * sizeof (enum reg_class)); 7160 for (i = 0; i < max_regno; i++) 7161 sched_regno_pressure_class[i] 7162 = (i < FIRST_PSEUDO_REGISTER 7163 ? ira_pressure_class_translate[REGNO_REG_CLASS (i)] 7164 : ira_pressure_class_translate[reg_allocno_class (i)]); 7165 curr_reg_live = BITMAP_ALLOC (NULL); 7166 if (sched_pressure == SCHED_PRESSURE_WEIGHTED) 7167 { 7168 saved_reg_live = BITMAP_ALLOC (NULL); 7169 region_ref_regs = BITMAP_ALLOC (NULL); 7170 } 7171 if (sched_pressure == SCHED_PRESSURE_MODEL) 7172 tmp_bitmap = BITMAP_ALLOC (NULL); 7173 7174 /* Calculate number of CALL_SAVED_REGS and FIXED_REGS in register classes 7175 that we calculate register pressure for. */ 7176 for (int c = 0; c < ira_pressure_classes_num; ++c) 7177 { 7178 enum reg_class cl = ira_pressure_classes[c]; 7179 7180 call_saved_regs_num[cl] = 0; 7181 fixed_regs_num[cl] = 0; 7182 7183 for (int i = 0; i < ira_class_hard_regs_num[cl]; ++i) 7184 if (!call_used_regs[ira_class_hard_regs[cl][i]]) 7185 ++call_saved_regs_num[cl]; 7186 else if (fixed_regs[ira_class_hard_regs[cl][i]]) 7187 ++fixed_regs_num[cl]; 7188 } 7189 } 7190 } 7191 7192 /* Free data for register pressure sensitive scheduling. Also called 7193 from schedule_region when stopping sched-pressure early. */ 7194 void 7195 free_global_sched_pressure_data (void) 7196 { 7197 if (sched_pressure != SCHED_PRESSURE_NONE) 7198 { 7199 if (regstat_n_sets_and_refs != NULL) 7200 regstat_free_n_sets_and_refs (); 7201 if (sched_pressure == SCHED_PRESSURE_WEIGHTED) 7202 { 7203 BITMAP_FREE (region_ref_regs); 7204 BITMAP_FREE (saved_reg_live); 7205 } 7206 if (sched_pressure == SCHED_PRESSURE_MODEL) 7207 BITMAP_FREE (tmp_bitmap); 7208 BITMAP_FREE (curr_reg_live); 7209 free (sched_regno_pressure_class); 7210 } 7211 } 7212 7213 /* Initialize some global state for the scheduler. This function works 7214 with the common data shared between all the schedulers. It is called 7215 from the scheduler specific initialization routine. */ 7216 7217 void 7218 sched_init (void) 7219 { 7220 /* Disable speculative loads in their presence if cc0 defined. */ 7221 if (HAVE_cc0) 7222 flag_schedule_speculative_load = 0; 7223 7224 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON)) 7225 targetm.sched.dispatch_do (NULL, DISPATCH_INIT); 7226 7227 if (live_range_shrinkage_p) 7228 sched_pressure = SCHED_PRESSURE_WEIGHTED; 7229 else if (flag_sched_pressure 7230 && !reload_completed 7231 && common_sched_info->sched_pass_id == SCHED_RGN_PASS) 7232 sched_pressure = ((enum sched_pressure_algorithm) 7233 PARAM_VALUE (PARAM_SCHED_PRESSURE_ALGORITHM)); 7234 else 7235 sched_pressure = SCHED_PRESSURE_NONE; 7236 7237 if (sched_pressure != SCHED_PRESSURE_NONE) 7238 ira_setup_eliminable_regset (); 7239 7240 /* Initialize SPEC_INFO. */ 7241 if (targetm.sched.set_sched_flags) 7242 { 7243 spec_info = &spec_info_var; 7244 targetm.sched.set_sched_flags (spec_info); 7245 7246 if (spec_info->mask != 0) 7247 { 7248 spec_info->data_weakness_cutoff = 7249 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF) * MAX_DEP_WEAK) / 100; 7250 spec_info->control_weakness_cutoff = 7251 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF) 7252 * REG_BR_PROB_BASE) / 100; 7253 } 7254 else 7255 /* So we won't read anything accidentally. */ 7256 spec_info = NULL; 7257 7258 } 7259 else 7260 /* So we won't read anything accidentally. */ 7261 spec_info = 0; 7262 7263 /* Initialize issue_rate. */ 7264 if (targetm.sched.issue_rate) 7265 issue_rate = targetm.sched.issue_rate (); 7266 else 7267 issue_rate = 1; 7268 7269 if (targetm.sched.first_cycle_multipass_dfa_lookahead 7270 /* Don't use max_issue with reg_pressure scheduling. Multipass 7271 scheduling and reg_pressure scheduling undo each other's decisions. */ 7272 && sched_pressure == SCHED_PRESSURE_NONE) 7273 dfa_lookahead = targetm.sched.first_cycle_multipass_dfa_lookahead (); 7274 else 7275 dfa_lookahead = 0; 7276 7277 /* Set to "0" so that we recalculate. */ 7278 max_lookahead_tries = 0; 7279 7280 if (targetm.sched.init_dfa_pre_cycle_insn) 7281 targetm.sched.init_dfa_pre_cycle_insn (); 7282 7283 if (targetm.sched.init_dfa_post_cycle_insn) 7284 targetm.sched.init_dfa_post_cycle_insn (); 7285 7286 dfa_start (); 7287 dfa_state_size = state_size (); 7288 7289 init_alias_analysis (); 7290 7291 if (!sched_no_dce) 7292 df_set_flags (DF_LR_RUN_DCE); 7293 df_note_add_problem (); 7294 7295 /* More problems needed for interloop dep calculation in SMS. */ 7296 if (common_sched_info->sched_pass_id == SCHED_SMS_PASS) 7297 { 7298 df_rd_add_problem (); 7299 df_chain_add_problem (DF_DU_CHAIN + DF_UD_CHAIN); 7300 } 7301 7302 df_analyze (); 7303 7304 /* Do not run DCE after reload, as this can kill nops inserted 7305 by bundling. */ 7306 if (reload_completed) 7307 df_clear_flags (DF_LR_RUN_DCE); 7308 7309 regstat_compute_calls_crossed (); 7310 7311 if (targetm.sched.init_global) 7312 targetm.sched.init_global (sched_dump, sched_verbose, get_max_uid () + 1); 7313 7314 alloc_global_sched_pressure_data (); 7315 7316 curr_state = xmalloc (dfa_state_size); 7317 } 7318 7319 static void haifa_init_only_bb (basic_block, basic_block); 7320 7321 /* Initialize data structures specific to the Haifa scheduler. */ 7322 void 7323 haifa_sched_init (void) 7324 { 7325 setup_sched_dump (); 7326 sched_init (); 7327 7328 scheduled_insns.create (0); 7329 7330 if (spec_info != NULL) 7331 { 7332 sched_deps_info->use_deps_list = 1; 7333 sched_deps_info->generate_spec_deps = 1; 7334 } 7335 7336 /* Initialize luids, dependency caches, target and h_i_d for the 7337 whole function. */ 7338 { 7339 sched_init_bbs (); 7340 7341 auto_vec<basic_block> bbs (n_basic_blocks_for_fn (cfun)); 7342 basic_block bb; 7343 FOR_EACH_BB_FN (bb, cfun) 7344 bbs.quick_push (bb); 7345 sched_init_luids (bbs); 7346 sched_deps_init (true); 7347 sched_extend_target (); 7348 haifa_init_h_i_d (bbs); 7349 } 7350 7351 sched_init_only_bb = haifa_init_only_bb; 7352 sched_split_block = sched_split_block_1; 7353 sched_create_empty_bb = sched_create_empty_bb_1; 7354 haifa_recovery_bb_ever_added_p = false; 7355 7356 nr_begin_data = nr_begin_control = nr_be_in_data = nr_be_in_control = 0; 7357 before_recovery = 0; 7358 after_recovery = 0; 7359 7360 modulo_ii = 0; 7361 } 7362 7363 /* Finish work with the data specific to the Haifa scheduler. */ 7364 void 7365 haifa_sched_finish (void) 7366 { 7367 sched_create_empty_bb = NULL; 7368 sched_split_block = NULL; 7369 sched_init_only_bb = NULL; 7370 7371 if (spec_info && spec_info->dump) 7372 { 7373 char c = reload_completed ? 'a' : 'b'; 7374 7375 fprintf (spec_info->dump, 7376 ";; %s:\n", current_function_name ()); 7377 7378 fprintf (spec_info->dump, 7379 ";; Procedure %cr-begin-data-spec motions == %d\n", 7380 c, nr_begin_data); 7381 fprintf (spec_info->dump, 7382 ";; Procedure %cr-be-in-data-spec motions == %d\n", 7383 c, nr_be_in_data); 7384 fprintf (spec_info->dump, 7385 ";; Procedure %cr-begin-control-spec motions == %d\n", 7386 c, nr_begin_control); 7387 fprintf (spec_info->dump, 7388 ";; Procedure %cr-be-in-control-spec motions == %d\n", 7389 c, nr_be_in_control); 7390 } 7391 7392 scheduled_insns.release (); 7393 7394 /* Finalize h_i_d, dependency caches, and luids for the whole 7395 function. Target will be finalized in md_global_finish (). */ 7396 sched_deps_finish (); 7397 sched_finish_luids (); 7398 current_sched_info = NULL; 7399 insn_queue = NULL; 7400 sched_finish (); 7401 } 7402 7403 /* Free global data used during insn scheduling. This function works with 7404 the common data shared between the schedulers. */ 7405 7406 void 7407 sched_finish (void) 7408 { 7409 haifa_finish_h_i_d (); 7410 free_global_sched_pressure_data (); 7411 free (curr_state); 7412 7413 if (targetm.sched.finish_global) 7414 targetm.sched.finish_global (sched_dump, sched_verbose); 7415 7416 end_alias_analysis (); 7417 7418 regstat_free_calls_crossed (); 7419 7420 dfa_finish (); 7421 } 7422 7423 /* Free all delay_pair structures that were recorded. */ 7424 void 7425 free_delay_pairs (void) 7426 { 7427 if (delay_htab) 7428 { 7429 delay_htab->empty (); 7430 delay_htab_i2->empty (); 7431 } 7432 } 7433 7434 /* Fix INSN_TICKs of the instructions in the current block as well as 7435 INSN_TICKs of their dependents. 7436 HEAD and TAIL are the begin and the end of the current scheduled block. */ 7437 static void 7438 fix_inter_tick (rtx_insn *head, rtx_insn *tail) 7439 { 7440 /* Set of instructions with corrected INSN_TICK. */ 7441 auto_bitmap processed; 7442 /* ??? It is doubtful if we should assume that cycle advance happens on 7443 basic block boundaries. Basically insns that are unconditionally ready 7444 on the start of the block are more preferable then those which have 7445 a one cycle dependency over insn from the previous block. */ 7446 int next_clock = clock_var + 1; 7447 7448 /* Iterates over scheduled instructions and fix their INSN_TICKs and 7449 INSN_TICKs of dependent instructions, so that INSN_TICKs are consistent 7450 across different blocks. */ 7451 for (tail = NEXT_INSN (tail); head != tail; head = NEXT_INSN (head)) 7452 { 7453 if (INSN_P (head)) 7454 { 7455 int tick; 7456 sd_iterator_def sd_it; 7457 dep_t dep; 7458 7459 tick = INSN_TICK (head); 7460 gcc_assert (tick >= MIN_TICK); 7461 7462 /* Fix INSN_TICK of instruction from just scheduled block. */ 7463 if (bitmap_set_bit (processed, INSN_LUID (head))) 7464 { 7465 tick -= next_clock; 7466 7467 if (tick < MIN_TICK) 7468 tick = MIN_TICK; 7469 7470 INSN_TICK (head) = tick; 7471 } 7472 7473 if (DEBUG_INSN_P (head)) 7474 continue; 7475 7476 FOR_EACH_DEP (head, SD_LIST_RES_FORW, sd_it, dep) 7477 { 7478 rtx_insn *next; 7479 7480 next = DEP_CON (dep); 7481 tick = INSN_TICK (next); 7482 7483 if (tick != INVALID_TICK 7484 /* If NEXT has its INSN_TICK calculated, fix it. 7485 If not - it will be properly calculated from 7486 scratch later in fix_tick_ready. */ 7487 && bitmap_set_bit (processed, INSN_LUID (next))) 7488 { 7489 tick -= next_clock; 7490 7491 if (tick < MIN_TICK) 7492 tick = MIN_TICK; 7493 7494 if (tick > INTER_TICK (next)) 7495 INTER_TICK (next) = tick; 7496 else 7497 tick = INTER_TICK (next); 7498 7499 INSN_TICK (next) = tick; 7500 } 7501 } 7502 } 7503 } 7504 } 7505 7506 /* Check if NEXT is ready to be added to the ready or queue list. 7507 If "yes", add it to the proper list. 7508 Returns: 7509 -1 - is not ready yet, 7510 0 - added to the ready list, 7511 0 < N - queued for N cycles. */ 7512 int 7513 try_ready (rtx_insn *next) 7514 { 7515 ds_t old_ts, new_ts; 7516 7517 old_ts = TODO_SPEC (next); 7518 7519 gcc_assert (!(old_ts & ~(SPECULATIVE | HARD_DEP | DEP_CONTROL | DEP_POSTPONED)) 7520 && (old_ts == HARD_DEP 7521 || old_ts == DEP_POSTPONED 7522 || (old_ts & SPECULATIVE) 7523 || old_ts == DEP_CONTROL)); 7524 7525 new_ts = recompute_todo_spec (next, false); 7526 7527 if (new_ts & (HARD_DEP | DEP_POSTPONED)) 7528 gcc_assert (new_ts == old_ts 7529 && QUEUE_INDEX (next) == QUEUE_NOWHERE); 7530 else if (current_sched_info->new_ready) 7531 new_ts = current_sched_info->new_ready (next, new_ts); 7532 7533 /* * if !(old_ts & SPECULATIVE) (e.g. HARD_DEP or 0), then insn might 7534 have its original pattern or changed (speculative) one. This is due 7535 to changing ebb in region scheduling. 7536 * But if (old_ts & SPECULATIVE), then we are pretty sure that insn 7537 has speculative pattern. 7538 7539 We can't assert (!(new_ts & HARD_DEP) || new_ts == old_ts) here because 7540 control-speculative NEXT could have been discarded by sched-rgn.c 7541 (the same case as when discarded by can_schedule_ready_p ()). */ 7542 7543 if ((new_ts & SPECULATIVE) 7544 /* If (old_ts == new_ts), then (old_ts & SPECULATIVE) and we don't 7545 need to change anything. */ 7546 && new_ts != old_ts) 7547 { 7548 int res; 7549 rtx new_pat; 7550 7551 gcc_assert ((new_ts & SPECULATIVE) && !(new_ts & ~SPECULATIVE)); 7552 7553 res = haifa_speculate_insn (next, new_ts, &new_pat); 7554 7555 switch (res) 7556 { 7557 case -1: 7558 /* It would be nice to change DEP_STATUS of all dependences, 7559 which have ((DEP_STATUS & SPECULATIVE) == new_ts) to HARD_DEP, 7560 so we won't reanalyze anything. */ 7561 new_ts = HARD_DEP; 7562 break; 7563 7564 case 0: 7565 /* We follow the rule, that every speculative insn 7566 has non-null ORIG_PAT. */ 7567 if (!ORIG_PAT (next)) 7568 ORIG_PAT (next) = PATTERN (next); 7569 break; 7570 7571 case 1: 7572 if (!ORIG_PAT (next)) 7573 /* If we gonna to overwrite the original pattern of insn, 7574 save it. */ 7575 ORIG_PAT (next) = PATTERN (next); 7576 7577 res = haifa_change_pattern (next, new_pat); 7578 gcc_assert (res); 7579 break; 7580 7581 default: 7582 gcc_unreachable (); 7583 } 7584 } 7585 7586 /* We need to restore pattern only if (new_ts == 0), because otherwise it is 7587 either correct (new_ts & SPECULATIVE), 7588 or we simply don't care (new_ts & HARD_DEP). */ 7589 7590 gcc_assert (!ORIG_PAT (next) 7591 || !IS_SPECULATION_BRANCHY_CHECK_P (next)); 7592 7593 TODO_SPEC (next) = new_ts; 7594 7595 if (new_ts & (HARD_DEP | DEP_POSTPONED)) 7596 { 7597 /* We can't assert (QUEUE_INDEX (next) == QUEUE_NOWHERE) here because 7598 control-speculative NEXT could have been discarded by sched-rgn.c 7599 (the same case as when discarded by can_schedule_ready_p ()). */ 7600 /*gcc_assert (QUEUE_INDEX (next) == QUEUE_NOWHERE);*/ 7601 7602 change_queue_index (next, QUEUE_NOWHERE); 7603 7604 return -1; 7605 } 7606 else if (!(new_ts & BEGIN_SPEC) 7607 && ORIG_PAT (next) && PREDICATED_PAT (next) == NULL_RTX 7608 && !IS_SPECULATION_CHECK_P (next)) 7609 /* We should change pattern of every previously speculative 7610 instruction - and we determine if NEXT was speculative by using 7611 ORIG_PAT field. Except one case - speculation checks have ORIG_PAT 7612 pat too, so skip them. */ 7613 { 7614 bool success = haifa_change_pattern (next, ORIG_PAT (next)); 7615 gcc_assert (success); 7616 ORIG_PAT (next) = 0; 7617 } 7618 7619 if (sched_verbose >= 2) 7620 { 7621 fprintf (sched_dump, ";;\t\tdependencies resolved: insn %s", 7622 (*current_sched_info->print_insn) (next, 0)); 7623 7624 if (spec_info && spec_info->dump) 7625 { 7626 if (new_ts & BEGIN_DATA) 7627 fprintf (spec_info->dump, "; data-spec;"); 7628 if (new_ts & BEGIN_CONTROL) 7629 fprintf (spec_info->dump, "; control-spec;"); 7630 if (new_ts & BE_IN_CONTROL) 7631 fprintf (spec_info->dump, "; in-control-spec;"); 7632 } 7633 if (TODO_SPEC (next) & DEP_CONTROL) 7634 fprintf (sched_dump, " predicated"); 7635 fprintf (sched_dump, "\n"); 7636 } 7637 7638 adjust_priority (next); 7639 7640 return fix_tick_ready (next); 7641 } 7642 7643 /* Calculate INSN_TICK of NEXT and add it to either ready or queue list. */ 7644 static int 7645 fix_tick_ready (rtx_insn *next) 7646 { 7647 int tick, delay; 7648 7649 if (!DEBUG_INSN_P (next) && !sd_lists_empty_p (next, SD_LIST_RES_BACK)) 7650 { 7651 int full_p; 7652 sd_iterator_def sd_it; 7653 dep_t dep; 7654 7655 tick = INSN_TICK (next); 7656 /* if tick is not equal to INVALID_TICK, then update 7657 INSN_TICK of NEXT with the most recent resolved dependence 7658 cost. Otherwise, recalculate from scratch. */ 7659 full_p = (tick == INVALID_TICK); 7660 7661 FOR_EACH_DEP (next, SD_LIST_RES_BACK, sd_it, dep) 7662 { 7663 rtx_insn *pro = DEP_PRO (dep); 7664 int tick1; 7665 7666 gcc_assert (INSN_TICK (pro) >= MIN_TICK); 7667 7668 tick1 = INSN_TICK (pro) + dep_cost (dep); 7669 if (tick1 > tick) 7670 tick = tick1; 7671 7672 if (!full_p) 7673 break; 7674 } 7675 } 7676 else 7677 tick = -1; 7678 7679 INSN_TICK (next) = tick; 7680 7681 delay = tick - clock_var; 7682 if (delay <= 0 || sched_pressure != SCHED_PRESSURE_NONE || sched_fusion) 7683 delay = QUEUE_READY; 7684 7685 change_queue_index (next, delay); 7686 7687 return delay; 7688 } 7689 7690 /* Move NEXT to the proper queue list with (DELAY >= 1), 7691 or add it to the ready list (DELAY == QUEUE_READY), 7692 or remove it from ready and queue lists at all (DELAY == QUEUE_NOWHERE). */ 7693 static void 7694 change_queue_index (rtx_insn *next, int delay) 7695 { 7696 int i = QUEUE_INDEX (next); 7697 7698 gcc_assert (QUEUE_NOWHERE <= delay && delay <= max_insn_queue_index 7699 && delay != 0); 7700 gcc_assert (i != QUEUE_SCHEDULED); 7701 7702 if ((delay > 0 && NEXT_Q_AFTER (q_ptr, delay) == i) 7703 || (delay < 0 && delay == i)) 7704 /* We have nothing to do. */ 7705 return; 7706 7707 /* Remove NEXT from wherever it is now. */ 7708 if (i == QUEUE_READY) 7709 ready_remove_insn (next); 7710 else if (i >= 0) 7711 queue_remove (next); 7712 7713 /* Add it to the proper place. */ 7714 if (delay == QUEUE_READY) 7715 ready_add (readyp, next, false); 7716 else if (delay >= 1) 7717 queue_insn (next, delay, "change queue index"); 7718 7719 if (sched_verbose >= 2) 7720 { 7721 fprintf (sched_dump, ";;\t\ttick updated: insn %s", 7722 (*current_sched_info->print_insn) (next, 0)); 7723 7724 if (delay == QUEUE_READY) 7725 fprintf (sched_dump, " into ready\n"); 7726 else if (delay >= 1) 7727 fprintf (sched_dump, " into queue with cost=%d\n", delay); 7728 else 7729 fprintf (sched_dump, " removed from ready or queue lists\n"); 7730 } 7731 } 7732 7733 static int sched_ready_n_insns = -1; 7734 7735 /* Initialize per region data structures. */ 7736 void 7737 sched_extend_ready_list (int new_sched_ready_n_insns) 7738 { 7739 int i; 7740 7741 if (sched_ready_n_insns == -1) 7742 /* At the first call we need to initialize one more choice_stack 7743 entry. */ 7744 { 7745 i = 0; 7746 sched_ready_n_insns = 0; 7747 scheduled_insns.reserve (new_sched_ready_n_insns); 7748 } 7749 else 7750 i = sched_ready_n_insns + 1; 7751 7752 ready.veclen = new_sched_ready_n_insns + issue_rate; 7753 ready.vec = XRESIZEVEC (rtx_insn *, ready.vec, ready.veclen); 7754 7755 gcc_assert (new_sched_ready_n_insns >= sched_ready_n_insns); 7756 7757 ready_try = (signed char *) xrecalloc (ready_try, new_sched_ready_n_insns, 7758 sched_ready_n_insns, 7759 sizeof (*ready_try)); 7760 7761 /* We allocate +1 element to save initial state in the choice_stack[0] 7762 entry. */ 7763 choice_stack = XRESIZEVEC (struct choice_entry, choice_stack, 7764 new_sched_ready_n_insns + 1); 7765 7766 for (; i <= new_sched_ready_n_insns; i++) 7767 { 7768 choice_stack[i].state = xmalloc (dfa_state_size); 7769 7770 if (targetm.sched.first_cycle_multipass_init) 7771 targetm.sched.first_cycle_multipass_init (&(choice_stack[i] 7772 .target_data)); 7773 } 7774 7775 sched_ready_n_insns = new_sched_ready_n_insns; 7776 } 7777 7778 /* Free per region data structures. */ 7779 void 7780 sched_finish_ready_list (void) 7781 { 7782 int i; 7783 7784 free (ready.vec); 7785 ready.vec = NULL; 7786 ready.veclen = 0; 7787 7788 free (ready_try); 7789 ready_try = NULL; 7790 7791 for (i = 0; i <= sched_ready_n_insns; i++) 7792 { 7793 if (targetm.sched.first_cycle_multipass_fini) 7794 targetm.sched.first_cycle_multipass_fini (&(choice_stack[i] 7795 .target_data)); 7796 7797 free (choice_stack [i].state); 7798 } 7799 free (choice_stack); 7800 choice_stack = NULL; 7801 7802 sched_ready_n_insns = -1; 7803 } 7804 7805 static int 7806 haifa_luid_for_non_insn (rtx x) 7807 { 7808 gcc_assert (NOTE_P (x) || LABEL_P (x)); 7809 7810 return 0; 7811 } 7812 7813 /* Generates recovery code for INSN. */ 7814 static void 7815 generate_recovery_code (rtx_insn *insn) 7816 { 7817 if (TODO_SPEC (insn) & BEGIN_SPEC) 7818 begin_speculative_block (insn); 7819 7820 /* Here we have insn with no dependencies to 7821 instructions other then CHECK_SPEC ones. */ 7822 7823 if (TODO_SPEC (insn) & BE_IN_SPEC) 7824 add_to_speculative_block (insn); 7825 } 7826 7827 /* Helper function. 7828 Tries to add speculative dependencies of type FS between instructions 7829 in deps_list L and TWIN. */ 7830 static void 7831 process_insn_forw_deps_be_in_spec (rtx_insn *insn, rtx_insn *twin, ds_t fs) 7832 { 7833 sd_iterator_def sd_it; 7834 dep_t dep; 7835 7836 FOR_EACH_DEP (insn, SD_LIST_FORW, sd_it, dep) 7837 { 7838 ds_t ds; 7839 rtx_insn *consumer; 7840 7841 consumer = DEP_CON (dep); 7842 7843 ds = DEP_STATUS (dep); 7844 7845 if (/* If we want to create speculative dep. */ 7846 fs 7847 /* And we can do that because this is a true dep. */ 7848 && (ds & DEP_TYPES) == DEP_TRUE) 7849 { 7850 gcc_assert (!(ds & BE_IN_SPEC)); 7851 7852 if (/* If this dep can be overcome with 'begin speculation'. */ 7853 ds & BEGIN_SPEC) 7854 /* Then we have a choice: keep the dep 'begin speculative' 7855 or transform it into 'be in speculative'. */ 7856 { 7857 if (/* In try_ready we assert that if insn once became ready 7858 it can be removed from the ready (or queue) list only 7859 due to backend decision. Hence we can't let the 7860 probability of the speculative dep to decrease. */ 7861 ds_weak (ds) <= ds_weak (fs)) 7862 { 7863 ds_t new_ds; 7864 7865 new_ds = (ds & ~BEGIN_SPEC) | fs; 7866 7867 if (/* consumer can 'be in speculative'. */ 7868 sched_insn_is_legitimate_for_speculation_p (consumer, 7869 new_ds)) 7870 /* Transform it to be in speculative. */ 7871 ds = new_ds; 7872 } 7873 } 7874 else 7875 /* Mark the dep as 'be in speculative'. */ 7876 ds |= fs; 7877 } 7878 7879 { 7880 dep_def _new_dep, *new_dep = &_new_dep; 7881 7882 init_dep_1 (new_dep, twin, consumer, DEP_TYPE (dep), ds); 7883 sd_add_dep (new_dep, false); 7884 } 7885 } 7886 } 7887 7888 /* Generates recovery code for BEGIN speculative INSN. */ 7889 static void 7890 begin_speculative_block (rtx_insn *insn) 7891 { 7892 if (TODO_SPEC (insn) & BEGIN_DATA) 7893 nr_begin_data++; 7894 if (TODO_SPEC (insn) & BEGIN_CONTROL) 7895 nr_begin_control++; 7896 7897 create_check_block_twin (insn, false); 7898 7899 TODO_SPEC (insn) &= ~BEGIN_SPEC; 7900 } 7901 7902 static void haifa_init_insn (rtx_insn *); 7903 7904 /* Generates recovery code for BE_IN speculative INSN. */ 7905 static void 7906 add_to_speculative_block (rtx_insn *insn) 7907 { 7908 ds_t ts; 7909 sd_iterator_def sd_it; 7910 dep_t dep; 7911 auto_vec<rtx_insn *, 10> twins; 7912 7913 ts = TODO_SPEC (insn); 7914 gcc_assert (!(ts & ~BE_IN_SPEC)); 7915 7916 if (ts & BE_IN_DATA) 7917 nr_be_in_data++; 7918 if (ts & BE_IN_CONTROL) 7919 nr_be_in_control++; 7920 7921 TODO_SPEC (insn) &= ~BE_IN_SPEC; 7922 gcc_assert (!TODO_SPEC (insn)); 7923 7924 DONE_SPEC (insn) |= ts; 7925 7926 /* First we convert all simple checks to branchy. */ 7927 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 7928 sd_iterator_cond (&sd_it, &dep);) 7929 { 7930 rtx_insn *check = DEP_PRO (dep); 7931 7932 if (IS_SPECULATION_SIMPLE_CHECK_P (check)) 7933 { 7934 create_check_block_twin (check, true); 7935 7936 /* Restart search. */ 7937 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 7938 } 7939 else 7940 /* Continue search. */ 7941 sd_iterator_next (&sd_it); 7942 } 7943 7944 auto_vec<rtx_insn *> priorities_roots; 7945 clear_priorities (insn, &priorities_roots); 7946 7947 while (1) 7948 { 7949 rtx_insn *check, *twin; 7950 basic_block rec; 7951 7952 /* Get the first backward dependency of INSN. */ 7953 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 7954 if (!sd_iterator_cond (&sd_it, &dep)) 7955 /* INSN has no backward dependencies left. */ 7956 break; 7957 7958 gcc_assert ((DEP_STATUS (dep) & BEGIN_SPEC) == 0 7959 && (DEP_STATUS (dep) & BE_IN_SPEC) != 0 7960 && (DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE); 7961 7962 check = DEP_PRO (dep); 7963 7964 gcc_assert (!IS_SPECULATION_CHECK_P (check) && !ORIG_PAT (check) 7965 && QUEUE_INDEX (check) == QUEUE_NOWHERE); 7966 7967 rec = BLOCK_FOR_INSN (check); 7968 7969 twin = emit_insn_before (copy_insn (PATTERN (insn)), BB_END (rec)); 7970 haifa_init_insn (twin); 7971 7972 sd_copy_back_deps (twin, insn, true); 7973 7974 if (sched_verbose && spec_info->dump) 7975 /* INSN_BB (insn) isn't determined for twin insns yet. 7976 So we can't use current_sched_info->print_insn. */ 7977 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n", 7978 INSN_UID (twin), rec->index); 7979 7980 twins.safe_push (twin); 7981 7982 /* Add dependences between TWIN and all appropriate 7983 instructions from REC. */ 7984 FOR_EACH_DEP (insn, SD_LIST_SPEC_BACK, sd_it, dep) 7985 { 7986 rtx_insn *pro = DEP_PRO (dep); 7987 7988 gcc_assert (DEP_TYPE (dep) == REG_DEP_TRUE); 7989 7990 /* INSN might have dependencies from the instructions from 7991 several recovery blocks. At this iteration we process those 7992 producers that reside in REC. */ 7993 if (BLOCK_FOR_INSN (pro) == rec) 7994 { 7995 dep_def _new_dep, *new_dep = &_new_dep; 7996 7997 init_dep (new_dep, pro, twin, REG_DEP_TRUE); 7998 sd_add_dep (new_dep, false); 7999 } 8000 } 8001 8002 process_insn_forw_deps_be_in_spec (insn, twin, ts); 8003 8004 /* Remove all dependencies between INSN and insns in REC. */ 8005 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 8006 sd_iterator_cond (&sd_it, &dep);) 8007 { 8008 rtx_insn *pro = DEP_PRO (dep); 8009 8010 if (BLOCK_FOR_INSN (pro) == rec) 8011 sd_delete_dep (sd_it); 8012 else 8013 sd_iterator_next (&sd_it); 8014 } 8015 } 8016 8017 /* We couldn't have added the dependencies between INSN and TWINS earlier 8018 because that would make TWINS appear in the INSN_BACK_DEPS (INSN). */ 8019 unsigned int i; 8020 rtx_insn *twin; 8021 FOR_EACH_VEC_ELT_REVERSE (twins, i, twin) 8022 { 8023 dep_def _new_dep, *new_dep = &_new_dep; 8024 8025 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT); 8026 sd_add_dep (new_dep, false); 8027 } 8028 8029 calc_priorities (priorities_roots); 8030 } 8031 8032 /* Extends and fills with zeros (only the new part) array pointed to by P. */ 8033 void * 8034 xrecalloc (void *p, size_t new_nmemb, size_t old_nmemb, size_t size) 8035 { 8036 gcc_assert (new_nmemb >= old_nmemb); 8037 p = XRESIZEVAR (void, p, new_nmemb * size); 8038 memset (((char *) p) + old_nmemb * size, 0, (new_nmemb - old_nmemb) * size); 8039 return p; 8040 } 8041 8042 /* Helper function. 8043 Find fallthru edge from PRED. */ 8044 edge 8045 find_fallthru_edge_from (basic_block pred) 8046 { 8047 edge e; 8048 basic_block succ; 8049 8050 succ = pred->next_bb; 8051 gcc_assert (succ->prev_bb == pred); 8052 8053 if (EDGE_COUNT (pred->succs) <= EDGE_COUNT (succ->preds)) 8054 { 8055 e = find_fallthru_edge (pred->succs); 8056 8057 if (e) 8058 { 8059 gcc_assert (e->dest == succ); 8060 return e; 8061 } 8062 } 8063 else 8064 { 8065 e = find_fallthru_edge (succ->preds); 8066 8067 if (e) 8068 { 8069 gcc_assert (e->src == pred); 8070 return e; 8071 } 8072 } 8073 8074 return NULL; 8075 } 8076 8077 /* Extend per basic block data structures. */ 8078 static void 8079 sched_extend_bb (void) 8080 { 8081 /* The following is done to keep current_sched_info->next_tail non null. */ 8082 rtx_insn *end = BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb); 8083 rtx_insn *insn = DEBUG_INSN_P (end) ? prev_nondebug_insn (end) : end; 8084 if (NEXT_INSN (end) == 0 8085 || (!NOTE_P (insn) 8086 && !LABEL_P (insn) 8087 /* Don't emit a NOTE if it would end up before a BARRIER. */ 8088 && !BARRIER_P (next_nondebug_insn (end)))) 8089 { 8090 rtx_note *note = emit_note_after (NOTE_INSN_DELETED, end); 8091 /* Make note appear outside BB. */ 8092 set_block_for_insn (note, NULL); 8093 BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb) = end; 8094 } 8095 } 8096 8097 /* Init per basic block data structures. */ 8098 void 8099 sched_init_bbs (void) 8100 { 8101 sched_extend_bb (); 8102 } 8103 8104 /* Initialize BEFORE_RECOVERY variable. */ 8105 static void 8106 init_before_recovery (basic_block *before_recovery_ptr) 8107 { 8108 basic_block last; 8109 edge e; 8110 8111 last = EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb; 8112 e = find_fallthru_edge_from (last); 8113 8114 if (e) 8115 { 8116 /* We create two basic blocks: 8117 1. Single instruction block is inserted right after E->SRC 8118 and has jump to 8119 2. Empty block right before EXIT_BLOCK. 8120 Between these two blocks recovery blocks will be emitted. */ 8121 8122 basic_block single, empty; 8123 8124 /* If the fallthrough edge to exit we've found is from the block we've 8125 created before, don't do anything more. */ 8126 if (last == after_recovery) 8127 return; 8128 8129 adding_bb_to_current_region_p = false; 8130 8131 single = sched_create_empty_bb (last); 8132 empty = sched_create_empty_bb (single); 8133 8134 /* Add new blocks to the root loop. */ 8135 if (current_loops != NULL) 8136 { 8137 add_bb_to_loop (single, (*current_loops->larray)[0]); 8138 add_bb_to_loop (empty, (*current_loops->larray)[0]); 8139 } 8140 8141 single->count = last->count; 8142 empty->count = last->count; 8143 BB_COPY_PARTITION (single, last); 8144 BB_COPY_PARTITION (empty, last); 8145 8146 redirect_edge_succ (e, single); 8147 make_single_succ_edge (single, empty, 0); 8148 make_single_succ_edge (empty, EXIT_BLOCK_PTR_FOR_FN (cfun), 8149 EDGE_FALLTHRU); 8150 8151 rtx_code_label *label = block_label (empty); 8152 rtx_jump_insn *x = emit_jump_insn_after (targetm.gen_jump (label), 8153 BB_END (single)); 8154 JUMP_LABEL (x) = label; 8155 LABEL_NUSES (label)++; 8156 haifa_init_insn (x); 8157 8158 emit_barrier_after (x); 8159 8160 sched_init_only_bb (empty, NULL); 8161 sched_init_only_bb (single, NULL); 8162 sched_extend_bb (); 8163 8164 adding_bb_to_current_region_p = true; 8165 before_recovery = single; 8166 after_recovery = empty; 8167 8168 if (before_recovery_ptr) 8169 *before_recovery_ptr = before_recovery; 8170 8171 if (sched_verbose >= 2 && spec_info->dump) 8172 fprintf (spec_info->dump, 8173 ";;\t\tFixed fallthru to EXIT : %d->>%d->%d->>EXIT\n", 8174 last->index, single->index, empty->index); 8175 } 8176 else 8177 before_recovery = last; 8178 } 8179 8180 /* Returns new recovery block. */ 8181 basic_block 8182 sched_create_recovery_block (basic_block *before_recovery_ptr) 8183 { 8184 rtx_insn *barrier; 8185 basic_block rec; 8186 8187 haifa_recovery_bb_recently_added_p = true; 8188 haifa_recovery_bb_ever_added_p = true; 8189 8190 init_before_recovery (before_recovery_ptr); 8191 8192 barrier = get_last_bb_insn (before_recovery); 8193 gcc_assert (BARRIER_P (barrier)); 8194 8195 rtx_insn *label = emit_label_after (gen_label_rtx (), barrier); 8196 8197 rec = create_basic_block (label, label, before_recovery); 8198 8199 /* A recovery block always ends with an unconditional jump. */ 8200 emit_barrier_after (BB_END (rec)); 8201 8202 if (BB_PARTITION (before_recovery) != BB_UNPARTITIONED) 8203 BB_SET_PARTITION (rec, BB_COLD_PARTITION); 8204 8205 if (sched_verbose && spec_info->dump) 8206 fprintf (spec_info->dump, ";;\t\tGenerated recovery block rec%d\n", 8207 rec->index); 8208 8209 return rec; 8210 } 8211 8212 /* Create edges: FIRST_BB -> REC; FIRST_BB -> SECOND_BB; REC -> SECOND_BB 8213 and emit necessary jumps. */ 8214 void 8215 sched_create_recovery_edges (basic_block first_bb, basic_block rec, 8216 basic_block second_bb) 8217 { 8218 int edge_flags; 8219 8220 /* This is fixing of incoming edge. */ 8221 /* ??? Which other flags should be specified? */ 8222 if (BB_PARTITION (first_bb) != BB_PARTITION (rec)) 8223 /* Partition type is the same, if it is "unpartitioned". */ 8224 edge_flags = EDGE_CROSSING; 8225 else 8226 edge_flags = 0; 8227 8228 edge e2 = single_succ_edge (first_bb); 8229 edge e = make_edge (first_bb, rec, edge_flags); 8230 8231 /* TODO: The actual probability can be determined and is computed as 8232 'todo_spec' variable in create_check_block_twin and 8233 in sel-sched.c `check_ds' in create_speculation_check. */ 8234 e->probability = profile_probability::very_unlikely (); 8235 rec->count = e->count (); 8236 e2->probability = e->probability.invert (); 8237 8238 rtx_code_label *label = block_label (second_bb); 8239 rtx_jump_insn *jump = emit_jump_insn_after (targetm.gen_jump (label), 8240 BB_END (rec)); 8241 JUMP_LABEL (jump) = label; 8242 LABEL_NUSES (label)++; 8243 8244 if (BB_PARTITION (second_bb) != BB_PARTITION (rec)) 8245 /* Partition type is the same, if it is "unpartitioned". */ 8246 { 8247 /* Rewritten from cfgrtl.c. */ 8248 if (crtl->has_bb_partition && targetm_common.have_named_sections) 8249 { 8250 /* We don't need the same note for the check because 8251 any_condjump_p (check) == true. */ 8252 CROSSING_JUMP_P (jump) = 1; 8253 } 8254 edge_flags = EDGE_CROSSING; 8255 } 8256 else 8257 edge_flags = 0; 8258 8259 make_single_succ_edge (rec, second_bb, edge_flags); 8260 if (dom_info_available_p (CDI_DOMINATORS)) 8261 set_immediate_dominator (CDI_DOMINATORS, rec, first_bb); 8262 } 8263 8264 /* This function creates recovery code for INSN. If MUTATE_P is nonzero, 8265 INSN is a simple check, that should be converted to branchy one. */ 8266 static void 8267 create_check_block_twin (rtx_insn *insn, bool mutate_p) 8268 { 8269 basic_block rec; 8270 rtx_insn *label, *check, *twin; 8271 rtx check_pat; 8272 ds_t fs; 8273 sd_iterator_def sd_it; 8274 dep_t dep; 8275 dep_def _new_dep, *new_dep = &_new_dep; 8276 ds_t todo_spec; 8277 8278 gcc_assert (ORIG_PAT (insn) != NULL_RTX); 8279 8280 if (!mutate_p) 8281 todo_spec = TODO_SPEC (insn); 8282 else 8283 { 8284 gcc_assert (IS_SPECULATION_SIMPLE_CHECK_P (insn) 8285 && (TODO_SPEC (insn) & SPECULATIVE) == 0); 8286 8287 todo_spec = CHECK_SPEC (insn); 8288 } 8289 8290 todo_spec &= SPECULATIVE; 8291 8292 /* Create recovery block. */ 8293 if (mutate_p || targetm.sched.needs_block_p (todo_spec)) 8294 { 8295 rec = sched_create_recovery_block (NULL); 8296 label = BB_HEAD (rec); 8297 } 8298 else 8299 { 8300 rec = EXIT_BLOCK_PTR_FOR_FN (cfun); 8301 label = NULL; 8302 } 8303 8304 /* Emit CHECK. */ 8305 check_pat = targetm.sched.gen_spec_check (insn, label, todo_spec); 8306 8307 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8308 { 8309 /* To have mem_reg alive at the beginning of second_bb, 8310 we emit check BEFORE insn, so insn after splitting 8311 insn will be at the beginning of second_bb, which will 8312 provide us with the correct life information. */ 8313 check = emit_jump_insn_before (check_pat, insn); 8314 JUMP_LABEL (check) = label; 8315 LABEL_NUSES (label)++; 8316 } 8317 else 8318 check = emit_insn_before (check_pat, insn); 8319 8320 /* Extend data structures. */ 8321 haifa_init_insn (check); 8322 8323 /* CHECK is being added to current region. Extend ready list. */ 8324 gcc_assert (sched_ready_n_insns != -1); 8325 sched_extend_ready_list (sched_ready_n_insns + 1); 8326 8327 if (current_sched_info->add_remove_insn) 8328 current_sched_info->add_remove_insn (insn, 0); 8329 8330 RECOVERY_BLOCK (check) = rec; 8331 8332 if (sched_verbose && spec_info->dump) 8333 fprintf (spec_info->dump, ";;\t\tGenerated check insn : %s\n", 8334 (*current_sched_info->print_insn) (check, 0)); 8335 8336 gcc_assert (ORIG_PAT (insn)); 8337 8338 /* Initialize TWIN (twin is a duplicate of original instruction 8339 in the recovery block). */ 8340 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8341 { 8342 sd_iterator_def sd_it; 8343 dep_t dep; 8344 8345 FOR_EACH_DEP (insn, SD_LIST_RES_BACK, sd_it, dep) 8346 if ((DEP_STATUS (dep) & DEP_OUTPUT) != 0) 8347 { 8348 struct _dep _dep2, *dep2 = &_dep2; 8349 8350 init_dep (dep2, DEP_PRO (dep), check, REG_DEP_TRUE); 8351 8352 sd_add_dep (dep2, true); 8353 } 8354 8355 twin = emit_insn_after (ORIG_PAT (insn), BB_END (rec)); 8356 haifa_init_insn (twin); 8357 8358 if (sched_verbose && spec_info->dump) 8359 /* INSN_BB (insn) isn't determined for twin insns yet. 8360 So we can't use current_sched_info->print_insn. */ 8361 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n", 8362 INSN_UID (twin), rec->index); 8363 } 8364 else 8365 { 8366 ORIG_PAT (check) = ORIG_PAT (insn); 8367 HAS_INTERNAL_DEP (check) = 1; 8368 twin = check; 8369 /* ??? We probably should change all OUTPUT dependencies to 8370 (TRUE | OUTPUT). */ 8371 } 8372 8373 /* Copy all resolved back dependencies of INSN to TWIN. This will 8374 provide correct value for INSN_TICK (TWIN). */ 8375 sd_copy_back_deps (twin, insn, true); 8376 8377 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8378 /* In case of branchy check, fix CFG. */ 8379 { 8380 basic_block first_bb, second_bb; 8381 rtx_insn *jump; 8382 8383 first_bb = BLOCK_FOR_INSN (check); 8384 second_bb = sched_split_block (first_bb, check); 8385 8386 sched_create_recovery_edges (first_bb, rec, second_bb); 8387 8388 sched_init_only_bb (second_bb, first_bb); 8389 sched_init_only_bb (rec, EXIT_BLOCK_PTR_FOR_FN (cfun)); 8390 8391 jump = BB_END (rec); 8392 haifa_init_insn (jump); 8393 } 8394 8395 /* Move backward dependences from INSN to CHECK and 8396 move forward dependences from INSN to TWIN. */ 8397 8398 /* First, create dependencies between INSN's producers and CHECK & TWIN. */ 8399 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep) 8400 { 8401 rtx_insn *pro = DEP_PRO (dep); 8402 ds_t ds; 8403 8404 /* If BEGIN_DATA: [insn ~~TRUE~~> producer]: 8405 check --TRUE--> producer ??? or ANTI ??? 8406 twin --TRUE--> producer 8407 twin --ANTI--> check 8408 8409 If BEGIN_CONTROL: [insn ~~ANTI~~> producer]: 8410 check --ANTI--> producer 8411 twin --ANTI--> producer 8412 twin --ANTI--> check 8413 8414 If BE_IN_SPEC: [insn ~~TRUE~~> producer]: 8415 check ~~TRUE~~> producer 8416 twin ~~TRUE~~> producer 8417 twin --ANTI--> check */ 8418 8419 ds = DEP_STATUS (dep); 8420 8421 if (ds & BEGIN_SPEC) 8422 { 8423 gcc_assert (!mutate_p); 8424 ds &= ~BEGIN_SPEC; 8425 } 8426 8427 init_dep_1 (new_dep, pro, check, DEP_TYPE (dep), ds); 8428 sd_add_dep (new_dep, false); 8429 8430 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8431 { 8432 DEP_CON (new_dep) = twin; 8433 sd_add_dep (new_dep, false); 8434 } 8435 } 8436 8437 /* Second, remove backward dependencies of INSN. */ 8438 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK); 8439 sd_iterator_cond (&sd_it, &dep);) 8440 { 8441 if ((DEP_STATUS (dep) & BEGIN_SPEC) 8442 || mutate_p) 8443 /* We can delete this dep because we overcome it with 8444 BEGIN_SPECULATION. */ 8445 sd_delete_dep (sd_it); 8446 else 8447 sd_iterator_next (&sd_it); 8448 } 8449 8450 /* Future Speculations. Determine what BE_IN speculations will be like. */ 8451 fs = 0; 8452 8453 /* Fields (DONE_SPEC (x) & BEGIN_SPEC) and CHECK_SPEC (x) are set only 8454 here. */ 8455 8456 gcc_assert (!DONE_SPEC (insn)); 8457 8458 if (!mutate_p) 8459 { 8460 ds_t ts = TODO_SPEC (insn); 8461 8462 DONE_SPEC (insn) = ts & BEGIN_SPEC; 8463 CHECK_SPEC (check) = ts & BEGIN_SPEC; 8464 8465 /* Luckiness of future speculations solely depends upon initial 8466 BEGIN speculation. */ 8467 if (ts & BEGIN_DATA) 8468 fs = set_dep_weak (fs, BE_IN_DATA, get_dep_weak (ts, BEGIN_DATA)); 8469 if (ts & BEGIN_CONTROL) 8470 fs = set_dep_weak (fs, BE_IN_CONTROL, 8471 get_dep_weak (ts, BEGIN_CONTROL)); 8472 } 8473 else 8474 CHECK_SPEC (check) = CHECK_SPEC (insn); 8475 8476 /* Future speculations: call the helper. */ 8477 process_insn_forw_deps_be_in_spec (insn, twin, fs); 8478 8479 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8480 { 8481 /* Which types of dependencies should we use here is, 8482 generally, machine-dependent question... But, for now, 8483 it is not. */ 8484 8485 if (!mutate_p) 8486 { 8487 init_dep (new_dep, insn, check, REG_DEP_TRUE); 8488 sd_add_dep (new_dep, false); 8489 8490 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT); 8491 sd_add_dep (new_dep, false); 8492 } 8493 else 8494 { 8495 if (spec_info->dump) 8496 fprintf (spec_info->dump, ";;\t\tRemoved simple check : %s\n", 8497 (*current_sched_info->print_insn) (insn, 0)); 8498 8499 /* Remove all dependencies of the INSN. */ 8500 { 8501 sd_it = sd_iterator_start (insn, (SD_LIST_FORW 8502 | SD_LIST_BACK 8503 | SD_LIST_RES_BACK)); 8504 while (sd_iterator_cond (&sd_it, &dep)) 8505 sd_delete_dep (sd_it); 8506 } 8507 8508 /* If former check (INSN) already was moved to the ready (or queue) 8509 list, add new check (CHECK) there too. */ 8510 if (QUEUE_INDEX (insn) != QUEUE_NOWHERE) 8511 try_ready (check); 8512 8513 /* Remove old check from instruction stream and free its 8514 data. */ 8515 sched_remove_insn (insn); 8516 } 8517 8518 init_dep (new_dep, check, twin, REG_DEP_ANTI); 8519 sd_add_dep (new_dep, false); 8520 } 8521 else 8522 { 8523 init_dep_1 (new_dep, insn, check, REG_DEP_TRUE, DEP_TRUE | DEP_OUTPUT); 8524 sd_add_dep (new_dep, false); 8525 } 8526 8527 if (!mutate_p) 8528 /* Fix priorities. If MUTATE_P is nonzero, this is not necessary, 8529 because it'll be done later in add_to_speculative_block. */ 8530 { 8531 auto_vec<rtx_insn *> priorities_roots; 8532 8533 clear_priorities (twin, &priorities_roots); 8534 calc_priorities (priorities_roots); 8535 } 8536 } 8537 8538 /* Removes dependency between instructions in the recovery block REC 8539 and usual region instructions. It keeps inner dependences so it 8540 won't be necessary to recompute them. */ 8541 static void 8542 fix_recovery_deps (basic_block rec) 8543 { 8544 rtx_insn *note, *insn, *jump; 8545 auto_vec<rtx_insn *, 10> ready_list; 8546 auto_bitmap in_ready; 8547 8548 /* NOTE - a basic block note. */ 8549 note = NEXT_INSN (BB_HEAD (rec)); 8550 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note)); 8551 insn = BB_END (rec); 8552 gcc_assert (JUMP_P (insn)); 8553 insn = PREV_INSN (insn); 8554 8555 do 8556 { 8557 sd_iterator_def sd_it; 8558 dep_t dep; 8559 8560 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW); 8561 sd_iterator_cond (&sd_it, &dep);) 8562 { 8563 rtx_insn *consumer = DEP_CON (dep); 8564 8565 if (BLOCK_FOR_INSN (consumer) != rec) 8566 { 8567 sd_delete_dep (sd_it); 8568 8569 if (bitmap_set_bit (in_ready, INSN_LUID (consumer))) 8570 ready_list.safe_push (consumer); 8571 } 8572 else 8573 { 8574 gcc_assert ((DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE); 8575 8576 sd_iterator_next (&sd_it); 8577 } 8578 } 8579 8580 insn = PREV_INSN (insn); 8581 } 8582 while (insn != note); 8583 8584 /* Try to add instructions to the ready or queue list. */ 8585 unsigned int i; 8586 rtx_insn *temp; 8587 FOR_EACH_VEC_ELT_REVERSE (ready_list, i, temp) 8588 try_ready (temp); 8589 8590 /* Fixing jump's dependences. */ 8591 insn = BB_HEAD (rec); 8592 jump = BB_END (rec); 8593 8594 gcc_assert (LABEL_P (insn)); 8595 insn = NEXT_INSN (insn); 8596 8597 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (insn)); 8598 add_jump_dependencies (insn, jump); 8599 } 8600 8601 /* Change pattern of INSN to NEW_PAT. Invalidate cached haifa 8602 instruction data. */ 8603 static bool 8604 haifa_change_pattern (rtx_insn *insn, rtx new_pat) 8605 { 8606 int t; 8607 8608 t = validate_change (insn, &PATTERN (insn), new_pat, 0); 8609 if (!t) 8610 return false; 8611 8612 update_insn_after_change (insn); 8613 return true; 8614 } 8615 8616 /* -1 - can't speculate, 8617 0 - for speculation with REQUEST mode it is OK to use 8618 current instruction pattern, 8619 1 - need to change pattern for *NEW_PAT to be speculative. */ 8620 int 8621 sched_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat) 8622 { 8623 gcc_assert (current_sched_info->flags & DO_SPECULATION 8624 && (request & SPECULATIVE) 8625 && sched_insn_is_legitimate_for_speculation_p (insn, request)); 8626 8627 if ((request & spec_info->mask) != request) 8628 return -1; 8629 8630 if (request & BE_IN_SPEC 8631 && !(request & BEGIN_SPEC)) 8632 return 0; 8633 8634 return targetm.sched.speculate_insn (insn, request, new_pat); 8635 } 8636 8637 static int 8638 haifa_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat) 8639 { 8640 gcc_assert (sched_deps_info->generate_spec_deps 8641 && !IS_SPECULATION_CHECK_P (insn)); 8642 8643 if (HAS_INTERNAL_DEP (insn) 8644 || SCHED_GROUP_P (insn)) 8645 return -1; 8646 8647 return sched_speculate_insn (insn, request, new_pat); 8648 } 8649 8650 /* Print some information about block BB, which starts with HEAD and 8651 ends with TAIL, before scheduling it. 8652 I is zero, if scheduler is about to start with the fresh ebb. */ 8653 static void 8654 dump_new_block_header (int i, basic_block bb, rtx_insn *head, rtx_insn *tail) 8655 { 8656 if (!i) 8657 fprintf (sched_dump, 8658 ";; ======================================================\n"); 8659 else 8660 fprintf (sched_dump, 8661 ";; =====================ADVANCING TO=====================\n"); 8662 fprintf (sched_dump, 8663 ";; -- basic block %d from %d to %d -- %s reload\n", 8664 bb->index, INSN_UID (head), INSN_UID (tail), 8665 (reload_completed ? "after" : "before")); 8666 fprintf (sched_dump, 8667 ";; ======================================================\n"); 8668 fprintf (sched_dump, "\n"); 8669 } 8670 8671 /* Unlink basic block notes and labels and saves them, so they 8672 can be easily restored. We unlink basic block notes in EBB to 8673 provide back-compatibility with the previous code, as target backends 8674 assume, that there'll be only instructions between 8675 current_sched_info->{head and tail}. We restore these notes as soon 8676 as we can. 8677 FIRST (LAST) is the first (last) basic block in the ebb. 8678 NB: In usual case (FIRST == LAST) nothing is really done. */ 8679 void 8680 unlink_bb_notes (basic_block first, basic_block last) 8681 { 8682 /* We DON'T unlink basic block notes of the first block in the ebb. */ 8683 if (first == last) 8684 return; 8685 8686 bb_header = XNEWVEC (rtx_insn *, last_basic_block_for_fn (cfun)); 8687 8688 /* Make a sentinel. */ 8689 if (last->next_bb != EXIT_BLOCK_PTR_FOR_FN (cfun)) 8690 bb_header[last->next_bb->index] = 0; 8691 8692 first = first->next_bb; 8693 do 8694 { 8695 rtx_insn *prev, *label, *note, *next; 8696 8697 label = BB_HEAD (last); 8698 if (LABEL_P (label)) 8699 note = NEXT_INSN (label); 8700 else 8701 note = label; 8702 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note)); 8703 8704 prev = PREV_INSN (label); 8705 next = NEXT_INSN (note); 8706 gcc_assert (prev && next); 8707 8708 SET_NEXT_INSN (prev) = next; 8709 SET_PREV_INSN (next) = prev; 8710 8711 bb_header[last->index] = label; 8712 8713 if (last == first) 8714 break; 8715 8716 last = last->prev_bb; 8717 } 8718 while (1); 8719 } 8720 8721 /* Restore basic block notes. 8722 FIRST is the first basic block in the ebb. */ 8723 static void 8724 restore_bb_notes (basic_block first) 8725 { 8726 if (!bb_header) 8727 return; 8728 8729 /* We DON'T unlink basic block notes of the first block in the ebb. */ 8730 first = first->next_bb; 8731 /* Remember: FIRST is actually a second basic block in the ebb. */ 8732 8733 while (first != EXIT_BLOCK_PTR_FOR_FN (cfun) 8734 && bb_header[first->index]) 8735 { 8736 rtx_insn *prev, *label, *note, *next; 8737 8738 label = bb_header[first->index]; 8739 prev = PREV_INSN (label); 8740 next = NEXT_INSN (prev); 8741 8742 if (LABEL_P (label)) 8743 note = NEXT_INSN (label); 8744 else 8745 note = label; 8746 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note)); 8747 8748 bb_header[first->index] = 0; 8749 8750 SET_NEXT_INSN (prev) = label; 8751 SET_NEXT_INSN (note) = next; 8752 SET_PREV_INSN (next) = note; 8753 8754 first = first->next_bb; 8755 } 8756 8757 free (bb_header); 8758 bb_header = 0; 8759 } 8760 8761 /* Helper function. 8762 Fix CFG after both in- and inter-block movement of 8763 control_flow_insn_p JUMP. */ 8764 static void 8765 fix_jump_move (rtx_insn *jump) 8766 { 8767 basic_block bb, jump_bb, jump_bb_next; 8768 8769 bb = BLOCK_FOR_INSN (PREV_INSN (jump)); 8770 jump_bb = BLOCK_FOR_INSN (jump); 8771 jump_bb_next = jump_bb->next_bb; 8772 8773 gcc_assert (common_sched_info->sched_pass_id == SCHED_EBB_PASS 8774 || IS_SPECULATION_BRANCHY_CHECK_P (jump)); 8775 8776 if (!NOTE_INSN_BASIC_BLOCK_P (BB_END (jump_bb_next))) 8777 /* if jump_bb_next is not empty. */ 8778 BB_END (jump_bb) = BB_END (jump_bb_next); 8779 8780 if (BB_END (bb) != PREV_INSN (jump)) 8781 /* Then there are instruction after jump that should be placed 8782 to jump_bb_next. */ 8783 BB_END (jump_bb_next) = BB_END (bb); 8784 else 8785 /* Otherwise jump_bb_next is empty. */ 8786 BB_END (jump_bb_next) = NEXT_INSN (BB_HEAD (jump_bb_next)); 8787 8788 /* To make assertion in move_insn happy. */ 8789 BB_END (bb) = PREV_INSN (jump); 8790 8791 update_bb_for_insn (jump_bb_next); 8792 } 8793 8794 /* Fix CFG after interblock movement of control_flow_insn_p JUMP. */ 8795 static void 8796 move_block_after_check (rtx_insn *jump) 8797 { 8798 basic_block bb, jump_bb, jump_bb_next; 8799 vec<edge, va_gc> *t; 8800 8801 bb = BLOCK_FOR_INSN (PREV_INSN (jump)); 8802 jump_bb = BLOCK_FOR_INSN (jump); 8803 jump_bb_next = jump_bb->next_bb; 8804 8805 update_bb_for_insn (jump_bb); 8806 8807 gcc_assert (IS_SPECULATION_CHECK_P (jump) 8808 || IS_SPECULATION_CHECK_P (BB_END (jump_bb_next))); 8809 8810 unlink_block (jump_bb_next); 8811 link_block (jump_bb_next, bb); 8812 8813 t = bb->succs; 8814 bb->succs = 0; 8815 move_succs (&(jump_bb->succs), bb); 8816 move_succs (&(jump_bb_next->succs), jump_bb); 8817 move_succs (&t, jump_bb_next); 8818 8819 df_mark_solutions_dirty (); 8820 8821 common_sched_info->fix_recovery_cfg 8822 (bb->index, jump_bb->index, jump_bb_next->index); 8823 } 8824 8825 /* Helper function for move_block_after_check. 8826 This functions attaches edge vector pointed to by SUCCSP to 8827 block TO. */ 8828 static void 8829 move_succs (vec<edge, va_gc> **succsp, basic_block to) 8830 { 8831 edge e; 8832 edge_iterator ei; 8833 8834 gcc_assert (to->succs == 0); 8835 8836 to->succs = *succsp; 8837 8838 FOR_EACH_EDGE (e, ei, to->succs) 8839 e->src = to; 8840 8841 *succsp = 0; 8842 } 8843 8844 /* Remove INSN from the instruction stream. 8845 INSN should have any dependencies. */ 8846 static void 8847 sched_remove_insn (rtx_insn *insn) 8848 { 8849 sd_finish_insn (insn); 8850 8851 change_queue_index (insn, QUEUE_NOWHERE); 8852 current_sched_info->add_remove_insn (insn, 1); 8853 delete_insn (insn); 8854 } 8855 8856 /* Clear priorities of all instructions, that are forward dependent on INSN. 8857 Store in vector pointed to by ROOTS_PTR insns on which priority () should 8858 be invoked to initialize all cleared priorities. */ 8859 static void 8860 clear_priorities (rtx_insn *insn, rtx_vec_t *roots_ptr) 8861 { 8862 sd_iterator_def sd_it; 8863 dep_t dep; 8864 bool insn_is_root_p = true; 8865 8866 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED); 8867 8868 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep) 8869 { 8870 rtx_insn *pro = DEP_PRO (dep); 8871 8872 if (INSN_PRIORITY_STATUS (pro) >= 0 8873 && QUEUE_INDEX (insn) != QUEUE_SCHEDULED) 8874 { 8875 /* If DEP doesn't contribute to priority then INSN itself should 8876 be added to priority roots. */ 8877 if (contributes_to_priority_p (dep)) 8878 insn_is_root_p = false; 8879 8880 INSN_PRIORITY_STATUS (pro) = -1; 8881 clear_priorities (pro, roots_ptr); 8882 } 8883 } 8884 8885 if (insn_is_root_p) 8886 roots_ptr->safe_push (insn); 8887 } 8888 8889 /* Recompute priorities of instructions, whose priorities might have been 8890 changed. ROOTS is a vector of instructions whose priority computation will 8891 trigger initialization of all cleared priorities. */ 8892 static void 8893 calc_priorities (rtx_vec_t roots) 8894 { 8895 int i; 8896 rtx_insn *insn; 8897 8898 FOR_EACH_VEC_ELT (roots, i, insn) 8899 priority (insn); 8900 } 8901 8902 8903 /* Add dependences between JUMP and other instructions in the recovery 8904 block. INSN is the first insn the recovery block. */ 8905 static void 8906 add_jump_dependencies (rtx_insn *insn, rtx_insn *jump) 8907 { 8908 do 8909 { 8910 insn = NEXT_INSN (insn); 8911 if (insn == jump) 8912 break; 8913 8914 if (dep_list_size (insn, SD_LIST_FORW) == 0) 8915 { 8916 dep_def _new_dep, *new_dep = &_new_dep; 8917 8918 init_dep (new_dep, insn, jump, REG_DEP_ANTI); 8919 sd_add_dep (new_dep, false); 8920 } 8921 } 8922 while (1); 8923 8924 gcc_assert (!sd_lists_empty_p (jump, SD_LIST_BACK)); 8925 } 8926 8927 /* Extend data structures for logical insn UID. */ 8928 void 8929 sched_extend_luids (void) 8930 { 8931 int new_luids_max_uid = get_max_uid () + 1; 8932 8933 sched_luids.safe_grow_cleared (new_luids_max_uid); 8934 } 8935 8936 /* Initialize LUID for INSN. */ 8937 void 8938 sched_init_insn_luid (rtx_insn *insn) 8939 { 8940 int i = INSN_P (insn) ? 1 : common_sched_info->luid_for_non_insn (insn); 8941 int luid; 8942 8943 if (i >= 0) 8944 { 8945 luid = sched_max_luid; 8946 sched_max_luid += i; 8947 } 8948 else 8949 luid = -1; 8950 8951 SET_INSN_LUID (insn, luid); 8952 } 8953 8954 /* Initialize luids for BBS. 8955 The hook common_sched_info->luid_for_non_insn () is used to determine 8956 if notes, labels, etc. need luids. */ 8957 void 8958 sched_init_luids (bb_vec_t bbs) 8959 { 8960 int i; 8961 basic_block bb; 8962 8963 sched_extend_luids (); 8964 FOR_EACH_VEC_ELT (bbs, i, bb) 8965 { 8966 rtx_insn *insn; 8967 8968 FOR_BB_INSNS (bb, insn) 8969 sched_init_insn_luid (insn); 8970 } 8971 } 8972 8973 /* Free LUIDs. */ 8974 void 8975 sched_finish_luids (void) 8976 { 8977 sched_luids.release (); 8978 sched_max_luid = 1; 8979 } 8980 8981 /* Return logical uid of INSN. Helpful while debugging. */ 8982 int 8983 insn_luid (rtx_insn *insn) 8984 { 8985 return INSN_LUID (insn); 8986 } 8987 8988 /* Extend per insn data in the target. */ 8989 void 8990 sched_extend_target (void) 8991 { 8992 if (targetm.sched.h_i_d_extended) 8993 targetm.sched.h_i_d_extended (); 8994 } 8995 8996 /* Extend global scheduler structures (those, that live across calls to 8997 schedule_block) to include information about just emitted INSN. */ 8998 static void 8999 extend_h_i_d (void) 9000 { 9001 int reserve = (get_max_uid () + 1 - h_i_d.length ()); 9002 if (reserve > 0 9003 && ! h_i_d.space (reserve)) 9004 { 9005 h_i_d.safe_grow_cleared (3 * get_max_uid () / 2); 9006 sched_extend_target (); 9007 } 9008 } 9009 9010 /* Initialize h_i_d entry of the INSN with default values. 9011 Values, that are not explicitly initialized here, hold zero. */ 9012 static void 9013 init_h_i_d (rtx_insn *insn) 9014 { 9015 if (INSN_LUID (insn) > 0) 9016 { 9017 INSN_COST (insn) = -1; 9018 QUEUE_INDEX (insn) = QUEUE_NOWHERE; 9019 INSN_TICK (insn) = INVALID_TICK; 9020 INSN_EXACT_TICK (insn) = INVALID_TICK; 9021 INTER_TICK (insn) = INVALID_TICK; 9022 TODO_SPEC (insn) = HARD_DEP; 9023 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status 9024 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED; 9025 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status 9026 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED; 9027 } 9028 } 9029 9030 /* Initialize haifa_insn_data for BBS. */ 9031 void 9032 haifa_init_h_i_d (bb_vec_t bbs) 9033 { 9034 int i; 9035 basic_block bb; 9036 9037 extend_h_i_d (); 9038 FOR_EACH_VEC_ELT (bbs, i, bb) 9039 { 9040 rtx_insn *insn; 9041 9042 FOR_BB_INSNS (bb, insn) 9043 init_h_i_d (insn); 9044 } 9045 } 9046 9047 /* Finalize haifa_insn_data. */ 9048 void 9049 haifa_finish_h_i_d (void) 9050 { 9051 int i; 9052 haifa_insn_data_t data; 9053 reg_use_data *use, *next_use; 9054 reg_set_data *set, *next_set; 9055 9056 FOR_EACH_VEC_ELT (h_i_d, i, data) 9057 { 9058 free (data->max_reg_pressure); 9059 free (data->reg_pressure); 9060 for (use = data->reg_use_list; use != NULL; use = next_use) 9061 { 9062 next_use = use->next_insn_use; 9063 free (use); 9064 } 9065 for (set = data->reg_set_list; set != NULL; set = next_set) 9066 { 9067 next_set = set->next_insn_set; 9068 free (set); 9069 } 9070 9071 } 9072 h_i_d.release (); 9073 } 9074 9075 /* Init data for the new insn INSN. */ 9076 static void 9077 haifa_init_insn (rtx_insn *insn) 9078 { 9079 gcc_assert (insn != NULL); 9080 9081 sched_extend_luids (); 9082 sched_init_insn_luid (insn); 9083 sched_extend_target (); 9084 sched_deps_init (false); 9085 extend_h_i_d (); 9086 init_h_i_d (insn); 9087 9088 if (adding_bb_to_current_region_p) 9089 { 9090 sd_init_insn (insn); 9091 9092 /* Extend dependency caches by one element. */ 9093 extend_dependency_caches (1, false); 9094 } 9095 if (sched_pressure != SCHED_PRESSURE_NONE) 9096 init_insn_reg_pressure_info (insn); 9097 } 9098 9099 /* Init data for the new basic block BB which comes after AFTER. */ 9100 static void 9101 haifa_init_only_bb (basic_block bb, basic_block after) 9102 { 9103 gcc_assert (bb != NULL); 9104 9105 sched_init_bbs (); 9106 9107 if (common_sched_info->add_block) 9108 /* This changes only data structures of the front-end. */ 9109 common_sched_info->add_block (bb, after); 9110 } 9111 9112 /* A generic version of sched_split_block (). */ 9113 basic_block 9114 sched_split_block_1 (basic_block first_bb, rtx after) 9115 { 9116 edge e; 9117 9118 e = split_block (first_bb, after); 9119 gcc_assert (e->src == first_bb); 9120 9121 /* sched_split_block emits note if *check == BB_END. Probably it 9122 is better to rip that note off. */ 9123 9124 return e->dest; 9125 } 9126 9127 /* A generic version of sched_create_empty_bb (). */ 9128 basic_block 9129 sched_create_empty_bb_1 (basic_block after) 9130 { 9131 return create_empty_bb (after); 9132 } 9133 9134 /* Insert PAT as an INSN into the schedule and update the necessary data 9135 structures to account for it. */ 9136 rtx_insn * 9137 sched_emit_insn (rtx pat) 9138 { 9139 rtx_insn *insn = emit_insn_before (pat, first_nonscheduled_insn ()); 9140 haifa_init_insn (insn); 9141 9142 if (current_sched_info->add_remove_insn) 9143 current_sched_info->add_remove_insn (insn, 0); 9144 9145 (*current_sched_info->begin_schedule_ready) (insn); 9146 scheduled_insns.safe_push (insn); 9147 9148 last_scheduled_insn = insn; 9149 return insn; 9150 } 9151 9152 /* This function returns a candidate satisfying dispatch constraints from 9153 the ready list. */ 9154 9155 static rtx_insn * 9156 ready_remove_first_dispatch (struct ready_list *ready) 9157 { 9158 int i; 9159 rtx_insn *insn = ready_element (ready, 0); 9160 9161 if (ready->n_ready == 1 9162 || !INSN_P (insn) 9163 || INSN_CODE (insn) < 0 9164 || !active_insn_p (insn) 9165 || targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW)) 9166 return ready_remove_first (ready); 9167 9168 for (i = 1; i < ready->n_ready; i++) 9169 { 9170 insn = ready_element (ready, i); 9171 9172 if (!INSN_P (insn) 9173 || INSN_CODE (insn) < 0 9174 || !active_insn_p (insn)) 9175 continue; 9176 9177 if (targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW)) 9178 { 9179 /* Return ith element of ready. */ 9180 insn = ready_remove (ready, i); 9181 return insn; 9182 } 9183 } 9184 9185 if (targetm.sched.dispatch (NULL, DISPATCH_VIOLATION)) 9186 return ready_remove_first (ready); 9187 9188 for (i = 1; i < ready->n_ready; i++) 9189 { 9190 insn = ready_element (ready, i); 9191 9192 if (!INSN_P (insn) 9193 || INSN_CODE (insn) < 0 9194 || !active_insn_p (insn)) 9195 continue; 9196 9197 /* Return i-th element of ready. */ 9198 if (targetm.sched.dispatch (insn, IS_CMP)) 9199 return ready_remove (ready, i); 9200 } 9201 9202 return ready_remove_first (ready); 9203 } 9204 9205 /* Get number of ready insn in the ready list. */ 9206 9207 int 9208 number_in_ready (void) 9209 { 9210 return ready.n_ready; 9211 } 9212 9213 /* Get number of ready's in the ready list. */ 9214 9215 rtx_insn * 9216 get_ready_element (int i) 9217 { 9218 return ready_element (&ready, i); 9219 } 9220 9221 #endif /* INSN_SCHEDULING */ 9222