1 /// 2 /// Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. 3 /// Use of this file is governed by the BSD 3-clause license that 4 /// can be found in the LICENSE.txt file in the project root. 5 /// 6 7 8 9 /// 10 /// The embodiment of the adaptive LL(*), ALL(*), parsing strategy. 11 /// 12 /// 13 /// The basic complexity of the adaptive strategy makes it harder to understand. 14 /// We begin with ATN simulation to build paths in a DFA. Subsequent prediction 15 /// requests go through the DFA first. If they reach a state without an edge for 16 /// the current symbol, the algorithm fails over to the ATN simulation to 17 /// complete the DFA path for the current input (until it finds a conflict state 18 /// or uniquely predicting state). 19 /// 20 /// 21 /// All of that is done without using the outer context because we want to create 22 /// a DFA that is not dependent upon the rule invocation stack when we do a 23 /// prediction. One DFA works in all contexts. We avoid using context not 24 /// necessarily because it's slower, although it can be, but because of the DFA 25 /// caching problem. The closure routine only considers the rule invocation stack 26 /// created during prediction beginning in the decision rule. For example, if 27 /// prediction occurs without invoking another rule's ATN, there are no context 28 /// stacks in the configurations. When lack of context leads to a conflict, we 29 /// don't know if it's an ambiguity or a weakness in the strong LL(*) parsing 30 /// strategy (versus full LL(*)). 31 /// 32 /// 33 /// When SLL yields a configuration set with conflict, we rewind the input and 34 /// retry the ATN simulation, this time using full outer context without adding 35 /// to the DFA. Configuration context stacks will be the full invocation stacks 36 /// from the start rule. If we get a conflict using full context, then we can 37 /// definitively say we have a true ambiguity for that input sequence. If we 38 /// don't get a conflict, it implies that the decision is sensitive to the outer 39 /// context. (It is not context-sensitive in the sense of context-sensitive 40 /// grammars.) 41 /// 42 /// 43 /// The next time we reach this DFA state with an SLL conflict, through DFA 44 /// simulation, we will again retry the ATN simulation using full context mode. 45 /// This is slow because we can't save the results and have to "interpret" the 46 /// ATN each time we get that input. 47 /// 48 /// 49 /// __CACHING FULL CONTEXT PREDICTIONS__ 50 /// 51 /// 52 /// We could cache results from full context to predicted alternative easily and 53 /// that saves a lot of time but doesn't work in presence of predicates. The set 54 /// of visible predicates from the ATN start state changes depending on the 55 /// context, because closure can fall off the end of a rule. I tried to cache 56 /// tuples (stack context, semantic context, predicted alt) but it was slower 57 /// than interpreting and much more complicated. Also required a huge amount of 58 /// memory. The goal is not to create the world's fastest parser anyway. I'd like 59 /// to keep this algorithm simple. By launching multiple threads, we can improve 60 /// the speed of parsing across a large number of files. 61 /// 62 /// 63 /// There is no strict ordering between the amount of input used by SLL vs LL, 64 /// which makes it really hard to build a cache for full context. Let's say that 65 /// we have input A B C that leads to an SLL conflict with full context X. That 66 /// implies that using X we might only use A B but we could also use A B C D to 67 /// resolve conflict. Input A B C D could predict alternative 1 in one position 68 /// in the input and A B C E could predict alternative 2 in another position in 69 /// input. The conflicting SLL configurations could still be non-unique in the 70 /// full context prediction, which would lead us to requiring more input than the 71 /// original A B C. To make a prediction cache work, we have to track the exact 72 /// input used during the previous prediction. That amounts to a cache that maps 73 /// X to a specific DFA for that context. 74 /// 75 /// 76 /// Something should be done for left-recursive expression predictions. They are 77 /// likely LL(1) + pred eval. Easier to do the whole SLL unless error and retry 78 /// with full LL thing Sam does. 79 /// 80 /// 81 /// __AVOIDING FULL CONTEXT PREDICTION__ 82 /// 83 /// 84 /// We avoid doing full context retry when the outer context is empty, we did not 85 /// dip into the outer context by falling off the end of the decision state rule, 86 /// or when we force SLL mode. 87 /// 88 /// 89 /// As an example of the not dip into outer context case, consider as super 90 /// constructor calls versus function calls. One grammar might look like 91 /// this: 92 /// 93 /// 94 /// ctorBody 95 /// : '{' superCall? stat* '}' 96 /// ; 97 /// 98 /// 99 /// 100 /// Or, you might see something like 101 /// 102 /// 103 /// stat 104 /// : superCall ';' 105 /// | expression ';' 106 /// | ... 107 /// ; 108 /// 109 /// 110 /// 111 /// In both cases I believe that no closure operations will dip into the outer 112 /// context. In the first case ctorBody in the worst case will stop at the '}'. 113 /// In the 2nd case it should stop at the ';'. Both cases should stay within the 114 /// entry rule and not dip into the outer context. 115 /// 116 /// 117 /// __PREDICATES__ 118 /// 119 /// 120 /// Predicates are always evaluated if present in either SLL or LL both. SLL and 121 /// LL simulation deals with predicates differently. SLL collects predicates as 122 /// it performs closure operations like ANTLR v3 did. It delays predicate 123 /// evaluation until it reaches and accept state. This allows us to cache the SLL 124 /// ATN simulation whereas, if we had evaluated predicates on-the-fly during 125 /// closure, the DFA state configuration sets would be different and we couldn't 126 /// build up a suitable DFA. 127 /// 128 /// 129 /// When building a DFA accept state during ATN simulation, we evaluate any 130 /// predicates and return the sole semantically valid alternative. If there is 131 /// more than 1 alternative, we report an ambiguity. If there are 0 alternatives, 132 /// we throw an exception. Alternatives without predicates act like they have 133 /// true predicates. The simple way to think about it is to strip away all 134 /// alternatives with false predicates and choose the minimum alternative that 135 /// remains. 136 /// 137 /// 138 /// When we start in the DFA and reach an accept state that's predicated, we test 139 /// those and return the minimum semantically viable alternative. If no 140 /// alternatives are viable, we throw an exception. 141 /// 142 /// 143 /// During full LL ATN simulation, closure always evaluates predicates and 144 /// on-the-fly. This is crucial to reducing the configuration set size during 145 /// closure. It hits a landmine when parsing with the Java grammar, for example, 146 /// without this on-the-fly evaluation. 147 /// 148 /// 149 /// __SHARING DFA__ 150 /// 151 /// 152 /// All instances of the same parser share the same decision DFAs through a 153 /// static field. Each instance gets its own ATN simulator but they share the 154 /// same _#decisionToDFA_ field. They also share a 155 /// _org.antlr.v4.runtime.atn.PredictionContextCache_ object that makes sure that all 156 /// _org.antlr.v4.runtime.atn.PredictionContext_ objects are shared among the DFA states. This makes 157 /// a big size difference. 158 /// 159 /// 160 /// __THREAD SAFETY__ 161 /// 162 /// 163 /// The _org.antlr.v4.runtime.atn.ParserATNSimulator_ locks on the _#decisionToDFA_ field when 164 /// it adds a new DFA object to that array. _#addDFAEdge_ 165 /// locks on the DFA for the current decision when setting the 166 /// _org.antlr.v4.runtime.dfa.DFAState#edges_ field. _#addDFAState_ locks on 167 /// the DFA for the current decision when looking up a DFA state to see if it 168 /// already exists. We must make sure that all requests to add DFA states that 169 /// are equivalent result in the same shared DFA object. This is because lots of 170 /// threads will be trying to update the DFA at once. The 171 /// _#addDFAState_ method also locks inside the DFA lock 172 /// but this time on the shared context cache when it rebuilds the 173 /// configurations' _org.antlr.v4.runtime.atn.PredictionContext_ objects using cached 174 /// subgraphs/nodes. No other locking occurs, even during DFA simulation. This is 175 /// safe as long as we can guarantee that all threads referencing 176 /// `s.edge[t]` get the same physical target _org.antlr.v4.runtime.dfa.DFAState_, or 177 /// `null`. Once into the DFA, the DFA simulation does not reference the 178 /// _org.antlr.v4.runtime.dfa.DFA#states_ map. It follows the _org.antlr.v4.runtime.dfa.DFAState#edges_ field to new 179 /// targets. The DFA simulator will either find _org.antlr.v4.runtime.dfa.DFAState#edges_ to be 180 /// `null`, to be non-`null` and `dfa.edges[t]` null, or 181 /// `dfa.edges[t]` to be non-null. The 182 /// _#addDFAEdge_ method could be racing to set the field 183 /// but in either case the DFA simulator works; if `null`, and requests ATN 184 /// simulation. It could also race trying to get `dfa.edges[t]`, but either 185 /// way it will work because it's not doing a test and set operation. 186 /// 187 /// 188 /// __Starting with SLL then failing to combined SLL/LL (Two-Stage 189 /// Parsing)__ 190 /// 191 /// 192 /// Sam pointed out that if SLL does not give a syntax error, then there is no 193 /// point in doing full LL, which is slower. We only have to try LL if we get a 194 /// syntax error. For maximum speed, Sam starts the parser set to pure SLL 195 /// mode with the _org.antlr.v4.runtime.BailErrorStrategy_: 196 /// 197 /// 198 /// parser._org.antlr.v4.runtime.Parser#getInterpreter() getInterpreter()_._#setPredictionMode setPredictionMode_`(`_PredictionMode#SLL_`)`; 199 /// parser._org.antlr.v4.runtime.Parser#setErrorHandler setErrorHandler_(new _org.antlr.v4.runtime.BailErrorStrategy_()); 200 /// 201 /// 202 /// 203 /// If it does not get a syntax error, then we're done. If it does get a syntax 204 /// error, we need to retry with the combined SLL/LL strategy. 205 /// 206 /// 207 /// The reason this works is as follows. If there are no SLL conflicts, then the 208 /// grammar is SLL (at least for that input set). If there is an SLL conflict, 209 /// the full LL analysis must yield a set of viable alternatives which is a 210 /// subset of the alternatives reported by SLL. If the LL set is a singleton, 211 /// then the grammar is LL but not SLL. If the LL set is the same size as the SLL 212 /// set, the decision is SLL. If the LL set has size > 1, then that decision 213 /// is truly ambiguous on the current input. If the LL set is smaller, then the 214 /// SLL conflict resolution might choose an alternative that the full LL would 215 /// rule out as a possibility based upon better context information. If that's 216 /// the case, then the SLL parse will definitely get an error because the full LL 217 /// analysis says it's not viable. If SLL conflict resolution chooses an 218 /// alternative within the LL set, them both SLL and LL would choose the same 219 /// alternative because they both choose the minimum of multiple conflicting 220 /// alternatives. 221 /// 222 /// 223 /// Let's say we have a set of SLL conflicting alternatives `{1, 2, 3`} and 224 /// a smaller LL set called __s__. If __s__ is `{2, 3`}, then SLL 225 /// parsing will get an error because SLL will pursue alternative 1. If 226 /// __s__ is `{1, 2`} or `{1, 3`} then both SLL and LL will 227 /// choose the same alternative because alternative one is the minimum of either 228 /// set. If __s__ is `{2`} or `{3`} then SLL will get a syntax 229 /// error. If __s__ is `{1`} then SLL will succeed. 230 /// 231 /// 232 /// Of course, if the input is invalid, then we will get an error for sure in 233 /// both SLL and LL parsing. Erroneous input will therefore require 2 passes over 234 /// the input. 235 /// 236 import Foundation 237 238 open class ParserATNSimulator: ATNSimulator { 239 public let debug = false 240 public let debug_list_atn_decisions = false 241 public let dfa_debug = false 242 public let retry_debug = false 243 244 /// 245 /// Just in case this optimization is bad, add an ENV variable to turn it off 246 /// 247 public static let TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT: Bool = { 248 if let value = ProcessInfo.processInfo.environment["TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT"] { 249 return NSString(string: value).boolValue 250 } 251 return false 252 }() 253 254 internal final unowned let parser: Parser 255 256 public private(set) final var decisionToDFA: [DFA] 257 258 /// 259 /// SLL, LL, or LL + exact ambig detection? 260 /// 261 262 private var mode = PredictionMode.LL 263 264 /// 265 /// Each prediction operation uses a cache for merge of prediction contexts. 266 /// Don't keep around as it wastes huge amounts of memory. DoubleKeyMap 267 /// isn't synchronized but we're ok since two threads shouldn't reuse same 268 /// parser/atnsim object because it can only handle one input at a time. 269 /// This maps graphs a and b to merged result c. (a,b)→c. We can avoid 270 /// the merge if we ever see a and b again. Note that (b,a)→c should 271 /// also be examined during cache lookup. 272 /// 273 internal final var mergeCache: DoubleKeyMap<PredictionContext, PredictionContext, PredictionContext>? 274 275 // LAME globals to avoid parameters!!!!! I need these down deep in predTransition 276 internal var _input: TokenStream! 277 internal var _startIndex = 0 278 internal var _outerContext: ParserRuleContext! 279 internal var _dfa: DFA? 280 281 /// 282 /// mutex for DFAState change 283 /// 284 private let dfaStateMutex = Mutex() 285 286 /// 287 /// mutex for changes in a DFAStates map 288 /// 289 private let dfaStatesMutex = Mutex() 290 291 // /// Testing only! 292 // public convenience init(_ atn : ATN, _ decisionToDFA : [DFA], 293 // _ sharedContextCache : PredictionContextCache) { 294 // self.init(nil, atn, decisionToDFA, sharedContextCache); 295 // } 296 297 public init(_ parser: Parser, _ atn: ATN, 298 _ decisionToDFA: [DFA], 299 _ sharedContextCache: PredictionContextCache) { 300 301 self.parser = parser 302 self.decisionToDFA = decisionToDFA 303 super.init(atn, sharedContextCache) 304 // DOTGenerator dot = new DOTGenerator(null); 305 // print(dot.getDOT(atn.rules.get(0), parser.getRuleNames())); 306 // print(dot.getDOT(atn.rules.get(1), parser.getRuleNames())); 307 } 308 309 override resetnull310 open func reset() { 311 } 312 313 override clearDFAnull314 open func clearDFA() { 315 for d in 0..<decisionToDFA.count { 316 decisionToDFA[d] = DFA(atn.getDecisionState(d)!, d) 317 } 318 } 319 320 open func adaptivePredict(_ input: TokenStream, _ decision: Int, 321 _ outerContext: ParserRuleContext?) throws -> Int { 322 var outerContext = outerContext 323 if debug || debug_list_atn_decisions { 324 var debugInfo = "adaptivePredict decision \(decision) " 325 debugInfo += "exec LA(1)==\(try getLookaheadName(input)) " 326 debugInfo += "line \(try input.LT(1)!.getLine()):" 327 debugInfo += "\(try input.LT(1)!.getCharPositionInLine())" 328 print(debugInfo) 329 } 330 331 332 _input = input 333 _startIndex = input.index() 334 _outerContext = outerContext 335 let dfa = decisionToDFA[decision] 336 _dfa = dfa 337 338 let m = input.mark() 339 let index = _startIndex 340 341 // Now we are certain to have a specific decision's DFA 342 // But, do we still need an initial state? 343 //TODO: exception handler 344 do { 345 var s0: DFAState? 346 if dfa.isPrecedenceDfa() { 347 // the start state for a precedence DFA depends on the current 348 // parser precedence, and is provided by a DFA method. 349 s0 = try dfa.getPrecedenceStartState(parser.getPrecedence()) 350 } else { 351 // the start state for a "regular" DFA is just s0 352 s0 = dfa.s0 353 } 354 355 if s0 == nil { 356 //BIG BUG 357 if outerContext == nil { 358 outerContext = ParserRuleContext.EMPTY 359 } 360 if debug || debug_list_atn_decisions { 361 var debugInfo = "predictATN decision \(dfa.decision) " 362 debugInfo += "exec LA(1)==\(try getLookaheadName(input)), " 363 debugInfo += "outerContext=\(outerContext!.toString(parser))" 364 print(debugInfo) 365 } 366 367 let fullCtx = false 368 var s0_closure = try computeStartState(dfa.atnStartState, ParserRuleContext.EMPTY, fullCtx) 369 370 if dfa.isPrecedenceDfa() { 371 /// 372 /// If this is a precedence DFA, we use applyPrecedenceFilter 373 /// to convert the computed start state to a precedence start 374 /// state. We then use DFA.setPrecedenceStartState to set the 375 /// appropriate start state for the precedence level rather 376 /// than simply setting DFA.s0. 377 /// 378 //added by janyou 20160224 379 // dfa.s0!.configs = s0_closure // not used for prediction but useful to know start configs anyway 380 s0_closure = try applyPrecedenceFilter(s0_closure) 381 s0 = addDFAState(dfa, DFAState(s0_closure)) 382 try dfa.setPrecedenceStartState(parser.getPrecedence(), s0!) 383 } else { 384 s0 = addDFAState(dfa, DFAState(s0_closure)) 385 dfa.s0 = s0 386 } 387 } 388 389 let alt = try execATN(dfa, s0!, input, index, outerContext!) 390 if debug { 391 print("DFA after predictATN: \(dfa.toString(parser.getVocabulary()))") 392 } 393 mergeCache = nil // wack cache after each prediction 394 _dfa = nil 395 try! input.seek(index) 396 try! input.release(m) 397 return alt 398 } 399 400 } 401 402 /// 403 /// Performs ATN simulation to compute a predicted alternative based 404 /// upon the remaining input, but also updates the DFA cache to avoid 405 /// having to traverse the ATN again for the same input sequence. 406 /// 407 /// There are some key conditions we're looking for after computing a new 408 /// set of ATN configs (proposed DFA state): 409 /// if the set is empty, there is no viable alternative for current symbol 410 /// does the state uniquely predict an alternative? 411 /// does the state have a conflict that would prevent us from 412 /// putting it on the work list? 413 /// 414 /// We also have some key operations to do: 415 /// add an edge from previous DFA state to potentially new DFA state, D, 416 /// upon current symbol but only if adding to work list, which means in all 417 /// cases except no viable alternative (and possibly non-greedy decisions?) 418 /// collecting predicates and adding semantic context to DFA accept states 419 /// adding rule context to context-sensitive DFA accept states 420 /// consuming an input symbol 421 /// reporting a conflict 422 /// reporting an ambiguity 423 /// reporting a context sensitivity 424 /// reporting insufficient predicates 425 /// 426 /// cover these cases: 427 /// dead end 428 /// single alt 429 /// single alt + preds 430 /// conflict 431 /// conflict + preds 432 /// 433 final func execATN(_ dfa: DFA, _ s0: DFAState, 434 _ input: TokenStream, _ startIndex: Int, 435 _ outerContext: ParserRuleContext) throws -> Int { 436 if debug || debug_list_atn_decisions { 437 try print("execATN decision \(dfa.decision) exec LA(1)==\(getLookaheadName(input)) line \(input.LT(1)!.getLine()):\(input.LT(1)!.getCharPositionInLine())") 438 } 439 440 var previousD = s0 441 442 if debug { 443 print("s0 = \(s0)") 444 } 445 446 var t = try input.LA(1) 447 448 while true { 449 // while more work 450 var D: DFAState 451 if let dState = getExistingTargetState(previousD, t) { 452 D = dState 453 } else { 454 D = try computeTargetState(dfa, previousD, t) 455 } 456 457 if D == ATNSimulator.ERROR { 458 // if any configs in previous dipped into outer context, that 459 // means that input up to t actually finished entry rule 460 // at least for SLL decision. Full LL doesn't dip into outer 461 // so don't need special case. 462 // We will get an error no matter what so delay until after 463 // decision; better error message. Also, no reachable target 464 // ATN states in SLL implies LL will also get nowhere. 465 // If conflict in states that dip out, choose min since we 466 // will get error no matter what. 467 let e = noViableAlt(input, outerContext, previousD.configs, startIndex) 468 try input.seek(startIndex) 469 let alt = try getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(previousD.configs, outerContext) 470 if alt != ATN.INVALID_ALT_NUMBER { 471 return alt 472 } 473 474 throw ANTLRException.recognition(e: e) 475 476 } 477 478 if D.requiresFullContext && (mode != PredictionMode.SLL) { 479 // IF PREDS, MIGHT RESOLVE TO SINGLE ALT => SLL (or syntax error) 480 var conflictingAlts = D.configs.conflictingAlts! 481 if D.predicates != nil { 482 if debug { 483 print("DFA state has preds in DFA sim LL failover") 484 } 485 let conflictIndex = input.index() 486 if conflictIndex != startIndex { 487 try input.seek(startIndex) 488 } 489 490 conflictingAlts = try evalSemanticContext(D.predicates!, outerContext, true) 491 if conflictingAlts.cardinality() == 1 { 492 if debug { 493 print("Full LL avoided") 494 } 495 return conflictingAlts.firstSetBit() 496 } 497 498 if conflictIndex != startIndex { 499 // restore the index so reporting the fallback to full 500 // context occurs with the index at the correct spot 501 try input.seek(conflictIndex) 502 } 503 } 504 505 if dfa_debug { 506 print("ctx sensitive state \(outerContext) in \(D)") 507 } 508 let fullCtx = true 509 let s0_closure = try computeStartState(dfa.atnStartState, outerContext, fullCtx) 510 reportAttemptingFullContext(dfa, conflictingAlts, D.configs, startIndex, input.index()) 511 let alt = try execATNWithFullContext(dfa, D, s0_closure, 512 input, startIndex, 513 outerContext) 514 return alt 515 } 516 517 if D.isAcceptState { 518 if D.predicates == nil { 519 return D.prediction 520 } 521 522 let stopIndex = input.index() 523 try input.seek(startIndex) 524 let alts = try evalSemanticContext(D.predicates!, outerContext, true) 525 switch alts.cardinality() { 526 case 0: 527 throw ANTLRException.recognition(e: noViableAlt(input, outerContext, D.configs, startIndex)) 528 529 530 case 1: 531 return alts.firstSetBit() 532 533 default: 534 // report ambiguity after predicate evaluation to make sure the correct 535 // set of ambig alts is reported. 536 reportAmbiguity(dfa, D, startIndex, stopIndex, false, alts, D.configs) 537 return alts.firstSetBit() 538 } 539 } 540 541 previousD = D 542 543 if t != BufferedTokenStream.EOF { 544 try input.consume() 545 t = try input.LA(1) 546 } 547 } 548 } 549 550 /// 551 /// Get an existing target state for an edge in the DFA. If the target state 552 /// for the edge has not yet been computed or is otherwise not available, 553 /// this method returns `null`. 554 /// 555 /// - parameter previousD: The current DFA state 556 /// - parameter t: The next input symbol 557 /// - returns: The existing target DFA state for the given input symbol 558 /// `t`, or `null` if the target state for this edge is not 559 /// already cached 560 /// getExistingTargetStatenull561 func getExistingTargetState(_ previousD: DFAState, _ t: Int) -> DFAState? { 562 var edges = previousD.edges 563 if edges == nil || (t + 1) < 0 || (t + 1) >= (edges!.count) { 564 return nil 565 } 566 567 return edges![t + 1] 568 } 569 570 /// 571 /// Compute a target state for an edge in the DFA, and attempt to add the 572 /// computed state and corresponding edge to the DFA. 573 /// 574 /// - parameter dfa: The DFA 575 /// - parameter previousD: The current DFA state 576 /// - parameter t: The next input symbol 577 /// 578 /// - returns: The computed target DFA state for the given input symbol 579 /// `t`. If `t` does not lead to a valid DFA state, this method 580 /// returns _#ERROR_. 581 /// computeTargetStatenull582 func computeTargetState(_ dfa: DFA, _ previousD: DFAState, _ t: Int) throws -> DFAState { 583 584 guard let reach = try computeReachSet(previousD.configs, t, false) else { 585 addDFAEdge(dfa, previousD, t, ATNSimulator.ERROR) 586 return ATNSimulator.ERROR 587 } 588 589 // create new target state; we'll add to DFA after it's complete 590 let D = DFAState(reach) 591 592 let predictedAlt = ParserATNSimulator.getUniqueAlt(reach) 593 594 if debug { 595 let altSubSets = PredictionMode.getConflictingAltSubsets(reach) 596 print("SLL altSubSets=\(altSubSets), configs=\(reach), predict=\(predictedAlt), allSubsetsConflict=\(PredictionMode.allSubsetsConflict(altSubSets)), conflictingAlts=\(getConflictingAlts(reach))") 597 } 598 599 if predictedAlt != ATN.INVALID_ALT_NUMBER { 600 // NO CONFLICT, UNIQUELY PREDICTED ALT 601 D.isAcceptState = true 602 D.configs.uniqueAlt = predictedAlt 603 D.prediction = predictedAlt 604 } else { 605 if PredictionMode.hasSLLConflictTerminatingPrediction(mode, reach) { 606 // MORE THAN ONE VIABLE ALTERNATIVE 607 D.configs.conflictingAlts = getConflictingAlts(reach) 608 D.requiresFullContext = true 609 // in SLL-only mode, we will stop at this state and return the minimum alt 610 D.isAcceptState = true 611 D.prediction = D.configs.conflictingAlts!.firstSetBit() 612 } 613 } 614 615 if D.isAcceptState && D.configs.hasSemanticContext { 616 predicateDFAState(D, atn.getDecisionState(dfa.decision)!) 617 if D.predicates != nil { 618 D.prediction = ATN.INVALID_ALT_NUMBER 619 } 620 } 621 622 // all adds to dfa are done after we've created full D state 623 return addDFAEdge(dfa, previousD, t, D) 624 } 625 predicateDFAStatenull626 final func predicateDFAState(_ dfaState: DFAState, _ decisionState: DecisionState) { 627 // We need to test all predicates, even in DFA states that 628 // uniquely predict alternative. 629 let nalts = decisionState.getNumberOfTransitions() 630 // Update DFA so reach becomes accept state with (predicate,alt) 631 // pairs if preds found for conflicting alts 632 let altsToCollectPredsFrom = getConflictingAltsOrUniqueAlt(dfaState.configs) 633 if let altToPred = getPredsForAmbigAlts(altsToCollectPredsFrom, dfaState.configs, nalts) { 634 dfaState.predicates = getPredicatePredictions(altsToCollectPredsFrom, altToPred) 635 dfaState.prediction = ATN.INVALID_ALT_NUMBER // make sure we use preds 636 } else { 637 // There are preds in configs but they might go away 638 // when OR'd together like {p}? || NONE == NONE. If neither 639 // alt has preds, resolve to min alt 640 dfaState.prediction = altsToCollectPredsFrom.firstSetBit() 641 } 642 } 643 644 // comes back with reach.uniqueAlt set to a valid alt 645 final func execATNWithFullContext(_ dfa: DFA, 646 _ D: DFAState, // how far we got in SLL DFA before failing over 647 _ s0: ATNConfigSet, 648 _ input: TokenStream, _ startIndex: Int, 649 _ outerContext: ParserRuleContext) throws -> Int { 650 if debug || debug_list_atn_decisions { 651 print("execATNWithFullContext \(s0)") 652 } 653 let fullCtx = true 654 var foundExactAmbig = false 655 var reach: ATNConfigSet? = nil 656 var previous = s0 657 try input.seek(startIndex) 658 var t = try input.LA(1) 659 var predictedAlt = 0 660 while true { 661 // while more work 662 if let computeReach = try computeReachSet(previous, t, fullCtx) { 663 reach = computeReach 664 } else { 665 // if any configs in previous dipped into outer context, that 666 // means that input up to t actually finished entry rule 667 // at least for LL decision. Full LL doesn't dip into outer 668 // so don't need special case. 669 // We will get an error no matter what so delay until after 670 // decision; better error message. Also, no reachable target 671 // ATN states in SLL implies LL will also get nowhere. 672 // If conflict in states that dip out, choose min since we 673 // will get error no matter what. 674 let e = noViableAlt(input, outerContext, previous, startIndex) 675 try input.seek(startIndex) 676 let alt = try getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(previous, outerContext) 677 if alt != ATN.INVALID_ALT_NUMBER { 678 return alt 679 } 680 throw ANTLRException.recognition(e: e) 681 682 } 683 if let reach = reach { 684 let altSubSets = PredictionMode.getConflictingAltSubsets(reach) 685 if debug { 686 print("LL altSubSets=\(altSubSets), predict=\(PredictionMode.getUniqueAlt(altSubSets)), resolvesToJustOneViableAlt=\(PredictionMode.resolvesToJustOneViableAlt(altSubSets))") 687 } 688 689 690 reach.uniqueAlt = ParserATNSimulator.getUniqueAlt(reach) 691 // unique prediction? 692 if reach.uniqueAlt != ATN.INVALID_ALT_NUMBER { 693 predictedAlt = reach.uniqueAlt 694 break 695 } 696 if mode != PredictionMode.LL_EXACT_AMBIG_DETECTION { 697 predictedAlt = PredictionMode.resolvesToJustOneViableAlt(altSubSets) 698 if predictedAlt != ATN.INVALID_ALT_NUMBER { 699 break 700 } 701 } else { 702 // In exact ambiguity mode, we never try to terminate early. 703 // Just keeps scarfing until we know what the conflict is 704 if PredictionMode.allSubsetsConflict(altSubSets) && 705 PredictionMode.allSubsetsEqual(altSubSets) { 706 foundExactAmbig = true 707 predictedAlt = PredictionMode.getSingleViableAlt(altSubSets) 708 break 709 } 710 // else there are multiple non-conflicting subsets or 711 // we're not sure what the ambiguity is yet. 712 // So, keep going. 713 } 714 715 previous = reach 716 if t != BufferedTokenStream.EOF { 717 try input.consume() 718 t = try input.LA(1) 719 } 720 } 721 } 722 if let reach = reach { 723 // If the configuration set uniquely predicts an alternative, 724 // without conflict, then we know that it's a full LL decision 725 // not SLL. 726 if reach.uniqueAlt != ATN.INVALID_ALT_NUMBER { 727 reportContextSensitivity(dfa, predictedAlt, reach, startIndex, input.index()) 728 return predictedAlt 729 } 730 731 // We do not check predicates here because we have checked them 732 // on-the-fly when doing full context prediction. 733 734 /// 735 /// In non-exact ambiguity detection mode, we might actually be able to 736 /// detect an exact ambiguity, but I'm not going to spend the cycles 737 /// needed to check. We only emit ambiguity warnings in exact ambiguity 738 /// mode. 739 /// 740 /// For example, we might know that we have conflicting configurations. 741 /// But, that does not mean that there is no way forward without a 742 /// conflict. It's possible to have nonconflicting alt subsets as in: 743 /// 744 /// LL altSubSets=[{1, 2}, {1, 2}, {1}, {1, 2}] 745 /// 746 /// from 747 /// 748 /// [(17,1,[5 $]), (13,1,[5 10 $]), (21,1,[5 10 $]), (11,1,[$]), 749 /// (13,2,[5 10 $]), (21,2,[5 10 $]), (11,2,[$])] 750 /// 751 /// In this case, (17,1,[5 $]) indicates there is some next sequence that 752 /// would resolve this without conflict to alternative 1. Any other viable 753 /// next sequence, however, is associated with a conflict. We stop 754 /// looking for input because no amount of further lookahead will alter 755 /// the fact that we should predict alternative 1. We just can't say for 756 /// sure that there is an ambiguity without looking further. 757 /// 758 reportAmbiguity(dfa, D, startIndex, input.index(), foundExactAmbig, 759 reach.getAlts(), reach) 760 } 761 return predictedAlt 762 } 763 764 func computeReachSet(_ closureConfigSet: ATNConfigSet, _ t: Int, 765 _ fullCtx: Bool) throws -> ATNConfigSet? { 766 767 if debug { 768 print("in computeReachSet, starting closure: \(closureConfigSet)") 769 } 770 771 if mergeCache == nil { 772 mergeCache = DoubleKeyMap<PredictionContext, PredictionContext, PredictionContext>() 773 } 774 775 let intermediate = ATNConfigSet(fullCtx) 776 777 /// 778 /// Configurations already in a rule stop state indicate reaching the end 779 /// of the decision rule (local context) or end of the start rule (full 780 /// context). Once reached, these configurations are never updated by a 781 /// closure operation, so they are handled separately for the performance 782 /// advantage of having a smaller intermediate set when calling closure. 783 /// 784 /// For full-context reach operations, separate handling is required to 785 /// ensure that the alternative matching the longest overall sequence is 786 /// chosen when multiple such configurations can match the input. 787 /// 788 var skippedStopStates: [ATNConfig]? = nil 789 790 // First figure out where we can reach on input t 791 let configs = closureConfigSet.configs 792 for config in configs { 793 if debug { 794 print("testing \(getTokenName(t)) at \(config.description)") 795 } 796 797 if config.state is RuleStopState { 798 assert(config.context!.isEmpty(), "Expected: c.context.isEmpty()") 799 if fullCtx || t == BufferedTokenStream.EOF { 800 if skippedStopStates == nil { 801 skippedStopStates = [ATNConfig]() 802 } 803 skippedStopStates!.append(config) 804 } 805 806 continue 807 } 808 809 let n = config.state.getNumberOfTransitions() 810 for ti in 0..<n { 811 // for each transition 812 let trans = config.state.transition(ti) 813 if let target = getReachableTarget(trans, t) { 814 try! intermediate.add(ATNConfig(config, target), &mergeCache) 815 } 816 } 817 } 818 819 // Now figure out where the reach operation can take us... 820 821 var reach: ATNConfigSet? = nil 822 823 /// 824 /// This block optimizes the reach operation for intermediate sets which 825 /// trivially indicate a termination state for the overall 826 /// adaptivePredict operation. 827 /// 828 /// The conditions assume that intermediate 829 /// contains all configurations relevant to the reach set, but this 830 /// condition is not true when one or more configurations have been 831 /// withheld in skippedStopStates, or when the current symbol is EOF. 832 /// 833 if skippedStopStates == nil && t != CommonToken.EOF { 834 if intermediate.size() == 1 { 835 // Don't pursue the closure if there is just one state. 836 // It can only have one alternative; just add to result 837 // Also don't pursue the closure if there is unique alternative 838 // among the configurations. 839 reach = intermediate 840 } else { 841 if ParserATNSimulator.getUniqueAlt(intermediate) != ATN.INVALID_ALT_NUMBER { 842 // Also don't pursue the closure if there is unique alternative 843 // among the configurations. 844 reach = intermediate 845 } 846 } 847 } 848 849 /// 850 /// If the reach set could not be trivially determined, perform a closure 851 /// operation on the intermediate set to compute its initial value. 852 /// 853 if reach == nil { 854 reach = ATNConfigSet(fullCtx) 855 var closureBusy = Set<ATNConfig>() 856 let treatEofAsEpsilon = (t == CommonToken.EOF) 857 for config in intermediate.configs { 858 try closure(config, reach!, &closureBusy, false, fullCtx, treatEofAsEpsilon) 859 } 860 } 861 862 if t == BufferedTokenStream.EOF { 863 /// 864 /// After consuming EOF no additional input is possible, so we are 865 /// only interested in configurations which reached the end of the 866 /// decision rule (local context) or end of the start rule (full 867 /// context). Update reach to contain only these configurations. This 868 /// handles both explicit EOF transitions in the grammar and implicit 869 /// EOF transitions following the end of the decision or start rule. 870 /// 871 /// When reach==intermediate, no closure operation was performed. In 872 /// this case, removeAllConfigsNotInRuleStopState needs to check for 873 /// reachable rule stop states as well as configurations already in 874 /// a rule stop state. 875 /// 876 /// This is handled before the configurations in skippedStopStates, 877 /// because any configurations potentially added from that list are 878 /// already guaranteed to meet this condition whether or not it's 879 /// required. 880 /// 881 reach = removeAllConfigsNotInRuleStopState(reach!, reach! === intermediate) 882 } 883 884 /// 885 /// If skippedStopStates is not null, then it contains at least one 886 /// configuration. For full-context reach operations, these 887 /// configurations reached the end of the start rule, in which case we 888 /// only add them back to reach if no configuration during the current 889 /// closure operation reached such a state. This ensures adaptivePredict 890 /// chooses an alternative matching the longest overall sequence when 891 /// multiple alternatives are viable. 892 /// 893 if let reach = reach { 894 if let skippedStopStates = skippedStopStates, (!fullCtx || !PredictionMode.hasConfigInRuleStopState(reach)) { 895 assert(!skippedStopStates.isEmpty, "Expected: !skippedStopStates.isEmpty()") 896 for c in skippedStopStates { 897 try! reach.add(c, &mergeCache) 898 } 899 } 900 901 if reach.isEmpty() { 902 return nil 903 } 904 } 905 return reach 906 } 907 908 /// 909 /// Return a configuration set containing only the configurations from 910 /// `configs` which are in a _org.antlr.v4.runtime.atn.RuleStopState_. If all 911 /// configurations in `configs` are already in a rule stop state, this 912 /// method simply returns `configs`. 913 /// 914 /// When `lookToEndOfRule` is true, this method uses 915 /// _org.antlr.v4.runtime.atn.ATN#nextTokens_ for each configuration in `configs` which is 916 /// not already in a rule stop state to see if a rule stop state is reachable 917 /// from the configuration via epsilon-only transitions. 918 /// 919 /// - parameter configs: the configuration set to update 920 /// - parameter lookToEndOfRule: when true, this method checks for rule stop states 921 /// reachable by epsilon-only transitions from each configuration in 922 /// `configs`. 923 /// 924 /// - returns: `configs` if all configurations in `configs` are in a 925 /// rule stop state, otherwise return a new configuration set containing only 926 /// the configurations from `configs` which are in a rule stop state 927 /// removeAllConfigsNotInRuleStopStatenull928 final func removeAllConfigsNotInRuleStopState(_ configs: ATNConfigSet, _ lookToEndOfRule: Bool) -> ATNConfigSet { 929 return configs.removeAllConfigsNotInRuleStopState(&mergeCache,lookToEndOfRule,atn) 930 } 931 932 computeStartStatenull933 final func computeStartState(_ p: ATNState, _ ctx: RuleContext, _ fullCtx: Bool) throws -> ATNConfigSet { 934 let initialContext = PredictionContext.fromRuleContext(atn, ctx) 935 let configs = ATNConfigSet(fullCtx) 936 let length = p.getNumberOfTransitions() 937 for i in 0..<length { 938 let target = p.transition(i).target 939 let c = ATNConfig(target, i + 1, initialContext) 940 var closureBusy = Set<ATNConfig>() 941 try closure(c, configs, &closureBusy, true, fullCtx, false) 942 } 943 944 return configs 945 } 946 947 /// 948 /// parrt internal source braindump that doesn't mess up 949 /// external API spec. 950 /// 951 /// applyPrecedenceFilter is an optimization to avoid highly 952 /// nonlinear prediction of expressions and other left recursive 953 /// rules. The precedence predicates such as {3>=prec}? Are highly 954 /// context-sensitive in that they can only be properly evaluated 955 /// in the context of the proper prec argument. Without pruning, 956 /// these predicates are normal predicates evaluated when we reach 957 /// conflict state (or unique prediction). As we cannot evaluate 958 /// these predicates out of context, the resulting conflict leads 959 /// to full LL evaluation and nonlinear prediction which shows up 960 /// very clearly with fairly large expressions. 961 /// 962 /// Example grammar: 963 /// 964 /// e : e '*' e 965 /// | e '+' e 966 /// | INT 967 /// ; 968 /// 969 /// We convert that to the following: 970 /// 971 /// e[int prec] 972 /// : INT 973 /// ( {3>=prec}? '*' e[4] 974 /// | {2>=prec}? '+' e[3] 975 /// )* 976 /// ; 977 /// 978 /// The (..)* loop has a decision for the inner block as well as 979 /// an enter or exit decision, which is what concerns us here. At 980 /// the 1st + of input 1+2+3, the loop entry sees both predicates 981 /// and the loop exit also sees both predicates by falling off the 982 /// edge of e. This is because we have no stack information with 983 /// SLL and find the follow of e, which will hit the return states 984 /// inside the loop after e[4] and e[3], which brings it back to 985 /// the enter or exit decision. In this case, we know that we 986 /// cannot evaluate those predicates because we have fallen off 987 /// the edge of the stack and will in general not know which prec 988 /// parameter is the right one to use in the predicate. 989 /// 990 /// Because we have special information, that these are precedence 991 /// predicates, we can resolve them without failing over to full 992 /// LL despite their context sensitive nature. We make an 993 /// assumption that prec[-1] <= prec[0], meaning that the current 994 /// precedence level is greater than or equal to the precedence 995 /// level of recursive invocations above us in the stack. For 996 /// example, if predicate {3>=prec}? is true of the current prec, 997 /// then one option is to enter the loop to match it now. The 998 /// other option is to exit the loop and the left recursive rule 999 /// to match the current operator in rule invocation further up 1000 /// the stack. But, we know that all of those prec are lower or 1001 /// the same value and so we can decide to enter the loop instead 1002 /// of matching it later. That means we can strip out the other 1003 /// configuration for the exit branch. 1004 /// 1005 /// So imagine we have (14,1,$,{2>=prec}?) and then 1006 /// (14,2,$-dipsIntoOuterContext,{2>=prec}?). The optimization 1007 /// allows us to collapse these two configurations. We know that 1008 /// if {2>=prec}? is true for the current prec parameter, it will 1009 /// also be true for any prec from an invoking e call, indicated 1010 /// by dipsIntoOuterContext. As the predicates are both true, we 1011 /// have the option to evaluate them early in the decision start 1012 /// state. We do this by stripping both predicates and choosing to 1013 /// enter the loop as it is consistent with the notion of operator 1014 /// precedence. It's also how the full LL conflict resolution 1015 /// would work. 1016 /// 1017 /// The solution requires a different DFA start state for each 1018 /// precedence level. 1019 /// 1020 /// The basic filter mechanism is to remove configurations of the 1021 /// form (p, 2, pi) if (p, 1, pi) exists for the same p and pi. In 1022 /// other words, for the same ATN state and predicate context, 1023 /// remove any configuration associated with an exit branch if 1024 /// there is a configuration associated with the enter branch. 1025 /// 1026 /// It's also the case that the filter evaluates precedence 1027 /// predicates and resolves conflicts according to precedence 1028 /// levels. For example, for input 1+2+3 at the first +, we see 1029 /// prediction filtering 1030 /// 1031 /// [(11,1,[$],{3>=prec}?), (14,1,[$],{2>=prec}?), (5,2,[$],up=1), 1032 /// (11,2,[$],up=1), (14,2,[$],up=1)],hasSemanticContext=true,dipsIntoOuterContext 1033 /// 1034 /// to 1035 /// 1036 /// [(11,1,[$]), (14,1,[$]), (5,2,[$],up=1)],dipsIntoOuterContext 1037 /// 1038 /// This filters because {3>=prec}? evals to true and collapses 1039 /// (11,1,[$],{3>=prec}?) and (11,2,[$],up=1) since early conflict 1040 /// resolution based upon rules of operator precedence fits with 1041 /// our usual match first alt upon conflict. 1042 /// 1043 /// We noticed a problem where a recursive call resets precedence 1044 /// to 0. Sam's fix: each config has flag indicating if it has 1045 /// returned from an expr[0] call. then just don't filter any 1046 /// config with that flag set. flag is carried along in 1047 /// closure(). so to avoid adding field, set bit just under sign 1048 /// bit of dipsIntoOuterContext (SUPPRESS_PRECEDENCE_FILTER). 1049 /// With the change you filter "unless (p, 2, pi) was reached 1050 /// after leaving the rule stop state of the LR rule containing 1051 /// state p, corresponding to a rule invocation with precedence 1052 /// level 0" 1053 /// 1054 1055 /// 1056 /// This method transforms the start state computed by 1057 /// _#computeStartState_ to the special start state used by a 1058 /// precedence DFA for a particular precedence value. The transformation 1059 /// process applies the following changes to the start state's configuration 1060 /// set. 1061 /// 1062 /// * Evaluate the precedence predicates for each configuration using 1063 /// _org.antlr.v4.runtime.atn.SemanticContext#evalPrecedence_. 1064 /// * When _org.antlr.v4.runtime.atn.ATNConfig#isPrecedenceFilterSuppressed_ is `false`, 1065 /// remove all configurations which predict an alternative greater than 1, 1066 /// for which another configuration that predicts alternative 1 is in the 1067 /// same ATN state with the same prediction context. This transformation is 1068 /// valid for the following reasons: 1069 /// 1070 /// * The closure block cannot contain any epsilon transitions which bypass 1071 /// the body of the closure, so all states reachable via alternative 1 are 1072 /// part of the precedence alternatives of the transformed left-recursive 1073 /// rule. 1074 /// * The "primary" portion of a left recursive rule cannot contain an 1075 /// epsilon transition, so the only way an alternative other than 1 can exist 1076 /// in a state that is also reachable via alternative 1 is by nesting calls 1077 /// to the left-recursive rule, with the outer calls not being at the 1078 /// preferred precedence level. The 1079 /// _org.antlr.v4.runtime.atn.ATNConfig#isPrecedenceFilterSuppressed_ property marks ATN 1080 /// configurations which do not meet this condition, and therefore are not 1081 /// eligible for elimination during the filtering process. 1082 /// 1083 /// The prediction context must be considered by this filter to address 1084 /// situations like the following. 1085 /// ``` 1086 /// grammar TA; 1087 /// prog: statement* EOF; 1088 /// statement: letterA | statement letterA 'b' ; 1089 /// letterA: 'a'; 1090 /// ``` 1091 /// If the above grammar, the ATN state immediately before the token 1092 /// reference `'a'` in `letterA` is reachable from the left edge 1093 /// of both the primary and closure blocks of the left-recursive rule 1094 /// `statement`. The prediction context associated with each of these 1095 /// configurations distinguishes between them, and prevents the alternative 1096 /// which stepped out to `prog` (and then back in to `statement` 1097 /// from being eliminated by the filter. 1098 /// 1099 /// - parameter configs: The configuration set computed by 1100 /// _#computeStartState_ as the start state for the DFA. 1101 /// - returns: The transformed configuration set representing the start state 1102 /// for a precedence DFA at a particular precedence level (determined by 1103 /// calling _org.antlr.v4.runtime.Parser#getPrecedence_). 1104 /// applyPrecedenceFilternull1105 final internal func applyPrecedenceFilter(_ configs: ATNConfigSet) throws -> ATNConfigSet { 1106 return try configs.applyPrecedenceFilter(&mergeCache,parser,_outerContext) 1107 } 1108 getReachableTargetnull1109 final internal func getReachableTarget(_ trans: Transition, _ ttype: Int) -> ATNState? { 1110 1111 if trans.matches(ttype, 0, atn.maxTokenType) { 1112 return trans.target 1113 } 1114 1115 return nil 1116 } 1117 1118 final internal func getPredsForAmbigAlts(_ ambigAlts: BitSet, 1119 _ configs: ATNConfigSet, 1120 _ nalts: Int) -> [SemanticContext?]? { 1121 // REACH=[1|1|[]|0:0, 1|2|[]|0:1] 1122 /// 1123 /// altToPred starts as an array of all null contexts. The entry at index i 1124 /// corresponds to alternative i. altToPred[i] may have one of three values: 1125 /// 1. null: no ATNConfig c is found such that c.alt==i 1126 /// 2. SemanticContext.NONE: At least one ATNConfig c exists such that 1127 /// c.alt==i and c.semanticContext==SemanticContext.NONE. In other words, 1128 /// alt i has at least one unpredicated config. 1129 /// 3. Non-NONE Semantic Context: There exists at least one, and for all 1130 /// ATNConfig c such that c.alt==i, c.semanticContext!=SemanticContext.NONE. 1131 /// 1132 /// From this, it is clear that NONE||anything==NONE. 1133 /// 1134 let altToPred = configs.getPredsForAmbigAlts(ambigAlts,nalts) 1135 if debug { 1136 print("getPredsForAmbigAlts result \(String(describing: altToPred))") 1137 } 1138 return altToPred 1139 } 1140 1141 final internal func getPredicatePredictions(_ ambigAlts: BitSet?, 1142 _ altToPred: [SemanticContext?]) -> [DFAState.PredPrediction]? { 1143 var pairs = [DFAState.PredPrediction]() 1144 var containsPredicate = false 1145 let length = altToPred.count 1146 for i in 1..<length { 1147 let pred = altToPred[i] 1148 1149 // unpredicated is indicated by SemanticContext.NONE 1150 assert(pred != nil, "Expected: pred!=null") 1151 1152 if let ambigAlts = ambigAlts, try! ambigAlts.get(i) { 1153 pairs.append(DFAState.PredPrediction(pred!, i)) 1154 } 1155 if pred != SemanticContext.NONE { 1156 containsPredicate = true 1157 } 1158 } 1159 1160 if !containsPredicate { 1161 return nil 1162 } 1163 1164 return pairs ///pairs.toArray(new, DFAState.PredPrediction[pairs.size()]); 1165 } 1166 1167 /// 1168 /// This method is used to improve the localization of error messages by 1169 /// choosing an alternative rather than throwing a 1170 /// _org.antlr.v4.runtime.NoViableAltException_ in particular prediction scenarios where the 1171 /// _#ERROR_ state was reached during ATN simulation. 1172 /// 1173 /// The default implementation of this method uses the following 1174 /// algorithm to identify an ATN configuration which successfully parsed the 1175 /// decision entry rule. Choosing such an alternative ensures that the 1176 /// _org.antlr.v4.runtime.ParserRuleContext_ returned by the calling rule will be complete 1177 /// and valid, and the syntax error will be reported later at a more 1178 /// localized location. 1179 /// 1180 /// * If a syntactically valid path or paths reach the end of the decision rule and 1181 /// they are semantically valid if predicated, return the min associated alt. 1182 /// * Else, if a semantically invalid but syntactically valid path exist 1183 /// or paths exist, return the minimum associated alt. 1184 /// * Otherwise, return _org.antlr.v4.runtime.atn.ATN#INVALID_ALT_NUMBER_. 1185 /// 1186 /// In some scenarios, the algorithm described above could predict an 1187 /// alternative which will result in a _org.antlr.v4.runtime.FailedPredicateException_ in 1188 /// the parser. Specifically, this could occur if the __only__ configuration 1189 /// capable of successfully parsing to the end of the decision rule is 1190 /// blocked by a semantic predicate. By choosing this alternative within 1191 /// _#adaptivePredict_ instead of throwing a 1192 /// _org.antlr.v4.runtime.NoViableAltException_, the resulting 1193 /// _org.antlr.v4.runtime.FailedPredicateException_ in the parser will identify the specific 1194 /// predicate which is preventing the parser from successfully parsing the 1195 /// decision rule, which helps developers identify and correct logic errors 1196 /// in semantic predicates. 1197 /// 1198 /// - parameter configs: The ATN configurations which were valid immediately before 1199 /// the _#ERROR_ state was reached 1200 /// - parameter outerContext: The is the \gamma_0 initial parser context from the paper 1201 /// or the parser stack at the instant before prediction commences. 1202 /// 1203 /// - returns: The value to return from _#adaptivePredict_, or 1204 /// _org.antlr.v4.runtime.atn.ATN#INVALID_ALT_NUMBER_ if a suitable alternative was not 1205 /// identified and _#adaptivePredict_ should report an error instead. 1206 /// 1207 final internal func getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(_ configs: ATNConfigSet, 1208 _ outerContext: ParserRuleContext) throws -> Int { 1209 let (semValidConfigs, semInvalidConfigs) = try splitAccordingToSemanticValidity(configs, outerContext) 1210 var alt = getAltThatFinishedDecisionEntryRule(semValidConfigs) 1211 if alt != ATN.INVALID_ALT_NUMBER { 1212 // semantically/syntactically viable path exists 1213 return alt 1214 } 1215 // Is there a syntactically valid path with a failed pred? 1216 if semInvalidConfigs.size() > 0 { 1217 alt = getAltThatFinishedDecisionEntryRule(semInvalidConfigs) 1218 if alt != ATN.INVALID_ALT_NUMBER { 1219 // syntactically viable path exists 1220 return alt 1221 } 1222 } 1223 return ATN.INVALID_ALT_NUMBER 1224 } 1225 getAltThatFinishedDecisionEntryRulenull1226 final internal func getAltThatFinishedDecisionEntryRule(_ configs: ATNConfigSet) -> Int { 1227 1228 return configs.getAltThatFinishedDecisionEntryRule() 1229 } 1230 1231 /// 1232 /// Walk the list of configurations and split them according to 1233 /// those that have preds evaluating to true/false. If no pred, assume 1234 /// true pred and include in succeeded set. Returns Pair of sets. 1235 /// 1236 /// Create a new set so as not to alter the incoming parameter. 1237 /// 1238 /// Assumption: the input stream has been restored to the starting point 1239 /// prediction, which is where predicates need to evaluate. 1240 /// 1241 final internal func splitAccordingToSemanticValidity( 1242 _ configs: ATNConfigSet, 1243 _ outerContext: ParserRuleContext) throws -> (ATNConfigSet, ATNConfigSet) { 1244 1245 return try configs.splitAccordingToSemanticValidity(outerContext, evalSemanticContext) 1246 } 1247 1248 /// 1249 /// Look through a list of predicate/alt pairs, returning alts for the 1250 /// pairs that win. A `NONE` predicate indicates an alt containing an 1251 /// unpredicated config which behaves as "always true." If !complete 1252 /// then we stop at the first predicate that evaluates to true. This 1253 /// includes pairs with null predicates. 1254 /// 1255 final internal func evalSemanticContext(_ predPredictions: [DFAState.PredPrediction], 1256 _ outerContext: ParserRuleContext, 1257 _ complete: Bool) throws -> BitSet { 1258 let predictions = BitSet() 1259 for pair in predPredictions { 1260 if pair.pred == SemanticContext.NONE { 1261 try! predictions.set(pair.alt) 1262 if !complete { 1263 break 1264 } 1265 continue 1266 } 1267 1268 let fullCtx = false // in dfa 1269 let predicateEvaluationResult = try evalSemanticContext(pair.pred, outerContext, pair.alt, fullCtx) 1270 if debug || dfa_debug { 1271 print("eval pred \(pair)= \(predicateEvaluationResult)") 1272 } 1273 1274 if predicateEvaluationResult { 1275 if debug || dfa_debug { 1276 print("PREDICT \(pair.alt)") 1277 } 1278 try! predictions.set(pair.alt) 1279 if !complete { 1280 break 1281 } 1282 } 1283 } 1284 1285 return predictions 1286 } 1287 1288 /// 1289 /// Evaluate a semantic context within a specific parser context. 1290 /// 1291 /// This method might not be called for every semantic context evaluated 1292 /// during the prediction process. In particular, we currently do not 1293 /// evaluate the following but it may change in the future: 1294 /// 1295 /// * Precedence predicates (represented by 1296 /// _org.antlr.v4.runtime.atn.SemanticContext.PrecedencePredicate_) are not currently evaluated 1297 /// through this method. 1298 /// * Operator predicates (represented by _org.antlr.v4.runtime.atn.SemanticContext.AND_ and 1299 /// _org.antlr.v4.runtime.atn.SemanticContext.OR_) are evaluated as a single semantic 1300 /// context, rather than evaluating the operands individually. 1301 /// Implementations which require evaluation results from individual 1302 /// predicates should override this method to explicitly handle evaluation of 1303 /// the operands within operator predicates. 1304 /// 1305 /// - parameter pred: The semantic context to evaluate 1306 /// - parameter parserCallStack: The parser context in which to evaluate the 1307 /// semantic context 1308 /// - parameter alt: The alternative which is guarded by `pred` 1309 /// - parameter fullCtx: `true` if the evaluation is occurring during LL 1310 /// prediction; otherwise, `false` if the evaluation is occurring 1311 /// during SLL prediction 1312 /// 1313 /// - since: 4.3 1314 /// evalSemanticContextnull1315 internal func evalSemanticContext(_ pred: SemanticContext, _ parserCallStack: ParserRuleContext, _ alt: Int, _ fullCtx: Bool) throws -> Bool { 1316 return try pred.eval(parser, parserCallStack) 1317 } 1318 1319 /// 1320 /// TODO: If we are doing predicates, there is no point in pursuing 1321 /// closure operations if we reach a DFA state that uniquely predicts 1322 /// alternative. We will not be caching that DFA state and it is a 1323 /// waste to pursue the closure. Might have to advance when we do 1324 /// ambig detection thought :( 1325 /// 1326 final internal func closure(_ config: ATNConfig, 1327 _ configs: ATNConfigSet, 1328 _ closureBusy: inout Set<ATNConfig>, 1329 _ collectPredicates: Bool, 1330 _ fullCtx: Bool, 1331 _ treatEofAsEpsilon: Bool) throws { 1332 let initialDepth = 0 1333 try closureCheckingStopState(config, configs, &closureBusy, collectPredicates, fullCtx, initialDepth, treatEofAsEpsilon) 1334 assert(!fullCtx || !configs.dipsIntoOuterContext, "Expected: !fullCtx||!configs.dipsIntoOuterContext") 1335 } 1336 1337 1338 final internal func closureCheckingStopState(_ config: ATNConfig, 1339 _ configs: ATNConfigSet, 1340 _ closureBusy: inout Set<ATNConfig>, 1341 _ collectPredicates: Bool, 1342 _ fullCtx: Bool, 1343 _ depth: Int, 1344 _ treatEofAsEpsilon: Bool) throws { 1345 1346 if debug { 1347 print("closure(" + config.toString(parser, true) + ")") 1348 } 1349 1350 if config.state is RuleStopState { 1351 let configContext = config.context! 1352 // We hit rule end. If we have context info, use it 1353 // run thru all possible stack tops in ctx 1354 if !configContext.isEmpty() { 1355 let length = configContext.size() 1356 for i in 0..<length { 1357 if configContext.getReturnState(i) == PredictionContext.EMPTY_RETURN_STATE { 1358 if fullCtx { 1359 try! configs.add(ATNConfig(config, config.state, PredictionContext.EMPTY), &mergeCache) 1360 continue 1361 } else { 1362 // we have no context info, just chase follow links (if greedy) 1363 if debug { 1364 print("FALLING off rule\(getRuleName(config.state.ruleIndex!))") 1365 } 1366 try closure_(config, configs, &closureBusy, collectPredicates, 1367 fullCtx, depth, treatEofAsEpsilon) 1368 } 1369 continue 1370 } 1371 let returnState: ATNState = atn.states[configContext.getReturnState(i)]! 1372 let newContext: PredictionContext? = configContext.getParent(i) // "pop" return state 1373 let c: ATNConfig = ATNConfig(returnState, config.alt, newContext, 1374 config.semanticContext) 1375 // While we have context to pop back from, we may have 1376 // gotten that context AFTER having falling off a rule. 1377 // Make sure we track that we are now out of context. 1378 // 1379 // This assignment also propagates the 1380 // isPrecedenceFilterSuppressed() value to the new 1381 // configuration. 1382 c.reachesIntoOuterContext = config.reachesIntoOuterContext 1383 assert(depth > Int.min, "Expected: depth>Integer.MIN_VALUE") 1384 try closureCheckingStopState(c, configs, &closureBusy, collectPredicates, 1385 fullCtx, depth - 1, treatEofAsEpsilon) 1386 } 1387 return 1388 } else if fullCtx { 1389 // reached end of start rule 1390 try! configs.add(config, &mergeCache) 1391 return 1392 } else { 1393 // else if we have no context info, just chase follow links (if greedy) 1394 if debug { 1395 print("FALLING off rule \(getRuleName(config.state.ruleIndex!))") 1396 } 1397 1398 } 1399 } 1400 try closure_(config, configs, &closureBusy, collectPredicates, fullCtx, depth, treatEofAsEpsilon) 1401 } 1402 1403 /// 1404 /// Do the actual work of walking epsilon edges 1405 /// 1406 final internal func closure_(_ config: ATNConfig, 1407 _ configs: ATNConfigSet, 1408 _ closureBusy: inout Set<ATNConfig>, 1409 _ collectPredicates: Bool, 1410 _ fullCtx: Bool, 1411 _ depth: Int, 1412 _ treatEofAsEpsilon: Bool) throws { 1413 // print(__FUNCTION__) 1414 //long startTime = System.currentTimeMillis(); 1415 let p = config.state 1416 // optimization 1417 if !p.onlyHasEpsilonTransitions() { 1418 try! configs.add(config, &mergeCache) 1419 // make sure to not return here, because EOF transitions can act as 1420 // both epsilon transitions and non-epsilon transitions. 1421 // if ( debug ) print("added config "+configs); 1422 } 1423 let length = p.getNumberOfTransitions() 1424 for i in 0..<length { 1425 if i == 0 && 1426 canDropLoopEntryEdgeInLeftRecursiveRule(config) { 1427 continue 1428 } 1429 let t = p.transition(i) 1430 let continueCollecting = !(t is ActionTransition) && collectPredicates 1431 let c = try getEpsilonTarget(config, t, continueCollecting, depth == 0, fullCtx, treatEofAsEpsilon) 1432 if let c = c { 1433 var newDepth = depth 1434 if config.state is RuleStopState { 1435 assert(!fullCtx, "Expected: !fullCtx") 1436 // target fell off end of rule; mark resulting c as having dipped into outer context 1437 // We can't get here if incoming config was rule stop and we had context 1438 // track how far we dip into outer context. Might 1439 // come in handy and we avoid evaluating context dependent 1440 // preds if this is > 0. 1441 if let _dfa = _dfa , _dfa.isPrecedenceDfa() { 1442 let outermostPrecedenceReturn: Int = (t as! EpsilonTransition).outermostPrecedenceReturn() 1443 if outermostPrecedenceReturn == _dfa.atnStartState.ruleIndex { 1444 c.setPrecedenceFilterSuppressed(true) 1445 } 1446 } 1447 1448 c.reachesIntoOuterContext += 1 1449 if closureBusy.contains(c) { 1450 // avoid infinite recursion for right-recursive rules 1451 continue 1452 } else { 1453 closureBusy.insert(c) 1454 } 1455 1456 configs.dipsIntoOuterContext = true // TODO: can remove? only care when we add to set per middle of this method 1457 //print("newDepth=>\(newDepth)") 1458 assert(newDepth > Int.min, "Expected: newDepth>Integer.MIN_VALUE") 1459 newDepth -= 1 1460 1461 if debug { 1462 print("dips into outer ctx: \(c)") 1463 } 1464 } 1465 else { 1466 if !t.isEpsilon() { 1467 if closureBusy.contains(c) { 1468 // avoid infinite recursion for EOF* and EOF+ 1469 continue 1470 } 1471 else { 1472 closureBusy.insert(c) 1473 } 1474 } 1475 1476 if t is RuleTransition { 1477 // latch when newDepth goes negative - once we step out of the entry context we can't return 1478 if newDepth >= 0 { 1479 newDepth += 1 1480 } 1481 } 1482 } 1483 1484 try closureCheckingStopState(c, configs, &closureBusy, continueCollecting, 1485 fullCtx, newDepth, treatEofAsEpsilon) 1486 } 1487 } 1488 //long finishTime = System.currentTimeMillis(); 1489 // if ((finishTime-startTime)>1) 1490 //print("That took: "+(finishTime-startTime)+ " ms"); 1491 } 1492 1493 /// 1494 /// Implements first-edge (loop entry) elimination as an optimization 1495 /// during closure operations. See antlr/antlr4#1398. 1496 /// 1497 /// The optimization is to avoid adding the loop entry config when 1498 /// the exit path can only lead back to the same 1499 /// StarLoopEntryState after popping context at the rule end state 1500 /// (traversing only epsilon edges, so we're still in closure, in 1501 /// this same rule). 1502 /// 1503 /// We need to detect any state that can reach loop entry on 1504 /// epsilon w/o exiting rule. We don't have to look at FOLLOW 1505 /// links, just ensure that all stack tops for config refer to key 1506 /// states in LR rule. 1507 /// 1508 /// To verify we are in the right situation we must first check 1509 /// closure is at a StarLoopEntryState generated during LR removal. 1510 /// Then we check that each stack top of context is a return state 1511 /// from one of these cases: 1512 /// 1513 /// 1. 'not' expr, '(' type ')' expr. The return state points at loop entry state 1514 /// 2. expr op expr. The return state is the block end of internal block of (...)* 1515 /// 3. 'between' expr 'and' expr. The return state of 2nd expr reference. 1516 /// That state points at block end of internal block of (...)*. 1517 /// 4. expr '?' expr ':' expr. The return state points at block end, 1518 /// which points at loop entry state. 1519 /// 1520 /// If any is true for each stack top, then closure does not add a 1521 /// config to the current config set for edge[0], the loop entry branch. 1522 /// 1523 /// Conditions fail if any context for the current config is: 1524 /// 1525 /// a. empty (we'd fall out of expr to do a global FOLLOW which could 1526 /// even be to some weird spot in expr) or, 1527 /// b. lies outside of expr or, 1528 /// c. lies within expr but at a state not the BlockEndState 1529 /// generated during LR removal 1530 /// 1531 /// Do we need to evaluate predicates ever in closure for this case? 1532 /// 1533 /// No. Predicates, including precedence predicates, are only 1534 /// evaluated when computing a DFA start state. I.e., only before 1535 /// the lookahead (but not parser) consumes a token. 1536 /// 1537 /// There are no epsilon edges allowed in LR rule alt blocks or in 1538 /// the "primary" part (ID here). If closure is in 1539 /// StarLoopEntryState any lookahead operation will have consumed a 1540 /// token as there are no epsilon-paths that lead to 1541 /// StarLoopEntryState. We do not have to evaluate predicates 1542 /// therefore if we are in the generated StarLoopEntryState of a LR 1543 /// rule. Note that when making a prediction starting at that 1544 /// decision point, decision d=2, compute-start-state performs 1545 /// closure starting at edges[0], edges[1] emanating from 1546 /// StarLoopEntryState. That means it is not performing closure on 1547 /// StarLoopEntryState during compute-start-state. 1548 /// 1549 /// How do we know this always gives same prediction answer? 1550 /// 1551 /// Without predicates, loop entry and exit paths are ambiguous 1552 /// upon remaining input +b (in, say, a+b). Either paths lead to 1553 /// valid parses. Closure can lead to consuming + immediately or by 1554 /// falling out of this call to expr back into expr and loop back 1555 /// again to StarLoopEntryState to match +b. In this special case, 1556 /// we choose the more efficient path, which is to take the bypass 1557 /// path. 1558 /// 1559 /// The lookahead language has not changed because closure chooses 1560 /// one path over the other. Both paths lead to consuming the same 1561 /// remaining input during a lookahead operation. If the next token 1562 /// is an operator, lookahead will enter the choice block with 1563 /// operators. If it is not, lookahead will exit expr. Same as if 1564 /// closure had chosen to enter the choice block immediately. 1565 /// 1566 /// Closure is examining one config (some loopentrystate, some alt, 1567 /// context) which means it is considering exactly one alt. Closure 1568 /// always copies the same alt to any derived configs. 1569 /// 1570 /// How do we know this optimization doesn't mess up precedence in 1571 /// our parse trees? 1572 /// 1573 /// Looking through expr from left edge of stat only has to confirm 1574 /// that an input, say, a+b+c; begins with any valid interpretation 1575 /// of an expression. The precedence actually doesn't matter when 1576 /// making a decision in stat seeing through expr. It is only when 1577 /// parsing rule expr that we must use the precedence to get the 1578 /// right interpretation and, hence, parse tree. 1579 /// 1580 /// - 4.6 1581 /// canDropLoopEntryEdgeInLeftRecursiveRulenull1582 internal func canDropLoopEntryEdgeInLeftRecursiveRule(_ config: ATNConfig) -> Bool { 1583 if ParserATNSimulator.TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT { 1584 return false 1585 } 1586 let p = config.state 1587 guard let configContext = config.context else { 1588 return false 1589 } 1590 // First check to see if we are in StarLoopEntryState generated during 1591 // left-recursion elimination. For efficiency, also check if 1592 // the context has an empty stack case. If so, it would mean 1593 // global FOLLOW so we can't perform optimization 1594 if p.getStateType() != ATNState.STAR_LOOP_ENTRY || 1595 !( (p as! StarLoopEntryState)).precedenceRuleDecision || // Are we the special loop entry/exit state? 1596 configContext.isEmpty() || // If SLL wildcard 1597 configContext.hasEmptyPath(){ 1598 return false 1599 } 1600 1601 // Require all return states to return back to the same rule 1602 // that p is in. 1603 let numCtxs = configContext.size() 1604 for i in 0 ..< numCtxs { // for each stack context 1605 let returnState = atn.states[configContext.getReturnState(i)]! 1606 if returnState.ruleIndex != p.ruleIndex 1607 {return false} 1608 } 1609 1610 let decisionStartState = (p.transition(0).target as! BlockStartState) 1611 let blockEndStateNum = decisionStartState.endState!.stateNumber 1612 let blockEndState = (atn.states[blockEndStateNum] as! BlockEndState) 1613 1614 // Verify that the top of each stack context leads to loop entry/exit 1615 // state through epsilon edges and w/o leaving rule. 1616 for i in 0 ..< numCtxs { // for each stack context 1617 let returnStateNumber = configContext.getReturnState(i) 1618 let returnState = atn.states[returnStateNumber]! 1619 // all states must have single outgoing epsilon edge 1620 if returnState.getNumberOfTransitions() != 1 || !returnState.transition(0).isEpsilon(){ 1621 return false 1622 } 1623 // Look for prefix op case like 'not expr', (' type ')' expr 1624 let returnStateTarget = returnState.transition(0).target 1625 if returnState.getStateType() == ATNState.BLOCK_END && 1626 returnStateTarget == p { 1627 continue 1628 } 1629 // Look for 'expr op expr' or case where expr's return state is block end 1630 // of (...)* internal block; the block end points to loop back 1631 // which points to p but we don't need to check that 1632 if returnState == blockEndState { 1633 continue 1634 } 1635 // Look for ternary expr ? expr : expr. The return state points at block end, 1636 // which points at loop entry state 1637 if returnStateTarget == blockEndState { 1638 continue 1639 } 1640 // Look for complex prefix 'between expr and expr' case where 2nd expr's 1641 // return state points at block end state of (...)* internal block 1642 if returnStateTarget.getStateType() == ATNState.BLOCK_END && 1643 returnStateTarget.getNumberOfTransitions() == 1 && 1644 returnStateTarget.transition(0).isEpsilon() && 1645 returnStateTarget.transition(0).target == p{ 1646 continue 1647 } 1648 1649 // anything else ain't conforming 1650 return false 1651 } 1652 1653 return true 1654 } 1655 getRuleNamenull1656 open func getRuleName(_ index: Int) -> String { 1657 if index >= 0 { 1658 return parser.getRuleNames()[index] 1659 } 1660 return "<rule \(index)>" 1661 } 1662 1663 1664 final func getEpsilonTarget(_ config: ATNConfig, 1665 _ t: Transition, 1666 _ collectPredicates: Bool, 1667 _ inContext: Bool, 1668 _ fullCtx: Bool, 1669 _ treatEofAsEpsilon: Bool) throws -> ATNConfig? { 1670 switch t.getSerializationType() { 1671 case Transition.RULE: 1672 return ruleTransition(config, t as! RuleTransition) 1673 1674 case Transition.PRECEDENCE: 1675 return try precedenceTransition(config, t as! PrecedencePredicateTransition, collectPredicates, inContext, fullCtx) 1676 1677 case Transition.PREDICATE: 1678 return try predTransition(config, t as! PredicateTransition, 1679 collectPredicates, 1680 inContext, 1681 fullCtx) 1682 1683 case Transition.ACTION: 1684 return actionTransition(config, t as! ActionTransition) 1685 1686 case Transition.EPSILON: 1687 return ATNConfig(config, t.target) 1688 1689 case Transition.ATOM: fallthrough 1690 case Transition.RANGE: fallthrough 1691 case Transition.SET: 1692 // EOF transitions act like epsilon transitions after the first EOF 1693 // transition is traversed 1694 if treatEofAsEpsilon { 1695 if t.matches(CommonToken.EOF, 0, 1) { 1696 return ATNConfig(config, t.target) 1697 } 1698 } 1699 1700 return nil 1701 1702 default: 1703 return nil 1704 } 1705 1706 //return nil; 1707 1708 } 1709 1710 actionTransitionnull1711 final func actionTransition(_ config: ATNConfig, _ t: ActionTransition) -> ATNConfig { 1712 if debug { 1713 print("ACTION edge \(t.ruleIndex):\(t.actionIndex)") 1714 } 1715 return ATNConfig(config, t.target) 1716 } 1717 1718 1719 final func precedenceTransition(_ config: ATNConfig, 1720 _ pt: PrecedencePredicateTransition, 1721 _ collectPredicates: Bool, 1722 _ inContext: Bool, 1723 _ fullCtx: Bool) throws -> ATNConfig? { 1724 if debug { 1725 print("PRED (collectPredicates=\(collectPredicates)) \(pt.precedence)>=_p, ctx dependent=true") 1726 //if ( parser != nil ) { 1727 print("context surrounding pred is \(parser.getRuleInvocationStack())") 1728 // } 1729 } 1730 1731 var c: ATNConfig? = nil 1732 if collectPredicates && inContext { 1733 if fullCtx { 1734 // In full context mode, we can evaluate predicates on-the-fly 1735 // during closure, which dramatically reduces the size of 1736 // the config sets. It also obviates the need to test predicates 1737 // later during conflict resolution. 1738 let currentPosition = _input.index() 1739 try _input.seek(_startIndex) 1740 let predSucceeds = try evalSemanticContext(pt.getPredicate(), _outerContext, config.alt, fullCtx) 1741 try _input.seek(currentPosition) 1742 if predSucceeds { 1743 c = ATNConfig(config, pt.target) // no pred context 1744 } 1745 } 1746 else { 1747 let newSemCtx = SemanticContext.and(config.semanticContext, pt.getPredicate()) 1748 c = ATNConfig(config, pt.target, newSemCtx) 1749 } 1750 } 1751 else { 1752 c = ATNConfig(config, pt.target) 1753 } 1754 1755 if debug { 1756 print("config from pred transition=\(c?.description ?? "nil")") 1757 } 1758 return c 1759 } 1760 1761 1762 final func predTransition(_ config: ATNConfig, 1763 _ pt: PredicateTransition, 1764 _ collectPredicates: Bool, 1765 _ inContext: Bool, 1766 _ fullCtx: Bool) throws -> ATNConfig? { 1767 if debug { 1768 print("PRED (collectPredicates=\(collectPredicates)) \(pt.ruleIndex):\(pt.predIndex), ctx dependent=\(pt.isCtxDependent)") 1769 //if ( parser != nil ) { 1770 print("context surrounding pred is \(parser.getRuleInvocationStack())") 1771 //} 1772 } 1773 1774 var c: ATNConfig? = nil 1775 if collectPredicates && 1776 (!pt.isCtxDependent || (pt.isCtxDependent && inContext)) { 1777 if fullCtx { 1778 // In full context mode, we can evaluate predicates on-the-fly 1779 // during closure, which dramatically reduces the size of 1780 // the config sets. It also obviates the need to test predicates 1781 // later during conflict resolution. 1782 let currentPosition = _input.index() 1783 try _input.seek(_startIndex) 1784 let predSucceeds = try evalSemanticContext(pt.getPredicate(), _outerContext, config.alt, fullCtx) 1785 try _input.seek(currentPosition) 1786 if predSucceeds { 1787 c = ATNConfig(config, pt.target) // no pred context 1788 } 1789 } else { 1790 let newSemCtx = SemanticContext.and(config.semanticContext, pt.getPredicate()) 1791 c = ATNConfig(config, pt.target, newSemCtx) 1792 } 1793 } else { 1794 c = ATNConfig(config, pt.target) 1795 } 1796 1797 if debug { 1798 print("config from pred transition=\(c?.description ?? "nil")") 1799 } 1800 return c 1801 } 1802 1803 ruleTransitionnull1804 final func ruleTransition(_ config: ATNConfig, _ t: RuleTransition) -> ATNConfig { 1805 if debug { 1806 print("CALL rule \(getRuleName(t.target.ruleIndex!)), ctx=\(config.context?.description ?? "nil")") 1807 } 1808 1809 let returnState = t.followState 1810 let newContext = SingletonPredictionContext.create(config.context, returnState.stateNumber) 1811 return ATNConfig(config, t.target, newContext) 1812 } 1813 1814 /// 1815 /// Gets a _java.util.BitSet_ containing the alternatives in `configs` 1816 /// which are part of one or more conflicting alternative subsets. 1817 /// 1818 /// - parameter configs: The _org.antlr.v4.runtime.atn.ATNConfigSet_ to analyze. 1819 /// - returns: The alternatives in `configs` which are part of one or more 1820 /// conflicting alternative subsets. If `configs` does not contain any 1821 /// conflicting subsets, this method returns an empty _java.util.BitSet_. 1822 /// getConflictingAltsnull1823 final func getConflictingAlts(_ configs: ATNConfigSet) -> BitSet { 1824 let altsets = PredictionMode.getConflictingAltSubsets(configs) 1825 return PredictionMode.getAlts(altsets) 1826 } 1827 1828 /// 1829 /// Sam pointed out a problem with the previous definition, v3, of 1830 /// ambiguous states. If we have another state associated with conflicting 1831 /// alternatives, we should keep going. For example, the following grammar 1832 /// 1833 /// s : (ID | ID ID?) ';' ; 1834 /// 1835 /// When the ATN simulation reaches the state before ';', it has a DFA 1836 /// state that looks like: [12|1|[], 6|2|[], 12|2|[]]. Naturally 1837 /// 12|1|[] and 12|2|[] conflict, but we cannot stop processing this node 1838 /// because alternative to has another way to continue, via [6|2|[]]. 1839 /// The key is that we have a single state that has config's only associated 1840 /// with a single alternative, 2, and crucially the state transitions 1841 /// among the configurations are all non-epsilon transitions. That means 1842 /// we don't consider any conflicts that include alternative 2. So, we 1843 /// ignore the conflict between alts 1 and 2. We ignore a set of 1844 /// conflicting alts when there is an intersection with an alternative 1845 /// associated with a single alt state in the state→config-list map. 1846 /// 1847 /// It's also the case that we might have two conflicting configurations but 1848 /// also a 3rd nonconflicting configuration for a different alternative: 1849 /// [1|1|[], 1|2|[], 8|3|[]]. This can come about from grammar: 1850 /// 1851 /// a : A | A | A B ; 1852 /// 1853 /// After matching input A, we reach the stop state for rule A, state 1. 1854 /// State 8 is the state right before B. Clearly alternatives 1 and 2 1855 /// conflict and no amount of further lookahead will separate the two. 1856 /// However, alternative 3 will be able to continue and so we do not 1857 /// stop working on this state. In the previous example, we're concerned 1858 /// with states associated with the conflicting alternatives. Here alt 1859 /// 3 is not associated with the conflicting configs, but since we can continue 1860 /// looking for input reasonably, I don't declare the state done. We 1861 /// ignore a set of conflicting alts when we have an alternative 1862 /// that we still need to pursue. 1863 /// getConflictingAltsOrUniqueAltnull1864 final func getConflictingAltsOrUniqueAlt(_ configs: ATNConfigSet) -> BitSet { 1865 var conflictingAlts: BitSet 1866 if configs.uniqueAlt != ATN.INVALID_ALT_NUMBER { 1867 conflictingAlts = BitSet() 1868 try! conflictingAlts.set(configs.uniqueAlt) 1869 } else { 1870 conflictingAlts = configs.conflictingAlts! 1871 } 1872 return conflictingAlts 1873 } 1874 1875 getTokenNamenull1876 public final func getTokenName(_ t: Int) -> String { 1877 if t == CommonToken.EOF { 1878 return "EOF" 1879 } 1880 let vocabulary = parser.getVocabulary() 1881 let displayName = vocabulary.getDisplayName(t) 1882 if displayName == String(t) { 1883 return displayName 1884 } 1885 1886 return "\(displayName) <\(t)>" 1887 } 1888 getLookaheadNamenull1889 public final func getLookaheadName(_ input: TokenStream) throws -> String { 1890 return try getTokenName(input.LA(1)) 1891 } 1892 1893 /// 1894 /// Used for debugging in adaptivePredict around execATN but I cut 1895 /// it out for clarity now that alg. works well. We can leave this 1896 /// "dead" code for a bit. 1897 /// dumpDeadEndConfigsnull1898 public final func dumpDeadEndConfigs(_ nvae: NoViableAltException) { 1899 errPrint("dead end configs: ") 1900 for c in nvae.getDeadEndConfigs()!.configs { 1901 var trans = "no edges" 1902 if c.state.getNumberOfTransitions() > 0 { 1903 let t = c.state.transition(0) 1904 if let at = t as? AtomTransition { 1905 trans = "Atom " + getTokenName(at.label) 1906 } 1907 else if let st = t as? SetTransition { 1908 let not = st is NotSetTransition 1909 trans = (not ? "~" : "") + "Set " + st.set.description 1910 } 1911 } 1912 errPrint("\(c.toString(parser, true)):\(trans)") 1913 } 1914 } 1915 1916 1917 final func noViableAlt(_ input: TokenStream, 1918 _ outerContext: ParserRuleContext, 1919 _ configs: ATNConfigSet, 1920 _ startIndex: Int) -> NoViableAltException { 1921 let startToken = try! input.get(startIndex) 1922 var offendingToken: Token? = nil 1923 do { 1924 offendingToken = try input.LT(1) 1925 } 1926 catch { 1927 } 1928 return NoViableAltException(parser, input, startToken, offendingToken, configs, outerContext) 1929 } 1930 getUniqueAltnull1931 internal static func getUniqueAlt(_ configs: ATNConfigSet) -> Int { 1932 let alt = configs.getUniqueAlt() 1933 return alt 1934 } 1935 1936 /// 1937 /// Add an edge to the DFA, if possible. This method calls 1938 /// _#addDFAState_ to ensure the `to` state is present in the 1939 /// DFA. If `from` is `null`, or if `t` is outside the 1940 /// range of edges that can be represented in the DFA tables, this method 1941 /// returns without adding the edge to the DFA. 1942 /// 1943 /// If `to` is `null`, this method returns `null`. 1944 /// Otherwise, this method returns the _org.antlr.v4.runtime.dfa.DFAState_ returned by calling 1945 /// _#addDFAState_ for the `to` state. 1946 /// 1947 /// - parameter dfa: The DFA 1948 /// - parameter from: The source state for the edge 1949 /// - parameter t: The input symbol 1950 /// - parameter to: The target state for the edge 1951 /// 1952 /// - returns: the result of calling _#addDFAState_ on `to` 1953 /// 1954 @discardableResult 1955 private final func addDFAEdge(_ dfa: DFA, 1956 _ from: DFAState, 1957 _ t: Int, 1958 _ to: DFAState) -> DFAState { 1959 var to = to 1960 if debug { 1961 print("EDGE \(from) -> \(to) upon \(getTokenName(t))") 1962 } 1963 1964 to = addDFAState(dfa, to) // used existing if possible not incoming 1965 if t < -1 || t > atn.maxTokenType { 1966 return to 1967 } 1968 dfaStateMutex.synchronized { 1969 [unowned self] in 1970 if from.edges == nil { 1971 from.edges = [DFAState?](repeating: nil, count: self.atn.maxTokenType + 1 + 1) //new DFAState[atn.maxTokenType+1+1]; 1972 } 1973 1974 from.edges[t + 1] = to // connect 1975 } 1976 1977 if debug { 1978 print("DFA=\n" + dfa.toString(parser.getVocabulary())) 1979 } 1980 1981 return to 1982 } 1983 1984 /// 1985 /// Add state `D` to the DFA if it is not already present, and return 1986 /// the actual instance stored in the DFA. If a state equivalent to `D` 1987 /// is already in the DFA, the existing state is returned. Otherwise this 1988 /// method returns `D` after adding it to the DFA. 1989 /// 1990 /// If `D` is _#ERROR_, this method returns _#ERROR_ and 1991 /// does not change the DFA. 1992 /// 1993 /// - parameter dfa: The dfa 1994 /// - parameter D: The DFA state to add 1995 /// - returns: The state stored in the DFA. This will be either the existing 1996 /// state if `D` is already in the DFA, or `D` itself if the 1997 /// state was not already present. 1998 /// addDFAStatenull1999 private final func addDFAState(_ dfa: DFA, _ D: DFAState) -> DFAState { 2000 if D == ATNSimulator.ERROR { 2001 return D 2002 } 2003 2004 return dfaStatesMutex.synchronized { 2005 if let existing = dfa.states[D] { 2006 return existing 2007 } 2008 2009 D.stateNumber = dfa.states.count 2010 2011 if !D.configs.isReadonly() { 2012 try! D.configs.optimizeConfigs(self) 2013 D.configs.setReadonly(true) 2014 } 2015 2016 dfa.states[D] = D 2017 if debug { 2018 print("adding new DFA state: \(D)") 2019 } 2020 2021 return D 2022 } 2023 } 2024 reportAttemptingFullContextnull2025 func reportAttemptingFullContext(_ dfa: DFA, _ conflictingAlts: BitSet?, _ configs: ATNConfigSet, _ startIndex: Int, _ stopIndex: Int) { 2026 if debug || retry_debug { 2027 let input = getTextInInterval(startIndex, stopIndex) 2028 print("reportAttemptingFullContext decision=\(dfa.decision):\(configs), input=\(input)") 2029 } 2030 parser.getErrorListenerDispatch().reportAttemptingFullContext(parser, dfa, startIndex, stopIndex, conflictingAlts, configs) 2031 } 2032 reportContextSensitivitynull2033 func reportContextSensitivity(_ dfa: DFA, _ prediction: Int, _ configs: ATNConfigSet, _ startIndex: Int, _ stopIndex: Int) { 2034 if debug || retry_debug { 2035 let input = getTextInInterval(startIndex, stopIndex) 2036 print("reportContextSensitivity decision=\(dfa.decision):\(configs), input=\(input)") 2037 } 2038 parser.getErrorListenerDispatch().reportContextSensitivity(parser, dfa, startIndex, stopIndex, prediction, configs) 2039 } 2040 2041 /// 2042 /// If context sensitive parsing, we know it's ambiguity not conflict 2043 /// 2044 // configs that LL not SLL considered conflictin 2045 internal func reportAmbiguity(_ dfa: DFA, 2046 _ D: DFAState, // the DFA state from execATN() that had SLL conflicts 2047 _ startIndex: Int, _ stopIndex: Int, 2048 _ exact: Bool, 2049 _ ambigAlts: BitSet, 2050 _ configs: ATNConfigSet) 2051 { 2052 if debug || retry_debug { 2053 let input = getTextInInterval(startIndex, stopIndex) 2054 print("reportAmbiguity \(ambigAlts):\(configs), input=\(input)") 2055 } 2056 parser.getErrorListenerDispatch().reportAmbiguity(parser, dfa, startIndex, stopIndex, 2057 exact, ambigAlts, configs) 2058 } 2059 getTextInIntervalnull2060 private func getTextInInterval(_ startIndex: Int, _ stopIndex: Int) -> String { 2061 let interval = Interval.of(startIndex, stopIndex) 2062 do { 2063 return try parser.getTokenStream()?.getText(interval) ?? "<unknown>" 2064 } 2065 catch { 2066 return "<unknown>" 2067 } 2068 } 2069 setPredictionModenull2070 public final func setPredictionMode(_ mode: PredictionMode) { 2071 self.mode = mode 2072 } 2073 2074 getPredictionModenull2075 public final func getPredictionMode() -> PredictionMode { 2076 return mode 2077 } 2078 getParsernull2079 public final func getParser() -> Parser { 2080 return parser 2081 } 2082 } 2083