1 /* 2 Copyright (c) 2015, 2020, Oracle and/or its affiliates. 3 4 This program is free software; you can redistribute it and/or modify 5 it under the terms of the GNU General Public License, version 2.0, 6 as published by the Free Software Foundation. 7 8 This program is also distributed with certain software (including 9 but not limited to OpenSSL) that is licensed under separate terms, 10 as designated in a particular file or component or in included license 11 documentation. The authors of MySQL hereby grant you an additional 12 permission to link the program and your derivative works with the 13 separately licensed software that they have included with MySQL. 14 15 This program is distributed in the hope that it will be useful, 16 but WITHOUT ANY WARRANTY; without even the implied warranty of 17 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 18 GNU General Public License for more details. 19 20 You should have received a copy of the GNU General Public License 21 along with this program; if not, write to the Free Software 22 Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA 23 */ 24 25 /** 26 * @class mysql_harness::Loader 27 * 28 * @ingroup Loader 29 * 30 * 31 * 32 * ## Introduction 33 * 34 * The loader class is responsible for managing the life-cycle of 35 * plugins in the harness. Each plugin goes through seven steps in the 36 * life-cycle, of which steps #2, #3, #5 and #6 are optional: 37 * 38 * 1. Loading 39 * 2. Initialization 40 * 3. Starting 41 * 4. Running 42 * 5. Stopping 43 * 6. Deinitialization 44 * 7. Unloading 45 * 46 * ## Overview of Life-cycle Steps 47 * 48 * ### 1. Loading ### 49 * 50 * When *loading*, the plugin is loaded using the dynamic library 51 * support available on the operating system. Symbols are evaluated 52 * lazily (for example, the `RTLD_LAZY` flag is used for `dlopen`) to 53 * allow plugins to be loaded in any order. The symbols that are 54 * exported by the plugin are made available to all other plugins 55 * loaded (flag `RTLD_GLOBAL` to `dlopen`). 56 * 57 * As part of the loading procedure, the *plugin structure* (see 58 * Plugin class) is fetched from the module and used for the four 59 * optional steps below. 60 * 61 * 62 * 63 * ### 2. Initialization ### 64 * 65 * After all the plugins are successfully loaded, each plugin is given 66 * a chance to perform initialization. This step is only executed if 67 * the plugin structure defines an `init` function. Note that it is 68 * guaranteed that the init function of a plugin is called *after* the 69 * `init` function of all plugins it requires have been called. The 70 * list of these dependencies is specified via `requires` field of the 71 * `Plugin` struct. 72 * 73 * @note if some plugin `init()` function fails, any plugin `init()` 74 * functions schedulled to run after will not run, and harness will 75 * proceed straight to deinitialization step, bypassing calling 76 * `start()` and `stop()` functions. 77 * 78 * 79 * 80 * ### 3. Starting ### 81 * After all plugins have been successfully initialized, a thread is 82 * created for each plugin that has a non-NULL `start` field in the 83 * plugin structure. The threads are started in an arbitrary order, 84 * so you have to be careful about not assuming that, for example, 85 * other plugins required by the plugin have started their thread. If 86 * the plugin does not define a `start` function, no thread is created. 87 * There is a "running" flag associated with each such thread; this 88 * flag is set when the thread starts but before the `start` function 89 * is called. If necessary, the plugin can spawn more threads using 90 * standard C++11 thread calls, however, these threads should not 91 * call harness API functions. 92 * 93 * 94 * 95 * ### 4. Running ### 96 * After starting all plugins (that needed to be started), the harness 97 * will enter the *running step*. This is the "normal" phase, where the 98 * application spends most of its lifetime (application and plugins 99 * service requests or do whatever it is they do). Harness will remain 100 * in this step until one of two things happen: 101 * 102 * 1. shutdown signal is received by the harness 103 * 2. one of the plugins exits with error 104 * 105 * When one of these two events occurrs, harness progresses to the 106 * next step. 107 * 108 * 109 * 110 * ### 5. Stopping ### 111 * In this step, harness "tells" plugins running `start()` to exit this 112 * function by clearing the "running" flag. It also invokes `stop()` 113 * function for all plugins that provided it. It then waits for all 114 * running plugin threads to exit. 115 * 116 * @note under certain circumstances, `stop()` may overlap execution 117 * with `start()`, or even be called before `start()`. 118 * 119 * 120 * 121 * ### 6. Deinitialization ### 122 * After all threads have stopped, regardless of whether they stopped 123 * with an error or not, the plugins are deinitialized in reverse order 124 * of initialization by calling the function in the `deinit` field of 125 * the `Plugin` structure. Regardless of whether the `deinit()` functions 126 * return an error or not, all plugins schedulled for deinitialisation 127 * will be deinitialized. 128 * 129 * @note for any `init()` functions that failed, `deinit()` functions 130 * will not run. 131 * @note plugins may have a `deinit()` function despite not having a 132 * corresponding `init()`. In such cases, the missing `init()` is 133 * treated as if it existed and ran successfully. 134 * 135 * 136 * 137 * ### 7. Unloading ### 138 * After a plugin has deinitialized, it can be unloaded. It is 139 * guaranteed that no module is unloaded before it has been 140 * deinitialized. 141 * 142 * @note This step is currently unimplemented - meaning, it does nothing. 143 * The plugins will remain loaded in memory until the process shuts 144 * down. This makes no practical difference on application behavior 145 * at present, but might be needed if Harness gained ability to 146 * reload plugins in the future. 147 148 ## Behavior Diagrams 149 150 Previous section described quickly each step of the life-cycle process. In this 151 section, two flow charts are presented which show the operation of all seven 152 steps. First shows a high-level overview, and the second shows all 7 life-cycle 153 steps in more detail. Discussion of details follows in the following sections. 154 155 Some points to keep in mind while viewing the diagrams: 156 157 - diagrams describe code behavior rather than implementation. So for example: 158 - pseudocode does not directly correspond 1:1 to real code. However, it 159 behaves exactly like the real code. 160 161 - seven life-cycle functions shown are actual functions (@c Loader's methods, 162 to be more precise) 163 - load_all(), init_all(), start_all(), main_loop(), stop_all(), deinit_all() 164 are implemented functions (first 6 steps of life-cycle) 165 - unload_all() is the 7th step of life-cycle, but it's currently unimplemented 166 167 - when plugin functions exit with error, they do so by calling 168 set_error() before exiting 169 170 - some things are not shown to keep the diagram simple: 171 - first error returned by any of the 7 life-cycle functions is 172 saved and passed at the end of life-cycle flow to the calling code 173 174 175 ### Overview 176 177 @verbatim 178 179 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 180 \ \ 181 \ START \ 182 \ | \ 183 \ | \ 184 \ | \ 185 \ V \ 186 \ [load_all()] \ 187 \ | \ 188 \ V \ 189 \ <LOAD_OK?> \ 190 \ | | \ 191 \ +---N Y \ 192 \ | | \ 193 \ | v \ 194 \ | [init_all()] \ 195 \ | | \ 196 \ | v \ 197 \ | <INIT_OK?> ( each plugin runs ) \ 198 \ | | | (in a separate thread) \ 199 \ | N Y \ 200 \ | | | [plugin[1]->start()] \ 201 \ | | v start plugin threads [plugin[2]->start()] \ 202 \ | | [start_all()] - - - - - - - - - - - - - - - - ->[ .. .. ] \ 203 \ | | | [ .. .. ] \ 204 \ | | | + - - - - - - - - - - - - - - - - - - - - -[plugin[n]->start()] \ 205 \ | | | notification when each ^ \ 206 \ | | | | thread terminates \ 207 \ | | | | \ 208 \ | | | | stop plugin \ 209 \ | | | | threads \ 210 \ | | | | \ 211 \ | | | | \ 212 \ | | v v \ 213 \ | | [main_loop()]= call ==>[stop_all()] - - - - - - - - - - - + \ 214 \ | | | \ 215 \ | | | \ \ 216 \ | *<--+ \ \ 217 \ | | \__ waits for all plugin \ 218 \ | v threads to terminate \ 219 \ | [deinit_all()] \ 220 \ | | \ 221 \ | v \ 222 \ +-->* \ 223 \ | \ 224 \ v \ 225 \ [unload_all()] \ 226 \ | \ 227 \ | \ \ 228 \ | \ \ 229 \ v \__ currently not implemented \ 230 \ END \ 231 \ \ 232 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 233 234 @endverbatim 235 236 237 ### Detailed View 238 239 @verbatim 240 241 START 242 | 243 | 244 v 245 \\\\ load_all() \\\\\\\\\\\\\\\\\\\\\\ 246 \ \ 247 \ LOAD_OK = true \ 248 \ foreach plugin: \ 249 \ load plugin \ 250 \ if (exit_status != ok): \ 251 \ LOAD_OK = false \ 252 \ break loop \ 253 \ \ 254 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 255 | 256 | 257 v 258 <LOAD_OK?> 259 | | 260 Y N----> unload_all() (see further down) 261 | 262 | 263 \\\\\\\\\\\\\\|\\\ init_all() \\\\\\\\\\\\\\\\\ 264 \ | \ 265 \ v \ 266 \ [INIT_OK = true, i = 1] \ 267 \ | \ 268 \ v \ 269 \ +----><plugin[i] exists?> \ 270 \ | | | \ 271 \ [i++] Y N---------------------+ \ 272 \ ^ | | \ 273 \ | | | \ 274 \ | v | \ 275 \ | <plugin[i] has init()?> | \ 276 \ | | | | \ 277 \ | N Y---+ | \ 278 \ | | | | \ 279 \ | | | | \ 280 \ | | v | \ 281 \ | | [plugin[i]->init()] | \ 282 \ | | | | \ 283 \ | | | | \ 284 \ | | | | \ 285 \ | | | | \ 286 \ | | v | \ 287 \ | | <exit ok?> | \ 288 \ | v | | | \ 289 \ +-------*<------Y N | \ 290 \ | | \ 291 \ | | \ 292 \ v | \ 293 \ [INIT_OK = false] | \ 294 \ | | \ 295 \ v | \ 296 \ *<----------------+ \ 297 \ | \ 298 \ v \ 299 \ [LAST_PLUGIN = i-1] \ 300 \ | \ 301 \\\\\\\\\\\\\\\\\\\\\\|\\\\\\\\\\\\\\\\\\\\\\\\ 302 | 303 | 304 v 305 <INIT_OK?> 306 | | 307 Y N----> deinit_all() (see further down) 308 | 309 | 310 v 311 \\\\ start_all() \\\\\\\\\\\\\\\\\\\\\\\\\ 312 \ \ 313 \ for i = 1 to LAST_PLUGIN: \ 314 \ if plugin[i] has start(): \ start start() in new thread 315 \ new thread(plugin[i]->start()) - - - - - - - - - - - - - - - - + 316 \ \ 317 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ | 318 | 319 | | 320 +-----------------+ 321 | | 322 \\\\|\\\ main_loop() \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\ 323 | | \ 324 v \ 325 +-->* | \ 326 | | \ 327 | v | \ 328 | <any plugin threads running?> \ 329 | | | | \ 330 | N Y---+ \ 331 | | | | \ 332 | | <shutdown signal received && stop_all() not called yet?> \ 333 | | | | | \ 334 | | N Y \ 335 | | | == call ==>[stop_all()]- - - - - - - - - - - - - + | \ 336 | | | | tell (each) start() to exit \ 337 | | *<--+ | | \ 338 | | | \ 339 | | | v v \ 340 | | | [plugin[1]->start()] \ 341 | | v (one) plugin thread exits [plugin[2]->start()] \ 342 | | [wait for (any)]<- - - - - - - - - - - - - - - -[ .. .. ] \ 343 | | [ thread exit ] [ .. .. ] \ 344 | | | [plugin[n]->start()] \ 345 | | | ^ \ 346 | | | \ 347 | | v | \ 348 | | <thread exit ok?> \ 349 | | | | | \ 350 | | Y N---+ \ 351 | | | | | \ 352 | | | v \ 353 | | | <stop_all() called already?> | \ 354 | | v | | \ 355 | | *<------Y N tell (each) \ 356 | | | = call ==+ start() to exit \ 357 | | | | | | \ 358 | | v | | \ 359 +---|-------*<----------+ *==>[stop_all()]- - - - - - - - + \ 360 | | \ 361 | | | \ 362 v | | \ 363 <stop_all() called already?> | | \ 364 | | | | \ 365 Y N | | \ 366 | == call =================+ | \ 367 | | | \ 368 *---+ | \ 369 | | \ 370 \\\\|\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\|\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 371 | | 372 | | 373 v | 374 *<---- init_all() (if !INIT_OK) | 375 | | 376 | | 377 v | 378 \\\\ deinit_all() \\\\\\\\\\\\\\\\\\\\ | 379 \ \ | 380 \ for i = LAST_PLUGIN to 1: \ | 381 \ if plugin[i] has deinit(): \ | 382 \ plugin[i]->deinit() \ | 383 \ if (exit_status != ok): \ | 384 \ # ignore error \ | 385 \ \ | 386 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ | 387 | | 388 | | 389 v | 390 *<---- load_all() (if !LOAD_OK) | 391 | | 392 v | 393 \\\\ unload_all() \\\\\\\\\\\\\\\\\\\\ | 394 \ \ | 395 \ no-op (currently unimplemented) \ | 396 \ \ | 397 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ | 398 | | 399 | | 400 v / 401 END / 402 / 403 / 404 / ( each plugin runs ) 405 / (in a separate thread) 406 \\\\ stop_all() \\\\\\\\\\\\\\\\\\\\\\ run_flag == false 407 \ \ tells start() [plugin[1]->start()] 408 \ for i = 1 to LAST_PLUGIN: \ to exit [plugin[2]->start()] 409 \ run_flag[i] = false - - - - - - - - - - - - - - - - ->[ .. .. ] 410 \ if plugin[i] has stop(): \ [ .. .. ] 411 \ plugin[i]->stop() \ [ .. .. ] 412 \ if (exit_status != ok): \ [plugin[n]->start()] 413 \ # ignore error \ 414 \ \ 415 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 416 417 @endverbatim 418 419 420 421 422 ## Discussion 423 424 ### Persistence (definition) 425 426 Before continuing, we need to define the word "persist", used later on. When we 427 say "persist", we'll mean the opposite of "exit". So when we say a function or 428 a thread persists, it means that it's not returning/exiting, but instead is 429 running some forever-loop or is blocked on something, etc. What it's doing 430 exactly doesn't matter, what matters, is that it hasn't terminated but 431 "lives on" (in case of a function, to "persist" means the same as to "block", 432 but for threads that might sound confusing, this is why we need a new word). 433 So when we call start() in a new thread, it will either run and keep running 434 (thread will persist), or it will run briefly and return (thread will exit). 435 In short, "to persist" means the opposite of "to finish running and exit". 436 437 ### Plugin API functions 438 439 Each plugin can define none, any or all of the following 4 callbacks. 440 They're function pointers, that can be null if not implemented. They're 441 typically called in the order as listed below (under certain circumstances, 442 stop() may overlap execution with start(), or even be called before start()). 443 444 - init() -- called inside of main thread 445 - start() -- main thread creates a new thread, then calls this 446 - stop() -- called inside of main thread 447 - deinit() -- called inside of main thread 448 449 ### Starting and Stopping: Start() ### 450 451 It is typical to implement start function in such a way that it will 452 "persist" (i.e. it will run some forever-loop processing requests 453 rather than exit briefly after being called). In such case, Harness 454 must have a way to terminate it during shutdown operation. 455 456 For this purpose, Harness exposes a boolean "running" flag to each plugin, which 457 serves as means to communicate the need to shutdown; it is read by 458 `is_running()` function. This function should be routinely polled by plugin's 459 `start()` function to determine if it should shut down, and once it returns 460 false, plugin `start()` should terminate as soon as possible. Failure to 461 terminate will block Harness from progressing further in its shutdown procedure, 462 resulting in application "hanging" during shutdown. Typically, `start()` 463 would be implemented more-or-less like so: 464 465 void start() 466 { 467 // run-once code 468 469 while (is_running()) 470 { 471 // forever-loop code 472 } 473 474 // clean-up code 475 } 476 477 There is also an alternative blocking function available, `wait_for_stop()`, 478 should that be better suited for the particular plugin design. Instead of 479 quickly returning a boolean flag, it will block (with an optional timeout) 480 until Harness flags to shut down this plugin. It is an efficient functional 481 equivalent of: 482 483 while (is_running()) 484 { 485 // sleep a little or break on timeout 486 } 487 488 When entering shutdown phase, Harness will notify all plugins to shut down 489 via mechanisms described above. It is also permitted for plugins to exit on 490 their own, whether due to error or intended behavior, without consulting 491 this "running" flag. Polling the "running" flag is only needed when `start()` 492 "persists" and does not normally exit until told to do so. 493 494 Also, in some designs, `start()` function might find it convenient to be able to 495 set the "running" flag to false, in order to trigger its own shutdown in another 496 piece of code. For such cases, `clear_running()` function is provided, which 497 will do exactly that. 498 499 IMPORTANT! Please note that all 3 functions described above (`is_running()`, 500 `wait_for_stop()` and `clear_running()`) can only be called from a thread 501 running `start()` function. If `start()` spawns more theads, these 502 functions CANNOT be called from them. These functions also cannot be called 503 from the other three plugin functions (`init()`, `stop()` and `deinit()`). 504 505 ### Starting and Stopping: Stop() ### 506 507 During shutdown, or after plugin `start()` function exits (whichever comes 508 first), plugin's `stop()` function will be called, if defined. 509 510 IMPORTANT: `start()` function runs in a different thread than `stop()` 511 function. By the time `stop()` runs, depending on the circumstances, 512 `start()` thread may or may not exist. 513 514 IMPORTANT: `stop()` will always be called during shutdown, regardless of whether 515 start() exited with error, exited successfully or is still running. `stop()` 516 must be able to deal with all 3 scenarios. The rationale for this design 517 decision is given Error Handling section. 518 519 520 521 ### Persistence in Plugin Functions ### 522 523 While start() may persist, the other three functions (init(), stop() and 524 deinit()) must obviously not persist, since they run in the main thread. 525 Any blocking behavior exhibited in these functions (caused by a bug or 526 otherwise) will cause the entire application to hang, as will start() that 527 does not poll and/or honor is_running() flag. 528 529 530 531 ### Returning Success/Failure from Plugin Function ### 532 533 Harness expects all four plugin functions (`init(), `start()`, `stop()` and 534 `deinit()`) to notify it in case of an error. This is done via function: 535 536 set_error(PluginFuncEnv* env, ErrorType error, const char* format, ...); 537 538 Calling this function flags that the function has failed, and passes the 539 error type and string back to Harness. The converse is also true: not 540 calling this function prior to exiting the function implies success. 541 This distinction is important, because Harness may take certain actions 542 based on the status returned by each function. 543 544 IMPORTANT! Throwing exceptions from these functions is not supported. 545 If your plugin uses exceptions internally, that is fine, but please 546 ensure they are handled before reaching the Harness-Plugin boundary. 547 548 549 ### Threading Concerns ### 550 551 For each plugin (independent of other plugins): 552 Of the 4 plugin functions, `init()` runs first. It is guaranteed that 553 it will exit before `start()` and `stop()` are called. `start()` and 554 `stop()` can be called in parallel to each other, in any order, with 555 their lifetimes possibly overlapping. They are guaranteed to both have 556 exited before `deinit()` is called. 557 558 If any of the 4 plugin functions spawn any additional threads, Harness 559 makes no provisions for interacting with them in any way: calling 560 Harness functions from them is not supported in particular; also such 561 threads should exit before their parent function finishes running. 562 563 564 565 ### Error Handling ### 566 567 NOTE: WL#9558 HLD version of this section additionally discusses design, 568 rationale for the approach chosen, etc; look there if interested. 569 570 When plugin functions enounter an error, they are expected to signal it via 571 set_error(). What happens next, depends on the case, but in all four 572 cases the error will be logged automatically by the harness. Also, the first 573 error passed from any plugin will be saved until the end of life-cycle 574 processing, then passed down to the code calling the harness. This will allow 575 the application code to deal with it accordingly (probably do some of its own 576 cleaning and shut down, but that's up to the application). In general, the first 577 error from init() or start() will cause the harness to initiate shut down 578 procedure, while the errors from stop() and deinit() will be ignored completely 579 (except of course for being logged and possibly saved for passing on at the 580 end). Actions taken for each plugin function are as follows: 581 582 583 584 #### init() fails: 585 586 - skip init() for remaining plugins 587 588 - don't run any start() and stop() (proceed directly to deinitialisation) 589 590 - run deinit() only for plugins initialiased so far (excluding the failing 591 one), in reverse order of initialisation, and exit 592 593 - when init() is not provided (is null), it counts as if it ran, if it would 594 have run before the failing plugin (according to topological order) 595 596 597 598 #### start() fails: 599 600 - proceed to stop all plugins, then deinit() all in reverse order of 601 initialisation and exit. Please note that ALL plugins will be flagged 602 to stop and have their stop() function called (not just the ones that 603 succeeded in starting - plugin's stop() must be able to deal with such 604 a situation) 605 606 607 608 #### stop() or deinit() fails: 609 610 - log error and ignore, proceed as if it didn't happen 611 612 */ 613 614 #ifndef MYSQL_HARNESS_LOADER_INCLUDED 615 #define MYSQL_HARNESS_LOADER_INCLUDED 616 617 #include "router_config.h" 618 619 #include "config_parser.h" 620 #include "filesystem.h" 621 #include "mysql/harness/dynamic_loader.h" 622 #include "mysql/harness/loader_config.h" 623 #include "mysql/harness/plugin.h" 624 625 #include "harness_export.h" 626 627 #include "mpsc_queue.h" 628 #include "my_compiler.h" 629 630 #include <csignal> 631 #include <cstdarg> // va_list 632 #include <exception> 633 #include <future> 634 #include <istream> 635 #include <list> 636 #include <map> 637 #include <queue> 638 #include <set> 639 #include <stdexcept> 640 #include <string> 641 #include <thread> 642 #include <tuple> 643 644 typedef void (*log_reopen_callback)(const std::string); 645 646 #ifdef FRIEND_TEST 647 // TODO replace by #include after merge: 648 // #include "../../../mysqlrouter/utils.h" // DECLARE_TEST 649 /** @brief Declare test (class) 650 * 651 * When using FRIEND_TEST() on classes that are not in the same namespace 652 * as the test, the test (class) needs to be forward-declared. This marco 653 * eases this. 654 * 655 * @note We need this for unit tests, BUT on the TESTED code side (not in unit 656 * test code) 657 */ 658 #define DECLARE_TEST(test_case_name, test_name) \ 659 class test_case_name##_##test_name##_Test 660 661 class TestLoader; 662 class LifecycleTest; 663 664 DECLARE_TEST(TestStart, StartLogger); 665 DECLARE_TEST(LifecycleTest, Simple_None); 666 DECLARE_TEST(LifecycleTest, Simple_AllFunctions); 667 DECLARE_TEST(LifecycleTest, Simple_Init); 668 DECLARE_TEST(LifecycleTest, Simple_StartStop); 669 DECLARE_TEST(LifecycleTest, Simple_StartStopBlocking); 670 DECLARE_TEST(LifecycleTest, Simple_Start); 671 DECLARE_TEST(LifecycleTest, Simple_Stop); 672 DECLARE_TEST(LifecycleTest, Simple_Deinit); 673 DECLARE_TEST(LifecycleTest, ThreeInstances_NoError); 674 DECLARE_TEST(LifecycleTest, BothLifecycles_NoError); 675 DECLARE_TEST(LifecycleTest, OneInstance_NothingPersists_NoError); 676 DECLARE_TEST(LifecycleTest, OneInstance_NothingPersists_StopFails); 677 DECLARE_TEST(LifecycleTest, ThreeInstances_InitFails); 678 DECLARE_TEST(LifecycleTest, BothLifecycles_InitFails); 679 DECLARE_TEST(LifecycleTest, ThreeInstances_Start1Fails); 680 DECLARE_TEST(LifecycleTest, ThreeInstances_Start2Fails); 681 DECLARE_TEST(LifecycleTest, ThreeInstances_Start3Fails); 682 DECLARE_TEST(LifecycleTest, ThreeInstances_2StartsFail); 683 DECLARE_TEST(LifecycleTest, ThreeInstances_StopFails); 684 DECLARE_TEST(LifecycleTest, ThreeInstances_DeinintFails); 685 DECLARE_TEST(LifecycleTest, ThreeInstances_StartStopDeinitFail); 686 DECLARE_TEST(LifecycleTest, NoInstances); 687 DECLARE_TEST(LifecycleTest, EmptyErrorMessage); 688 DECLARE_TEST(LifecycleTest, send_signals); 689 DECLARE_TEST(LifecycleTest, send_signals2); 690 DECLARE_TEST(LifecycleTest, wait_for_stop); 691 DECLARE_TEST(LifecycleTest, InitThrows); 692 DECLARE_TEST(LifecycleTest, StartThrows); 693 DECLARE_TEST(LifecycleTest, StopThrows); 694 DECLARE_TEST(LifecycleTest, DeinitThrows); 695 DECLARE_TEST(LifecycleTest, InitThrowsWeird); 696 DECLARE_TEST(LifecycleTest, StartThrowsWeird); 697 DECLARE_TEST(LifecycleTest, StopThrowsWeird); 698 DECLARE_TEST(LifecycleTest, DeinitThrowsWeird); 699 DECLARE_TEST(LoaderReadTest, Loading); 700 #endif 701 702 namespace mysql_harness { 703 704 struct Plugin; 705 class Path; 706 707 /** 708 * PluginFuncEnv object 709 * 710 * This object is the basis of all communication between Harness and plugin 711 * functions. It is passed to plugin functions (as an opaque pointer), and 712 * plugin functions return it back to Harness when calling Harness API 713 * functions. It has several functions: 714 * 715 * - maintains a "running" flag, which controls starting/stopping 716 * of plugins 717 * 718 * - conveys exit status back to Harness after each plugin function exits 719 * 720 * - conveys more information (AppInfo, ConfigSection, ...) to 721 * plugin functions. Note that not all fields are set for all functions - 722 * setting ConfigSection ptr when calling init() makes no sense, for example 723 * 724 * @note Below we only briefly document the methods. For more information, see 725 * Harness API documentation for their corresponding free-function proxies 726 * in plugin.h 727 */ 728 class HARNESS_EXPORT PluginFuncEnv { 729 public: 730 /** 731 * Constructor 732 * 733 * @param info AppInfo to pass to plugin function. Can be NULL. 734 * Pointer is owned by the caller and must outlive plugin function call. 735 * @param section ConfigSection to pass to plugin function. Can be NULL. 736 * Pointer is owned by the caller and must outlive plugin function call. 737 * @param running Set "running" flag. true = plugin should be running 738 */ 739 PluginFuncEnv(const AppInfo *info, const ConfigSection *section, 740 bool running = false); 741 742 // further info getters 743 // (see also corresponding Harness API functions in plugin.h for more info) 744 const ConfigSection *get_config_section() const noexcept; 745 const AppInfo *get_app_info() const noexcept; 746 747 // running flag 748 // (see also corresponding Harness API functions in plugin.h for more info) 749 void set_running() noexcept; 750 void clear_running() noexcept; 751 bool is_running() const noexcept; 752 bool wait_for_stop( 753 uint32_t milliseconds) const noexcept; // 0 = infinite wait 754 755 // error handling 756 // (see also corresponding Harness API functions in plugin.h for more info) 757 bool exit_ok() const noexcept; 758 MY_ATTRIBUTE((format(printf, 3, 0))) 759 void set_error(ErrorType error_type, const char *fmt, va_list ap) noexcept; 760 std::tuple<std::string, std::exception_ptr> pop_error() noexcept; 761 762 private: 763 const AppInfo *app_info_; // \. 764 const ConfigSection *config_section_; // > initialized in ctor 765 bool running_; // / 766 std::string error_message_; 767 ErrorType error_type_ = kNoError; 768 769 mutable std::condition_variable cond_; 770 mutable std::mutex mutex_; 771 }; 772 773 class HARNESS_EXPORT PluginThreads { 774 public: 775 void push_back(std::thread &&thr); 776 777 // wait for the first non-fatal exit from plugin or all plugins exited 778 // cleanly 779 void try_stopped(std::exception_ptr &first_exc); 780 push_exit_status(std::exception_ptr && eptr)781 void push_exit_status(std::exception_ptr &&eptr) { 782 plugin_stopped_events_.push(std::move(eptr)); 783 } 784 running()785 size_t running() const { return running_; } 786 787 void wait_all_stopped(std::exception_ptr &first_exc); 788 789 void join(); 790 791 private: 792 std::vector<std::thread> threads_; 793 size_t running_{0}; 794 795 /** 796 * queue of events after plugin's start() function exited. 797 * 798 * nullptr if "finished without error", pointer to an exception otherwise 799 */ 800 WaitingMPSCQueue<std::exception_ptr> plugin_stopped_events_; 801 }; 802 803 class HARNESS_EXPORT Loader { 804 public: 805 /** 806 * Constructor for Loader. 807 * 808 * @param program Name of our program 809 * @param config Router configuration 810 */ Loader(const std::string & program,LoaderConfig & config)811 Loader(const std::string &program, LoaderConfig &config) 812 : config_(config), program_(program) {} 813 814 Loader(const Loader &) = delete; 815 Loader &operator=(const Loader &) = delete; 816 817 /** 818 * Destructor. 819 * 820 * The destructor will call dlclose() on all unclosed shared 821 * libraries. 822 */ 823 824 ~Loader(); 825 826 /** 827 * Fetch available plugins. 828 * 829 * @return List of names of available plugins. 830 */ 831 832 std::list<Config::SectionKey> available() const; 833 834 /** 835 * Initialize and start all loaded plugins. 836 * 837 * All registered plugins will be initialized in proper order and 838 * started (if they have a `start` callback). 839 * 840 * @throws first exception that was triggered by an error returned from any 841 * plugin function. 842 */ 843 void start(); 844 845 /** 846 * Get reference to configuration object. 847 * 848 * @note In production code we initialize Loader with LoaderConfig 849 * reference maintained by DIM, so this method will return this object. 850 */ get_config()851 LoaderConfig &get_config() { return config_; } 852 853 private: 854 enum class Status { UNVISITED, ONGOING, VISITED }; 855 856 /** 857 * Flags progress of Loader. The progress always proceeds from top to bottom 858 * order in this list. 859 */ 860 enum class Stage { 861 // NOTE: do not alter order of these enums! 862 Unset, 863 Loading, 864 Initializing, 865 Starting, 866 Running, 867 Stopping, 868 Deinitializing, 869 Unloading, 870 }; 871 872 /** 873 * Load the named plugin from a specific library. 874 * 875 * @param plugin_name Name of the plugin to be loaded. 876 * 877 * @param library_name Name of the library the plugin should be 878 * loaded from. 879 * 880 * @throws bad_plugin (std::runtime_error) on load error 881 */ 882 const Plugin *load_from(const std::string &plugin_name, 883 const std::string &library_name); 884 885 const Plugin *load(const std::string &plugin_name); 886 887 /** 888 * Load the named plugin and all dependent plugins. 889 * 890 * @param plugin_name Name of the plugin to be loaded. 891 * @param key Key of the plugin to be loaded. 892 * 893 * @throws bad_plugin (std::runtime_error) on load error 894 * @throws bad_section (std::runtime_error) when section 'plugin_name' is not 895 * present in configuration 896 * 897 * @post After the execution of this procedure, the plugin and all 898 * plugins required by that plugin will be loaded. 899 */ 900 /** @overload */ 901 const Plugin *load(const std::string &plugin_name, const std::string &key); 902 903 // IMPORTANT design note: start_all() will block until PluginFuncEnv objects 904 // have been created for all plugins. This guarantees that the required 905 // PluginFuncEnv will always exist when plugin stop() function is called. 906 907 // start() calls these, indents reflect call hierarchy 908 void load_all(); // throws bad_plugin on load error 909 void setup_info(); 910 std::exception_ptr 911 run(); // returns first exception returned from below harness functions 912 std::exception_ptr init_all(); // returns first exception triggered by init() 913 914 void 915 start_all(); // forwards first exception triggered by start() to main_loop() 916 917 std::exception_ptr 918 main_loop(); // returns first exception triggered by start() or stop() 919 920 // class stop_all() and waits for plugins the terminate 921 std::exception_ptr stop_and_wait_all(); 922 923 std::exception_ptr stop_all(); // returns first exception triggered by stop() 924 925 std::exception_ptr 926 deinit_all(); // returns first exception triggered by deinit() 927 928 void unload_all(); 929 size_t external_plugins_to_load_count(); 930 931 /** 932 * Topological sort of all plugins and their dependencies. 933 * 934 * Will create a list of plugins in topological order from "top" 935 * to "bottom". 936 */ 937 bool topsort(); 938 bool visit(const std::string &name, std::map<std::string, Status> *seen, 939 std::list<std::string> *order); 940 941 /** 942 * Holds plugin's API call information 943 * 944 * @note There's 1 instance per plugin type (not plugin instance) 945 */ 946 class HARNESS_EXPORT PluginInfo { 947 public: 948 PluginInfo(const std::string &folder, const std::string &libname); PluginInfo(const Plugin * const plugin)949 PluginInfo(const Plugin *const plugin) : plugin_(plugin) {} 950 951 void load_plugin_descriptor(const std::string &name); // throws bad_plugin 952 plugin()953 const Plugin *plugin() const { return plugin_; } 954 library()955 const DynamicLibrary &library() const { return module_; } 956 957 private: 958 DynamicLibrary module_; 959 const Plugin *plugin_{}; 960 }; 961 962 using PluginMap = std::map<std::string, PluginInfo>; 963 964 // Init order is important, so keep config_ first. 965 966 /** 967 * Configuration sections for all plugins. 968 */ 969 LoaderConfig &config_; 970 971 /** 972 * Map of all successfully-loaded plugins (without key name). 973 */ 974 PluginMap plugins_; 975 976 /** 977 * Map of all {plugin instance -> plugin start() PluginFuncEnv} objects. 978 * Initially these objects are created in Loader::start_all() and then kept 979 * around until the end of Loader::stop_all(). At the time of writing, 980 * PluginFuncEnv objects for remaining plugin functions (init(), stop() and 981 * deinit()) are not required to live beyond their respective functions calls, 982 * and are therefore created on stack (automatic variables) as needed during 983 * those calls. 984 */ 985 std::map<const ConfigSection *, std::shared_ptr<PluginFuncEnv>> 986 plugin_start_env_; 987 988 /** 989 * active plugin threads. 990 */ 991 PluginThreads plugin_threads_; 992 993 /** 994 * Initialization order. 995 */ 996 std::list<std::string> order_; 997 998 std::string logging_folder_; 999 std::string plugin_folder_; 1000 std::string runtime_folder_; 1001 std::string config_folder_; 1002 std::string data_folder_; 1003 std::string program_; 1004 AppInfo appinfo_; 1005 1006 void spawn_signal_handler_thread(); 1007 1008 std::thread signal_thread_; 1009 1010 #ifdef FRIEND_TEST 1011 friend class ::TestLoader; 1012 friend class ::LifecycleTest; 1013 1014 FRIEND_TEST(::TestStart, StartLogger); 1015 FRIEND_TEST(::LifecycleTest, Simple_None); 1016 FRIEND_TEST(::LifecycleTest, Simple_AllFunctions); 1017 FRIEND_TEST(::LifecycleTest, Simple_Init); 1018 FRIEND_TEST(::LifecycleTest, Simple_StartStop); 1019 FRIEND_TEST(::LifecycleTest, Simple_StartStopBlocking); 1020 FRIEND_TEST(::LifecycleTest, Simple_Start); 1021 FRIEND_TEST(::LifecycleTest, Simple_Stop); 1022 FRIEND_TEST(::LifecycleTest, Simple_Deinit); 1023 FRIEND_TEST(::LifecycleTest, ThreeInstances_NoError); 1024 FRIEND_TEST(::LifecycleTest, BothLifecycles_NoError); 1025 FRIEND_TEST(::LifecycleTest, OneInstance_NothingPersists_NoError); 1026 FRIEND_TEST(::LifecycleTest, OneInstance_NothingPersists_StopFails); 1027 FRIEND_TEST(::LifecycleTest, ThreeInstances_InitFails); 1028 FRIEND_TEST(::LifecycleTest, BothLifecycles_InitFails); 1029 FRIEND_TEST(::LifecycleTest, ThreeInstances_Start1Fails); 1030 FRIEND_TEST(::LifecycleTest, ThreeInstances_Start2Fails); 1031 FRIEND_TEST(::LifecycleTest, ThreeInstances_Start3Fails); 1032 FRIEND_TEST(::LifecycleTest, ThreeInstances_2StartsFail); 1033 FRIEND_TEST(::LifecycleTest, ThreeInstances_StopFails); 1034 FRIEND_TEST(::LifecycleTest, ThreeInstances_DeinintFails); 1035 FRIEND_TEST(::LifecycleTest, ThreeInstances_StartStopDeinitFail); 1036 FRIEND_TEST(::LifecycleTest, NoInstances); 1037 FRIEND_TEST(::LifecycleTest, EmptyErrorMessage); 1038 FRIEND_TEST(::LifecycleTest, send_signals); 1039 FRIEND_TEST(::LifecycleTest, send_signals2); 1040 FRIEND_TEST(::LifecycleTest, wait_for_stop); 1041 FRIEND_TEST(::LifecycleTest, InitThrows); 1042 FRIEND_TEST(::LifecycleTest, StartThrows); 1043 FRIEND_TEST(::LifecycleTest, StopThrows); 1044 FRIEND_TEST(::LifecycleTest, DeinitThrows); 1045 FRIEND_TEST(::LifecycleTest, InitThrowsWeird); 1046 FRIEND_TEST(::LifecycleTest, StartThrowsWeird); 1047 FRIEND_TEST(::LifecycleTest, StopThrowsWeird); 1048 FRIEND_TEST(::LifecycleTest, DeinitThrowsWeird); 1049 FRIEND_TEST(::LoaderReadTest, Loading); 1050 #endif 1051 1052 }; // class Loader 1053 1054 class LogReopenThread { 1055 public: 1056 /** 1057 * @throws std::system_error if out of threads 1058 */ LogReopenThread()1059 LogReopenThread() : reopen_thr_{}, state_{REOPEN_NONE}, errmsg_{""} { 1060 // rely on move semantics 1061 reopen_thr_ = 1062 std::thread{&LogReopenThread::log_reopen_thread_function, this}; 1063 } 1064 1065 /** 1066 * stop the log_reopen_thread_function. 1067 * 1068 * @throws std::system_error from request_application_shutdown() 1069 */ 1070 void stop(); 1071 1072 /** 1073 * join the log_reopen thread. 1074 * 1075 * @throws std::system_error same as std::thread::join 1076 */ 1077 void join(); 1078 1079 /** 1080 * destruct the thread. 1081 * 1082 * Same as std::thread it may call std::terminate in case the thread isn't 1083 * joined yet, but joinable. 1084 * 1085 * In case join() fails as best-effort, a log-message is attempted to be 1086 * written. 1087 */ 1088 ~LogReopenThread(); 1089 1090 /** 1091 * thread function 1092 */ 1093 static void log_reopen_thread_function(LogReopenThread *t); 1094 1095 /* 1096 * request reopen 1097 * 1098 * @note Empty dst will cause reopen only, and the old content will not be 1099 * moved to dst. 1100 * @note This method uses mutex::try_lock() to avoid blocking the interrupt 1101 * handler if a signal is received during an already ongoing concurrent 1102 * reopen. The consequence is that reopen requests are ignored if rotation is 1103 * already in progress. 1104 * 1105 * @param dst filename to use for old log file during reopen 1106 * @throws std::system_error same as std::unique_lock::lock does 1107 */ 1108 void request_reopen(const std::string dst = ""); 1109 1110 /* Log reopen state triplet */ 1111 enum LogReopenState { REOPEN_NONE, REOPEN_REQUESTED, REOPEN_ACTIVE }; 1112 1113 /* Check log reopen completed */ is_completed()1114 bool is_completed() const { return (state_ == REOPEN_NONE); } 1115 1116 /* Check log reopen requested */ is_requested()1117 bool is_requested() const { return (state_ == REOPEN_REQUESTED); } 1118 1119 /* Check log reopen active */ is_active()1120 bool is_active() const { return (state_ == REOPEN_ACTIVE); } 1121 1122 /* Retrieve error from the last reopen */ get_last_error()1123 std::string get_last_error() const { return errmsg_; } 1124 1125 private: 1126 /* The thread handle */ 1127 std::thread reopen_thr_; 1128 1129 /* The log reopen thread state */ 1130 LogReopenState state_; 1131 1132 /* The last error message from the log reopen thread */ 1133 std::string errmsg_; 1134 1135 /* The destination filename to use for the old logfile during reopen */ 1136 std::string dst_; 1137 1138 }; // class LogReopenThread 1139 1140 } // namespace mysql_harness 1141 1142 /** 1143 * Setter for the log reopen thread completion callback function. 1144 * 1145 * @param cb Function to call at completion. 1146 */ 1147 HARNESS_EXPORT 1148 void set_log_reopen_complete_callback(log_reopen_callback cb); 1149 1150 /** 1151 * The default implementation for log reopen thread completion callback 1152 * function. 1153 * 1154 * @param errmsg Error message. Empty string assumes successful completion. 1155 */ 1156 HARNESS_EXPORT 1157 void default_log_reopen_complete_cb(const std::string errmsg); 1158 1159 /* 1160 * Reason for shutdown 1161 */ 1162 enum ShutdownReason { SHUTDOWN_NONE, SHUTDOWN_REQUESTED, SHUTDOWN_FATAL_ERROR }; 1163 1164 /** 1165 * request application shutdown. 1166 * 1167 * @param reason reason for the shutdown 1168 * @throws std::system_error same as std::unique_lock::lock does 1169 */ 1170 HARNESS_EXPORT 1171 void request_application_shutdown( 1172 const ShutdownReason reason = SHUTDOWN_REQUESTED); 1173 1174 /** 1175 * notify a "log_reopen" is requested with optional filename for old logfile. 1176 * 1177 * @param dst rename old logfile to filename before reopen 1178 * @throws std::system_error same as std::unique_lock::lock does 1179 */ 1180 HARNESS_EXPORT 1181 void request_log_reopen(const std::string dst = ""); 1182 1183 /** 1184 * check reopen completed 1185 */ 1186 HARNESS_EXPORT 1187 bool log_reopen_completed(); 1188 1189 /** 1190 * get last log reopen error 1191 */ 1192 HARNESS_EXPORT 1193 std::string log_reopen_get_error(); 1194 1195 #ifdef _WIN32 1196 HARNESS_EXPORT 1197 void register_ctrl_c_handler(); 1198 #endif 1199 1200 #ifdef FRIEND_TEST 1201 namespace unittest_backdoor { 1202 HARNESS_EXPORT 1203 void set_shutdown_pending(bool shutdown_pending); 1204 } // namespace unittest_backdoor 1205 #endif 1206 1207 #endif /* MYSQL_HARNESS_LOADER_INCLUDED */ 1208