1 /* Copyright (c) 2014, 2019, Oracle and/or its affiliates. All rights reserved. 2 3 This program is free software; you can redistribute it and/or modify 4 it under the terms of the GNU General Public License, version 2.0, 5 as published by the Free Software Foundation. 6 7 This program is also distributed with certain software (including 8 but not limited to OpenSSL) that is licensed under separate terms, 9 as designated in a particular file or component or in included license 10 documentation. The authors of MySQL hereby grant you an additional 11 permission to link the program and your derivative works with the 12 separately licensed software that they have included with MySQL. 13 14 This program is distributed in the hope that it will be useful, 15 but WITHOUT ANY WARRANTY; without even the implied warranty of 16 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 GNU General Public License, version 2.0, for more details. 18 19 You should have received a copy of the GNU General Public License 20 along with this program; if not, write to the Free Software 21 Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ 22 23 #ifndef DD__BOOTSTRAPPER_INCLUDED 24 #define DD__BOOTSTRAPPER_INCLUDED 25 26 #include <sys/types.h> 27 28 #include "sql/dd/impl/system_registry.h" // dd::System_tables 29 #include "sql/dd/string_type.h" // dd::String_type 30 #include "sql/handler.h" // dict_init_mode_t 31 32 class THD; 33 34 /** 35 Data dictionary initialization. 36 37 The data dictionary is initialized whenever the mysqld process starts. 38 We distinguish between the first time start and the subsequent normal 39 restarts/upgrades, as explained below. However, there are three main 40 design principles that should be elaborated first. 41 42 1. Two-step process: The dictionary initialization is implemented as 43 a two step process. First, scaffolding is built to prepare the 44 synchronization with persistent storage, then, the actual synchronization 45 is done. The way this is done depends on the context, and is different 46 for first time start and the subsequent restarts. 47 48 2. Use SQL: The initialization uses SQL to build the scaffolding. This 49 means that we execute SQL statements to create the dictionary tables. 50 Since this is done at a stage where the physical tables either do not 51 exist yet, or are not known, we must instrument the DDL execution to 52 create the physical counterpart of the tables only on first time start. 53 The goal is to keep the instrumentation at a minimum. 54 55 3. Fake caching: As a consequence of keeping instrumentation at a minimum, 56 we provide uniform behavior of the caching layer in the data dictionary 57 also in the scaffolding phase. This means that as seen from the outside, 58 dictionary objects can be retrieved from the cache. Internally, below the 59 caching layer, the objects are only kept in a separate buffer until all 60 the required scaffolding is built. At that point, we can start using the 61 underlying physical tables, depending on the circumstances: 62 63 - For first time start (initialization), we can flush the meta data 64 generated in the scaffolding phase, to the DD tables. 65 - For ordinary restart, we can use the scaffolding to open the physical 66 tables, and then sync up the real meta data that is stored persistently. 67 - For upgrade, we first build scaffolding based on the actual DD tables, 68 then we create the target DD tables, migrate the meta data from the old 69 to the new tables, and finally switch from old to new tables 70 atomically by means of DML on the DD tables. This means that we update 71 the schema ids in the DD tables directly instead of executing 72 'RENAME TABLE', which would do auto commit and thus break atomicity. 73 74 After the scaffolding has been flushed or synced, what should be left is 75 a collection of the core DD meta data objects. This collection is located 76 in the storage adapter, and allows the DD cache to evict core DD objects 77 in the same way as other DD objects. 78 79 Please note that dictionary initialization is only a small part of server 80 initialization. There is a lot going on before and after dictionary 81 initialization while starting the server. 82 83 Please see more elaborated descriptions for the initialize() and restart() 84 methods below. 85 */ 86 87 namespace dd { 88 class Dictionary_impl; 89 90 namespace bootstrap { 91 92 /** 93 Initialize the dictionary while starting the server for the first time. 94 95 At this point, the DDSE has been initialized as a normal plugin. The 96 dictionary initialization proceeds as follows: 97 98 1. Preparation phase 99 100 1.1 Call dict_init() to initialize the DDSE. This will make the predefined 101 tablespaces be created physically, and their meta data be returned to 102 the SQL layer along with the meta data for the DD tables required by 103 the DDSE. The tables are not yet created physically. 104 1.2 Prepare the dd::Tablespace objects reflecting the predefined tablespace 105 objects and add them to the core registry in the storage adapter. 106 107 2. Scaffolding phase 108 109 2.1 Create and use the dictionary schema by executing SQL statements. 110 The schema is created physically since this is the first time start, 111 and the meta data is generated and stored in the core registry of 112 the storage adapter without being written to disk. 113 2.2 Create tables by executing SQL statements. Like for the schema, the 114 tables are created physically, and the meta data is generated 115 and stored in the core registry without being written to disk. 116 This is done to prepare enough meta data to actually be able to 117 open the DD tables. 118 119 3. Synchronization phase 120 121 3.1 Store meta data for the DD schema, tablespace and tables, i.e., the DD 122 objects that were generated in the scaffolding phase, and make sure the 123 IDs are maintained when the objects are stored. 124 3.2 Populate the DD tables which have some predefined static contents to 125 be inserted. This is, e.g., relevant for the 'catalogs' table, which 126 only has a single default entry in it. Dynamic contents is added in 127 other ways, e.g. by storing generated DD objects (see above) or by 128 inserting data from other sources (see re-population of character sets 129 in the context of server restart below). 130 3.3 Store various properties of the DD tables, including the SE private data, 131 a representation of the DDL statement used to create the table etc. 132 3.4 Verify that the dictionary objects representing the core DD table meta 133 data are present in the core registry of the storage adapter. If an 134 object representing the meta data of a core DD table is not available, 135 then we loose access to the DD tables, and we will not be able to handle 136 cache misses or updates to the meta data. 137 3.5 Update the version numbers that are stored, e.g. the DD version and the 138 current mysqld server version. 139 140 @param thd Thread context. 141 142 @return Upon failure, return true, otherwise false. 143 */ 144 145 bool initialize(THD *thd); 146 147 /** 148 Initialize the dictionary while restarting the server. 149 150 At this point, the DDSE has been initialized as a normal plugin. The 151 dictionary initialization proceeds as follows: 152 153 1. Preparation phase 154 155 1.1 Call dict_init() to initialize the DDSE. This will retrieve the meta data 156 of the predefined tablespaces and the DD tables required by the DDSE. 157 Both the tables and the tablespaces are already created physically, the 158 point here is just to get hold of enough meta data to start using the DD. 159 1.2 Prepare the dd::Tablespace objects reflecting the predefined tablespace 160 objects and add them to the core registry in the storage adapter. 161 162 2. Scaffolding phase 163 164 2.1 Create and use the dictionary schema by executing SQL statements. 165 The schema is not created physically, but the meta data is generated 166 and stored in the core registry without being written to disk. 167 2.2 Create tables by executing SQL statements. Like for the schema, the 168 tables are not created physically, but the meta data is generated 169 and stored in the core registry without being written to disk. 170 This is done to prepare enough meta data to actually be able to 171 open the DD tables. The SQL DDL statements are either retrieved from 172 the table definitions that are part of the server binary (for restart), 173 or from one of the DD tables (for upgrade). 174 175 3. Synchronization phase 176 177 3.1 Read meta data for the DD tables from the DD tables. Here, we use the 178 meta data from the scaffolding phase for the schema, tablespace and the 179 DD tables to open the physical DD tables. We read the stored objects, 180 and update the in-memory copies in the core registry with the real meta 181 data from the objects that are retrieved form persistent storage. Finally, 182 we flush the tables to empty the table definition cache to make sure the 183 table share structures for the DD tables are re-created based on the 184 actual meta data that was read from disk rather than the temporary meta 185 data from the scaffolding phase. 186 3.2 If this is a restart with a new DD version, we must upgrade the DD 187 tables. In that case, we create the new target DD tables in a temporary 188 schema, migrate the meta data to the new tables, and then do DML on the 189 DD tables to make sure the new DD tables will be used instead of the old 190 ones. This DML involves changing the schema ids directly in the DD tables, 191 and updating the meta data stored in the 'dd_properties' DD table. 192 This will make sure the switch from the old to the new tables is 193 atomic. After this is done, we will reset the DD cache and start over 194 the initialization from step 1.2. Then, the new DD tables will be used, 195 and a normal restart will be done. 196 3.3 Re-populate character sets and collations: The character set and 197 collation information is read from files and added to a server 198 internal data structure when the server starts. This data structure is, 199 in turn, used to populate the corresponding DD tables. The tables must 200 be re-populated on each server start if new character sets or collations 201 have been added. However, we can not do this if in read only mode. 202 3.4 Verify that the dictionary objects representing the core DD table meta 203 data are present in the core registry of the storage adapter. If an 204 object representing the meta data of a core DD table is not available, 205 then we loose access to the DD tables, and we will not be able to handle 206 cache misses or updates to the meta data. 207 3.5 If an upgrade was done, the persistent version numbers are updated, 208 e.g. the DD version and the current mysqld server version. 209 210 @param thd Thread context. 211 212 @return Upon failure, return true, otherwise false. 213 */ 214 215 bool restart(THD *thd); 216 217 /** 218 Iterate through all the plugins, and store IS table meta data 219 into dictionary, once during MySQL server bootstrap. 220 221 @param thd Thread context. 222 223 @return Upon failure, return true, otherwise false. 224 */ 225 bool store_plugin_IS_table_metadata(THD *thd); 226 227 /** 228 Initialization and verification of dictionary objects 229 after upgrade, similar to what is done after normal server 230 restart. 231 232 @param thd Thread context 233 */ 234 bool setup_dd_objects_and_collations(THD *thd); 235 236 /** 237 This function is used in case of crash during upgrade. 238 It tries to initialize dictionary and calls DDSE_dict_recover. 239 InnoDB should do the recovery and empty undo log. Upgrade 240 process will do the cleanup and exit. 241 242 @param thd Thread context. 243 */ 244 void recover_innodb_upon_upgrade(THD *thd); 245 246 /** 247 Initialize InnoDB for 248 - creating new data directory : InnoDB creates system tablespace and 249 dictionary tablespace. 250 - normal server restart. : Verifies existence of system and dictionary 251 tablespaces. 252 - in place upgrade : Verifies existence of system tablespace and 253 create dictionary tablespace. 254 255 @param thd Thread context. 256 @param dict_init_mode mode to initialize InnoDB 257 @param version Dictionary version. 258 259 @return Upon failure, return true, otherwise false. 260 */ 261 bool DDSE_dict_init(THD *thd, dict_init_mode_t dict_init_mode, uint version); 262 263 /** 264 Create mysql schema. Create dictionary tables inside InnoDB. 265 Create entry for dictionary tables inside dictionary tables. 266 Add hard coded data to dictionary tables. 267 Create Foreign key constraint on dictionary tables. 268 269 This function is used in both cases, new data directory initialization 270 and in place upgrade. 271 272 @param thd Thread context. 273 @param is_dd_upgrade Flag to indicate if it is in place upgrade. 274 @param d Dictionary instance 275 276 @return Upon failure, return true, otherwise false. 277 278 */ 279 bool initialize_dictionary(THD *thd, bool is_dd_upgrade, Dictionary_impl *d); 280 281 } // namespace bootstrap 282 283 /** 284 This function creates the meta data of the predefined tablespaces. 285 286 @param thd Thread context. 287 */ 288 void store_predefined_tablespace_metadata(THD *thd); 289 290 /** 291 Executes SQL queries to create and use the dictionary schema. 292 293 @param thd Thread context. 294 */ 295 bool create_dd_schema(THD *thd); 296 297 /** 298 During --initialize, we create the dd_properties table. During restart, 299 create its meta data, and use it to open and read its contents. 300 301 @param thd Thread context. 302 */ 303 bool initialize_dd_properties(THD *thd); 304 305 /** 306 Predicate to check if a table type is a non-inert DD ot DDSE table. 307 308 @param table_type Type as defined in the System_tables registry. 309 @returns true if the table is a non-inert DD or DDSE table, 310 false otherwise 311 */ 312 bool is_non_inert_dd_or_ddse_table(System_tables::Types table_type); 313 314 /** 315 Execute SQL statements to create the DD tables. 316 317 The tables created here will be a subset of the target DD tables for this 318 DD version. This function is called in the following four cases: 319 320 1. When a server is started the first time, with --initialize. Then, we 321 will iterate over all target tables and create them. This will also 322 make them be created physically in the DDSE. 323 2. When a server is restarted, and the data directory contains a dictionary 324 with the same DD version as the target DD version of the starting server. 325 In this case, we will iterate over all target tables and create them, 326 using the target table SQL DDL definitions. This is done only to create 327 the meta data, though; the tables will not be created physically in the 328 DDSE since they already exist. But we need to create the meta data to be 329 able top open them. 330 3. When a server is restarted, and the data directory was last used by a 331 more recent MRU within the same GA with a higher target DD version. 332 This is considered a 'minor downgrade'. In this case, the restarting 333 server will continue to run using the more recent DD version. This is 334 possible since only a subset of DD changes are allowed in a DD upgrade 335 that can also be downgraded. However, it means that we must create the 336 meta data reflecting the *actual* tables, not the target tables. So in 337 this case, we iterate over the target tables, but execute the DDL 338 statements of the actual tables. We get these statements from the 339 'dd_properties' table, where the more recent MRU has stored them. 340 4. When a server is restarted, and the data directory was last used by a 341 server with a DD version from which the starting server can upgrade. In 342 this case, this function is called three times: 343 344 - The first time, we need to create the meta data reflecting the actual 345 tables in the persistent DD. This is needed to be able to open the DD 346 tables and read the data. This is similar to use case 3. above. 347 - The second time, we create the tables that are modified in the new DD 348 version. Here, the tables are also created physically in the DDSE. 349 In this case, the 'create_set' specifies which subset of the target 350 tables should be created. After this stage, we replace the meta data 351 in 'dd_properties' by new meta data reflecting the modified tables. We 352 also replace the version numbers to make sure a new restart will use 353 the upgraded DD. 354 - The third time, we do the same as in case 2 above. This is basically 355 the same as a shutdown and restart of the server after upgrade was 356 completed. 357 358 @param thd Thread context. 359 @param create_set Subset of the target tables which should be created 360 during upgrade. 361 362 @returns false if success, otherwise true. 363 */ 364 bool create_tables(THD *thd, const std::set<String_type> *create_set); 365 366 /** 367 Acquire the DD schema, tablespace and table objects. Read the persisted 368 objects from the DD tables, and replace the contents of the core 369 registry in the storage adapter 370 371 @param thd Thread context. 372 */ 373 bool sync_meta_data(THD *thd); 374 375 /** 376 Update properties in the DD_properties table. Note that upon failure, we 377 will rollback, whereas upon success, commit will be delayed. 378 379 @param thd Thread context. 380 @param create_set A set of table names created/modified in 381 this version of DD. 382 @param remove_set A set of table names removed in this 383 version of DD. 384 @param target_table_schema_name Schema name in which the final changes are 385 required. 386 387 @return Upon failure, return true, otherwise false. 388 */ 389 bool update_properties(THD *thd, const std::set<String_type> *create_set, 390 const std::set<String_type> *remove_set, 391 const String_type &target_table_schema_name); 392 393 /** 394 Updates the DD Version in the DD_properties table to the current version. 395 This function is used during initialize and during server upgrade. 396 397 @param thd Thread context. 398 @param is_dd_upgrade_57 Flag to indicate if it is an upgrade from 5.7. 399 400 @return Upon failure, return true, otherwise false. 401 */ 402 bool update_versions(THD *thd, bool is_dd_upgrade_57); 403 404 } // namespace dd 405 #endif // DD__BOOTSTRAPPER_INCLUDED 406