1.. _versioning-system: 2.. currentmodule:: migrate.versioning 3.. highlight:: console 4 5*********************************** 6Database schema versioning workflow 7*********************************** 8 9SQLAlchemy migrate provides the :mod:`migrate.versioning` API that is 10also available as the :ref:`migrate <command-line-usage>` command. 11 12Purpose of this package is frontend for migrations. It provides commands to 13manage migrate :term:`repository` and database selection as well as script 14versioning. 15 16 17Project setup 18============= 19 20.. _create_change_repository: 21 22Create a change repository 23-------------------------- 24 25To begin, we'll need to create a :term:`repository` for our project. 26 27All work with repositories is done using the :ref:`migrate 28<command-line-usage>` command. Let's create our project's repository:: 29 30 $ migrate create my_repository "Example project" 31 32This creates an initially empty :term:`repository` relative to current 33directory at :file:`my_repository/` named `Example project`. 34 35The :term:`repository` directory contains a sub directory :file:`versions` that 36will store the :ref:`schema versions <changeset-system>`, a configuration file 37:file:`migrate.cfg` that contains :ref:`repository configuration 38<repository_configuration>` and a script :ref:`manage.py 39<project_management_script>` that has the same functionality as the 40:ref:`migrate <command-line-usage>` command but is preconfigured with 41repository specific parameters. 42 43.. note:: 44 45 Repositories are associated with a single database schema, and store 46 collections of change scripts to manage that schema. The scripts in a 47 :term:`repository` may be applied to any number of databases. Each 48 :term:`repository` has an unique name. This name is used to identify the 49 :term:`repository` we're working with. 50 51 52Version control a database 53-------------------------- 54 55Next we need to declare database to be under version control. Information on a 56database's version is stored in the database itself; declaring a database to be 57under version control creates a table named **migrate_version** and associates 58it with your :term:`repository`. 59 60The database is specified as a `SQLAlchemy database url`_. 61 62.. _`sqlalchemy database url`: 63 http://www.sqlalchemy.org/docs/core/engines.html#database-urls 64 65The :option:`version_control` command assigns a specified database with a 66:term:`repository`:: 67 68 $ python my_repository/manage.py version_control sqlite:///project.db my_repository 69 70We can have any number of databases under this :term:`repository's 71<repository>` version control. 72 73Each schema has a :term:`version` that SQLAlchemy Migrate manages. Each change 74script applied to the database increments this version number. You can retrieve 75a database's current :term:`version`:: 76 77 $ python my_repository/manage.py db_version sqlite:///project.db my_repository 78 0 79 80A freshly versioned database begins at version 0 by default. This assumes the 81database is empty or does only contain schema elements (tables, views, 82constraints, indices, ...) that will not be affected by the changes in the 83:term:`repository`. (If this is a bad assumption, you can specify the 84:term:`version` at the time the database is put under version control, with the 85:option:`version_control` command.) We'll see that creating and applying change 86scripts changes the database's :term:`version` number. 87 88Similarly, we can also see the latest :term:`version` available in a 89:term:`repository` with the command:: 90 91 $ python my_repository/manage.py version my_repository 92 0 93 94We've entered no changes so far, so our :term:`repository` cannot upgrade a 95database past version 0. 96 97Project management script 98------------------------- 99 100.. _project_management_script: 101 102Many commands need to know our project's database url and :term:`repository` 103path - typing them each time is tedious. We can create a script for our project 104that remembers the database and :term:`repository` we're using, and use it to 105perform commands:: 106 107 $ migrate manage manage.py --repository=my_repository --url=sqlite:///project.db 108 $ python manage.py db_version 109 0 110 111The script :file:`manage.py` was created. All commands we perform with it are 112the same as those performed with the :ref:`migrate <command-line-usage>` tool, 113using the :term:`repository` and database connection entered above. The 114difference between the script :file:`manage.py` in the current directory and 115the script inside the repository is, that the one in the current directory has 116the database URL preconfigured. 117 118.. note:: 119 120 Parameters specified in manage.py should be the same as in :ref:`versioning 121 api <versioning-api>`. Preconfigured parameter should just be omitted from 122 :ref:`migrate <command-line-usage>` command. 123 124 125Making schema changes 126===================== 127 128All changes to a database schema under version control should be done via 129change scripts - you should avoid schema modifications (creating tables, etc.) 130outside of change scripts. This allows you to determine what the schema looks 131like based on the version number alone, and helps ensure multiple databases 132you're working with are consistent. 133 134Create a change script 135---------------------- 136 137Our first change script will create a simple table 138 139.. code-block:: python 140 141 account = Table( 142 'account', meta, 143 Column('id', Integer, primary_key=True), 144 Column('login', String(40)), 145 Column('passwd', String(40)), 146 ) 147 148This table should be created in a change script. Let's create one:: 149 150 $ python manage.py script "Add account table" 151 152This creates an empty change script at 153:file:`my_repository/versions/001_Add_account_table.py`. Next, we'll 154edit this script to create our table. 155 156 157Edit the change script 158---------------------- 159 160Our change script predefines two functions, currently empty: 161:py:func:`upgrade` and :py:func:`downgrade`. We'll fill those in: 162 163.. code-block:: python 164 165 from sqlalchemy import Table, Column, Integer, String, MetaData 166 167 meta = MetaData() 168 169 account = Table( 170 'account', meta, 171 Column('id', Integer, primary_key=True), 172 Column('login', String(40)), 173 Column('passwd', String(40)), 174 ) 175 176 177 def upgrade(migrate_engine): 178 meta.bind = migrate_engine 179 account.create() 180 181 182 def downgrade(migrate_engine): 183 meta.bind = migrate_engine 184 account.drop() 185 186.. note:: 187 188 The generated script contains * imports from sqlalchemy and migrate. You 189 should tailor the imports to fit your actual demand. 190 191As you might have guessed, :py:func:`upgrade` upgrades the database to the next 192version. This function should contain the :ref:`schema changes 193<changeset-system>` we want to perform (in our example we're creating a 194table). 195 196:py:func:`downgrade` should reverse changes made by :py:func:`upgrade`. You'll 197need to write both functions for every change script. (Well, you don't *have* 198to write downgrade, but you won't be able to revert to an older version of the 199database or test your scripts without it.) If you really don't want to support 200downgrades it is a good idea to raise a :py:class:`NotImplementedError` or some 201equivalent custom exception. If you let :py:func:`downgrade` pass silently you 202might observe undesired behaviour for subsequent downgrade operations if 203downgrading multiple :term:`versions <version>`. 204 205 206.. note:: 207 208 As you can see, **migrate_engine** is passed to both functions. You should 209 use this in your change scripts, rather than creating your own engine. 210 211.. warning:: 212 213 You should be very careful about importing files from the rest of your 214 application, as your change scripts might break when your application 215 changes. Read more about `writing scripts with consistent behavior`_. 216 217 218Test the change script 219------------------------ 220 221Change scripts should be tested before they are committed. Testing a script 222will run its :func:`upgrade` and :func:`downgrade` functions on a specified 223database; you can ensure the script runs without error. You should be testing 224on a test database - if something goes wrong here, you'll need to correct it by 225hand. If the test is successful, the database should appear unchanged after 226:func:`upgrade` and :func:`downgrade` run. 227 228To test the script:: 229 230 $ python manage.py test 231 Upgrading... done 232 Downgrading... done 233 Success 234 235Our script runs on our database (:file:`sqlite:///project.db`, as specified in 236:file:`manage.py`) without any errors. 237 238Our :term:`repository's <repository>` :term:`version` is:: 239 240 $ python manage.py version 241 1 242 243.. note:: 244 245 Due to #41 the database must be exactly one :term:`version` behind the 246 :term:`repository` :term:`version`. 247 248.. _production testing warning: 249 250.. warning:: 251 252 The :option:`test` command executes actual scripts, be sure you are *NOT* 253 doing this on production database. 254 255 If you need to test production changes you should: 256 257 #. get a dump of your production database 258 #. import the dump into an empty database 259 #. run :option:`test` or :option:`upgrade` on that copy 260 261 262Upgrade the database 263-------------------- 264 265Now, we can apply this change script to our database:: 266 267 $ python manage.py upgrade 268 0 -> 1... 269 done 270 271This upgrades the database (:file:`sqlite:///project.db`, as specified when we 272created :file:`manage.py` above) to the latest available :term:`version`. (We 273could also specify a version number if we wished, using the :option:`--version` 274option.) We can see the database's :term:`version` number has changed, and our 275table has been created:: 276 277 $ python manage.py db_version 278 1 279 $ sqlite3 project.db 280 sqlite> .tables 281 account migrate_version 282 sqlite> .schema account 283 CREATE TABLE account ( 284 id INTEGER NOT NULL, 285 login VARCHAR(40), 286 passwd VARCHAR(40), 287 PRIMARY KEY (id) 288 ); 289 290Our account table was created - success! 291 292Modifying existing tables 293------------------------- 294 295After we have initialized the database schema we now want to add another Column 296to the `account` table that we already have in our schema. 297 298First start a new :term:`changeset` by the commands learned above:: 299 300 $ python manage.py script "Add email column" 301 302This creates a new :term:`changeset` template. Edit the resulting script 303:file:`my_repository/versions/002_Add_email_column.py`: 304 305.. code-block:: python 306 307 from sqlalchemy import Table, MetaData, String, Column 308 309 310 def upgrade(migrate_engine): 311 meta = MetaData(bind=migrate_engine) 312 account = Table('account', meta, autoload=True) 313 emailc = Column('email', String(128)) 314 emailc.create(account) 315 316 317 def downgrade(migrate_engine): 318 meta = MetaData(bind=migrate_engine) 319 account = Table('account', meta, autoload=True) 320 account.c.email.drop() 321 322As we can see in this example we can (and should) use SQLAlchemy's schema 323reflection (autoload) mechanism to reference existing schema objects. We could 324have defined the table objects as they are expected before upgrade or downgrade 325as well but this would have been more work and is not as convenient. 326 327We can now apply the changeset to :file:`sqlite:///project.db`:: 328 329 $ python manage.py upgrade 330 1 -> 2... 331 done 332 333and get the following expected result:: 334 335 $ sqlite3 project.db 336 sqlite> .schema account 337 CREATE TABLE account ( 338 id INTEGER NOT NULL, 339 login VARCHAR(40), 340 passwd VARCHAR(40), email VARCHAR(128), 341 PRIMARY KEY (id) 342 ); 343 344 345Writing change scripts 346====================== 347 348As our application evolves, we can create more change scripts using a similar 349process. 350 351By default, change scripts may do anything any other SQLAlchemy program can do. 352 353SQLAlchemy Migrate extends SQLAlchemy with several operations used to change 354existing schemas - ie. ``ALTER TABLE`` stuff. See :ref:`changeset 355<changeset-system>` documentation for details. 356 357 358Writing scripts with consistent behavior 359---------------------------------------- 360 361Normally, it's important to write change scripts in a way that's independent of 362your application - the same SQL should be generated every time, despite any 363changes to your app's source code. You don't want your change scripts' behavior 364changing when your source code does. 365 366.. warning:: 367 368 **Consider the following example of what NOT to do** 369 370 Let's say your application defines a table in the :file:`model.py` file: 371 372 .. code-block:: python 373 374 from sqlalchemy import * 375 376 meta = MetaData() 377 table = Table('mytable', meta, 378 Column('id', Integer, primary_key=True), 379 ) 380 381 ... and uses this file to create a table in a change script: 382 383 .. code-block:: python 384 385 from sqlalchemy import * 386 from migrate import * 387 import model 388 389 def upgrade(migrate_engine): 390 model.meta.bind = migrate_engine 391 392 def downgrade(migrate_engine): 393 model.meta.bind = migrate_engine 394 model.table.drop() 395 396 This runs successfully the first time. But what happens if we change the 397 table definition in :file:`model.py`? 398 399 .. code-block:: python 400 401 from sqlalchemy import * 402 403 meta = MetaData() 404 table = Table('mytable', meta, 405 Column('id', Integer, primary_key=True), 406 Column('data', String(42)), 407 ) 408 409 We'll create a new column with a matching change script 410 411 .. code-block:: python 412 413 from sqlalchemy import * 414 from migrate import * 415 import model 416 417 def upgrade(migrate_engine): 418 model.meta.bind = migrate_engine 419 model.table.create() 420 421 def downgrade(migrate_engine): 422 model.meta.bind = migrate_engine 423 model.table.drop() 424 425 This appears to run fine when upgrading an existing database - but the 426 first script's behavior changed! Running all our change scripts on a new 427 database will result in an error - the first script creates the table based 428 on the new definition, with both columns; the second cannot add the column 429 because it already exists. 430 431 To avoid the above problem, you should use SQLAlchemy schema reflection as 432 shown above or copy-paste your table definition into each change script 433 rather than importing parts of your application. 434 435 .. note:: 436 Sometimes it is enough to just reflect tables with SQLAlchemy instead 437 of copy-pasting - but remember, explicit is better than implicit! 438 439 440Writing for a specific database 441------------------------------- 442 443Sometimes you need to write code for a specific database. Migrate scripts can 444run under any database, however - the engine you're given might belong to any 445database. Use engine.name to get the name of the database you're working with 446 447.. code-block:: python 448 449 >>> from sqlalchemy import * 450 >>> from migrate import * 451 >>> 452 >>> engine = create_engine('sqlite:///:memory:') 453 >>> engine.name 454 'sqlite' 455 456 457Writings .sql scripts 458--------------------- 459 460You might prefer to write your change scripts in SQL, as .sql files, rather 461than as Python scripts. SQLAlchemy-migrate can work with that:: 462 463 $ python manage.py version 464 1 465 $ python manage.py script_sql postgresql 466 467This creates two scripts 468:file:`my_repository/versions/002_postgresql_upgrade.sql` and 469:file:`my_repository/versions/002_postgresql_downgrade.sql`, one for each 470*operation*, or function defined in a Python change script - upgrade and 471downgrade. Both are specified to run with PostgreSQL databases - we can add 472more for different databases if we like. Any database defined by SQLAlchemy may 473be used here - ex. sqlite, postgresql, oracle, mysql... 474 475 476.. _command-line-usage: 477 478Command line usage 479================== 480 481.. currentmodule:: migrate.versioning.shell 482 483:command:`migrate` command is used for API interface. For list of commands and 484help use:: 485 486 $ migrate --help 487 488:command:`migrate` command executes :func:`main` function. 489For ease of usage, generate your own :ref:`project management script 490<project_management_script>`, which calls :func:`main 491<migrate.versioning.shell.main>` function with keywords arguments. You may want 492to specify `url` and `repository` arguments which almost all API functions 493require. 494 495If api command looks like:: 496 497 $ migrate downgrade URL REPOSITORY VERSION [--preview_sql|--preview_py] 498 499and you have a project management script that looks like 500 501.. code-block:: python 502 503 from migrate.versioning.shell import main 504 505 main(url='sqlite://', repository='./project/migrations/') 506 507you have first two slots filed, and command line usage would look like:: 508 509 # preview Python script 510 $ migrate downgrade 2 --preview_py 511 512 # downgrade to version 2 513 $ migrate downgrade 2 514 515.. versionchanged:: 0.5.4 516 Command line parsing refactored: positional parameters usage 517 518Whole command line parsing was rewriten from scratch with use of OptionParser. 519Options passed as kwargs to :func:`~migrate.versioning.shell.main` are now 520parsed correctly. Options are passed to commands in the following priority 521(starting from highest): 522 523- optional (given by :option:`--some_option` in commandline) 524- positional arguments 525- kwargs passed to :func:`migrate.versioning.shell.main` 526 527 528Python API 529========== 530 531.. currentmodule:: migrate.versioning.api 532 533All commands available from the command line are also available for 534your Python scripts by importing :mod:`migrate.versioning.api`. See the 535:mod:`migrate.versioning.api` documentation for a list of functions; 536function names match equivalent shell commands. You can use this to 537help integrate SQLAlchemy Migrate with your existing update process. 538 539For example, the following commands are similar: 540 541*From the command line*:: 542 543 $ migrate help help 544 /usr/bin/migrate help COMMAND 545 546 Displays help on a given command. 547 548*From Python* 549 550.. code-block:: python 551 552 import migrate.versioning.api 553 migrate.versioning.api.help('help') 554 # Output: 555 # %prog help COMMAND 556 # 557 # Displays help on a given command. 558 559 560.. _migrate.versioning.api: module-migrate.versioning.api.html 561 562.. _repository_configuration: 563 564 565Experimental commands 566===================== 567 568Some interesting new features to create SQLAlchemy db models from existing 569databases and vice versa were developed by Christian Simms during the 570development of SQLAlchemy-migrate 0.4.5. These features are roughly documented 571in a `thread in migrate-users`_. 572 573.. _`thread in migrate-users`: 574 http://groups.google.com/group/migrate-users/browse_thread/thread/a5605184e08abf33#msg_85c803b71b29993f 575 576Here are the commands' descriptions as given by ``migrate help <command>``: 577 578- ``compare_model_to_db``: Compare the current model (assumed to be a 579 module level variable of type sqlalchemy.MetaData) against the 580 current database. 581- ``create_model``: Dump the current database as a Python model to 582 stdout. 583- ``make_update_script_for_model``: Create a script changing the old 584 Python model to the new (current) Python model, sending to stdout. 585 586As this sections headline says: These features are *EXPERIMENTAL*. Take the 587necessary arguments to the commands from the output of ``migrate 588help <command>``. 589 590 591Repository configuration 592======================== 593 594SQLAlchemy-migrate :term:`repositories <repository>` can be configured in their 595:file:`migrate.cfg` files. The initial configuration is performed by the 596`migrate create` call explained in :ref:`Create a change repository 597<create_change_repository>`. The following options are available currently: 598 599- :option:`repository_id` Used to identify which repository this database is 600 versioned under. You can use the name of your project. 601- :option:`version_table` The name of the database table used to track the 602 schema version. This name shouldn't already be used by your project. If this 603 is changed once a database is under version control, you'll need to change 604 the table name in each database too. 605- :option:`required_dbs` When committing a change script, SQLAlchemy-migrate 606 will attempt to generate the sql for all supported databases; normally, if 607 one of them fails - probably because you don't have that database installed - 608 it is ignored and the commit continues, perhaps ending successfully. 609 Databases in this list MUST compile successfully during a commit, or the 610 entire commit will fail. List the databases your application will actually be 611 using to ensure your updates to that database work properly. This must be a 612 list; example: `['postgres', 'sqlite']` 613- :option:`use_timestamp_numbering` When creating new change scripts, Migrate 614 will stamp the new script with a version number. By default this is 615 latest_version + 1. You can set this to 'true' to tell Migrate to use the UTC 616 timestamp instead. 617 618 .. versionadded:: 0.7.2 619 620.. _custom-templates: 621 622 623Customize templates 624=================== 625 626Users can pass ``templates_path`` to API functions to provide customized 627templates path. Path should be a collection of templates, like 628``migrate.versioning.templates`` package directory. 629 630One may also want to specify custom themes. API functions accept 631``templates_theme`` for this purpose (which defaults to `default`) 632 633Example:: 634 635 /home/user/templates/manage $ ls 636 default.py_tmpl 637 pylons.py_tmpl 638 639 /home/user/templates/manage $ migrate manage manage.py --templates_path=/home/user/templates --templates_theme=pylons 640 641.. versionadded:: 0.6.0 642