[ Z ]
wildebeest
migration framework

about wildebeest

Wildebeest is a migration framework designed to solve the problem of reliably migrating stateful system resources such as databases or file structures in continuous deployment projects

works with

  • Microsoft SQL Server
  • MySQL
Download Wildebeest
Version 3.0.120.91
Released 2014-01-12 under the terms of
the GNU General Public License v2.
See release history for change log.

Introduction

Wildebeest models stateful resources as directed graphs where each node represents a particular state for the resource, and migrations provide the pathways from state to state. Resource descriptors are expressed in XML files with the extension *.wbr. For example you might create a resource called database.wbr. which defines the states and migrations for managing your application's database.

Having defined a resource, you may want to create many instances of it. For example you may want to have an instance of the database for your application for each developer in your team. You may also have multiple staging, testing and production instances to maintain.

A Wildbeest resource descriptor is abstract in that it does not include details such as database server hostname and port. These details are separated into an instance descriptor which is also expressed in XML and uses the file extension *.wbi. For example you might have database-staging.wbi and database-production.wbi descriptors that can be used to manage different instances of your database.wbr resource.

Wildbeest is driven by command-line. For example to migrate a resource to a state called "Initial Schema Loaded" we would issue a command like this:

wb migrate --resource:database.wbr --instance:staging.wbi --targetState:"Core Schema Loaded"

Wildebeest reports progress as it migrates between steps, and asserts at each step:

wb migrate --resource:database.wbr --instance:staging.wbi --targetState:"Core Schema Loaded"

2013-06-09T17:57:24 - Wildebeest is migrating
2013-06-09T17:57:24 - Migrating from non-existent to "Database Created"
2013-06-09T17:57:24 - Migration complete
2013-06-09T17:57:24 - Assertion "Database exists" passed: Database ProductCatalogueStaging exists
2013-06-09T17:57:24 - Migrating from state "Database Created" to "Core Schema Loaded"
2013-06-09T17:57:24 - Migration complete
2013-06-09T17:57:24 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T17:57:24 - Assertion "Product table exists" passed: Table Product exists

Wildebeest will soon support other interface options including an embedded APIs for Java and .NET, and continuous integration server plugins.

In the next section you'll learn more about how resources and instances are defined, and see the full set of commands that Wildebeest provides. Later you'll see the specifics on the plugins for managing MySQL databases as Wildebeest resources.

Installing Wildebeest

Overview

The installation procedure for Wildebeest is similar to many other Java tools:

  • Obtain and unpack the release
  • Set the WB_HOME environment variable
  • Update the system PATH

Obtain and Unpack

Download the latest release of Wildebeest from the link at the top of this page. Once downloaded, unzip the release bundle and place it in your preferred location. For example on Windows you might place it in:

C:\Tools\ZD.Wildebeest-3.0.120.91
Or under Linux you might place it in:
/opt/ZD.Wildebeest-3.0.120.91

If you are installing Wildebeest to Linux, OSX or another UNIX-type operating system, you will need to make the wb command executable as follows:

chmod u+x <install location>/bin/wb

Set WB_HOME and Update PATH

The WB_HOME environment variable must be set to the location where you placed the unpacked Wildebeest release. On Windows this is done as follows:

  1. Click Start, right-click Computer and click Properties
  2. Click Advanced system settings from the left panel.
  3. On the System Properties dialog box, click Environment Variables
  4. Click the New button in the System Variables section
  5. Enter WB_HOME for the variable name, and the installation location for the variable value, and click OK

Now to update the PATH variable:

  1. Locate the PATH variable in the System Variables section
  2. Click Edit
  3. Append ";%WB_HOME%\bin" to the value of the variable, and click OK

Modeling Resources

In Wildebeest you model a resource by defining all the possible states in which it may exist, and the actions to migrate the resource from one state to another. Different types of migrations may be needed to move between states. You can visualize the definition of a resource as nodes that represents the states in which the resource may exist, with migrations providing the pathways between states.

Note that Wildebeest considers the resource not existing at all to be a state as well which is referred to as "non-existent". To create a resource we migrate it from non-existent to some state. To destroy a resource we can also migrate it from some defined state to non-existent.

Typically the series of states will be a linear progression like in this example, but there some interesting use cases that can be handled elegantly by creating multiple possible paths through the graph.

Resources are defined in XML with either *.wbresource.xml or *.wbr as the file extension. The descriptor for the resource shown above is as follows:

<?xml version="1.0"?>
<resource type="MySqlDatabase" id="0d39b8fb-5b5c-48cd-845c-9c4d55f94303" name="Product Catalogue Database">
    <states>
        <state id="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7" label="Database Created" />
        <state id="363568f1-aaed-4a50-bea0-9ddee713cc11" label="Core Schema Loaded" />
    </states>
    <migrations>
        <migration
            type="MySqlCreateDatabase"
            id="6b21e1e3-ff3a-44b3-84ec-e21fb01c0110"
            toStateId="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7">
        </migration>
        <migration
            type="SqlScript"
            id="8b57f16d-c690-4f10-b68f-6f1ee75fe32b"
            fromStateId="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7"
            toStateId="363568f1-aaed-4a50-bea0-9ddee713cc11">
            <sql><![CDATA[
/* ProductType */
CREATE TABLE  `ProductType` (
  `ProductTypeCode` char(2) NOT NULL,
  `Name` varchar(10) NOT NULL,
  PRIMARY KEY (`ProductTypeCode`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO
    ProductType(ProductTypeCode, Name)
VALUES
    ('HW', 'Hardware'),
    ('SW', 'Software');
/* Product */
CREATE TABLE  `Product` (
  `ProductId` char(36) NOT NULL,
  `ProductTypeCode` char(2) NOT NULL,
  `Name` varchar(50) NOT NULL,
  `Description` varchar(4000) NOT NULL,
  PRIMARY KEY (`ProductId`),
  KEY `FK_Product_ProductTypeCode` (`ProductTypeCode`),
  CONSTRAINT `FK_Product_ProductTypeCode` FOREIGN KEY (`ProductTypeCode`) REFERENCES `ProductType` (`ProductTypeCode`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
            ]]></sql>
        </migration>
        <migration
            type="SqlScript"
            id="8b57f16d-c690-4f10-b68f-6f1ee75fe32b"
            fromStateId="363568f1-aaed-4a50-bea0-9ddee713cc11"
            toStateId="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7">
            <sql><![CDATA[
DROP TABLE Product;
DROP TABLE ProductType;
            ]]></sql>
        </migration>
        
    </migrations>
</resource>

At any given state we can define a set of assertions that must hold true for the state to be considered valid. in Wildebeest the act of applying the assertions for a state is referred to as "verifying" the state. Wildebeest will verify the current state before attempting any migration, and will abort the migration if the current state is found to be invalid. Wildebeest also verifies the new state after performing a migration, and will report to you if the new state is found to be invalid.

Wildebeest offers various types of assertions such as MySqlDatabaseExists and DatabaseRowExists. These are all described in the reference section below. After adding the assertions shown in the diagram above, the states in our resource descriptor would now look like this:

<?xml version="1.0"?>
<resource type="MySqlDatabase" id="0d39b8fb-5b5c-48cd-845c-9c4d55f94303" name="Product Catalogue Database">
    <states>
        <state id="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7" label="Database Created">
            <assertions>
                <assertion type="MySqlDatabaseExists" id="19b64fb6-f95f-4221-bc64-303f62a3eda8" name="Database exists" />
            </assertions>
        </state>
        <state id="363568f1-aaed-4a50-bea0-9ddee713cc11" label="Core Schema Loaded">
            <assertions>
                <assertion type="MySqlTableExists" id="3808ba63-f055-4bf7-88fe-023546e6ed16" name="ProductType table exists">
                    <tableName>ProductType</tableName>
                </assertion>
                <assertion type="MySqlTableExists" id="54c78eca-9eda-4983-b9a5-0dcb1be686ff" name="Product table exists">
                    <tableName>Product</tableName>
                </assertion>
            </assertions>
        </state>
    </states>
    <migrations>
 
    ...
 
    </migrations>
</resource>

Instance

Having defined a resource, you can use that resource definition to create or migrate many instances of the resource. The information that defines a given instance of a resource is defined in a file with the suffix *.wbinstance.xml or *.wbi. Each type of resource requires different information to be specified. The following instance descriptor is for a MySQL database:

<instance type="MySqlDatabase">
    <hostName>10.12.48.27</hostName>
    <port>3306</port>
    <adminUsername>root</adminUsername>
    <adminPassword>password</adminPassword>
    <schemaName>ProductCatalogueStaging</schemaName>
</instance>

Command-Line

Introduction

Wildebeest includes a command-line interface called wb.

Migrate

To migrate an instance of a resource to a new state, we use the migrate command:

wb migrate --resource:database.wbr --instance:staging.wbi --targetState:"Core Schema Loaded"
 
2013-06-09T19:05:10 - Wildebeest is migrating
2013-06-09T19:05:10 - Migrating from non-existent to "Database Created"
2013-06-09T19:05:10 - Migration complete
2013-06-09T19:05:10 - Assertion "Database exists" passed: Database ProductCatalogueStaging exists
2013-06-09T19:05:10 - Migrating from state "Database Created" to "Core Schema Loaded"
2013-06-09T19:05:11 - Migration complete
2013-06-09T19:05:11 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T19:05:11 - Assertion "Product table exists" passed: Table Product exists

The wb program supports short switches, so the same migrate command can be expressed like this:

wb migrate -r:database.wbr -i:staging.wbi -t:"Core Schema Loaded"
 
2013-06-09T19:28:25 - Wildebeest is migrating
2013-06-09T19:28:25 - Migrating from non-existent to "Database Created"
2013-06-09T19:28:26 - Migration complete
2013-06-09T19:28:26 - Assertion "Database exists" passed: Database ProductCatalogueStaging exists
2013-06-09T19:28:26 - Migrating from state "Database Created" to "Core Schema Loaded"
2013-06-09T19:28:26 - Migration complete
2013-06-09T19:28:26 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T19:28:26 - Assertion "Product table exists" passed: Table Product exists

The migrate command allows the target state to be specified either using it's label, or using it's ID. Labels are optional for states, so if a state does not have a label you can migrate to it using it's ID. Note that if the state specified by ID has a label, then Wildebeest will show that label when reporting progress:

wb migrate -r:database.wbr -i:staging.wbi -t:363568f1-aaed-4a50-bea0-9ddee713cc11
 
2013-06-09T19:30:09 - Wildebeest migrate -r:database.wbr -i:staging.wbi -t:363568f1-aaed-4a50-bea0-9ddee713cc11
2013-06-09T19:30:09 - Migrating from non-existent to "Database Created"
2013-06-09T19:30:10 - Migration complete
2013-06-09T19:30:10 - Assertion "Database exists" passed: Database ProductCatalogueStaging exists
2013-06-09T19:30:10 - Migrating from state "Database Created" to "Core Schema Loaded"
2013-06-09T19:30:10 - Migration complete
2013-06-09T19:30:10 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T19:30:10 - Assertion "Product table exists" passed: Table Product exists

JumpState

Sometimes you need to tell Wildebeest that a resource instance is in a particular state without being able to migrate into that state.

The most common example of this situation is where you have an instance that already exists and has been developed and managed through some means other than Wildebeest. In such a situation you can define a resource with a state that represents the current state of the instance, and then use the jumpstate command to have Wildebeest start tracking from that state

The arguments to the jumpstate command are the same as for migrate.

Let's say for example we had already created our ProductCatalogueStaging database using raw SQL scripts. To put this under Wildebeest management, we would define our database.wbr resource with a state called "Existing" that represents the current state, with assertions to verify that the current state is as we believe it is. Then we would use the jumpstate command to put the resource under Wildebeest management as follows:

wb jumpstate -r:database.wbr -i:staging.wbi -t:Existing

2014-01-12T16:49:22 - Assertion "Database exists" passed: Database ProductCatalogueStaging exists
2014-01-12T16:49:23 - Assertion "ProductType table exists" passed: Table ProductType exists
2014-01-12T16:49:23 - Assertion "Product table exists" passed: Table Product exists

When performing jumpstate, Wildebeest checks the assertions for the target state against the current state of the resource instance. If the assertions pass, then Wildebeest updates it's state tracking meta data to the target state. From that point on the migrate command may be used to migrate the resource from the state that was jumped to, to some other state.

State

Using wb you can also check on the current state of a resource using the state command:

wb state -r:database.wbr -i:staging.wbi
 
2013-06-09T19:47:08 - Wildebeest is checking state
2013-06-09T19:47:08 - Current state: Core Schema Loaded
2013-06-09T19:49:11 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T19:49:11 - Assertion "Product table exists" passed: Table Product exists

As you can see, the state command also verifies the resource. If the resource is found not to be valid according to the assertions for the current state, Wildebeest will let you know:

wb state -r:database.wbr -i:staging.wbi
 
2013-06-09T19:49:10 - Wildebeest is checking state
2013-06-09T19:49:11 - Current state: Core Schema Loaded
2013-06-09T19:49:11 - Assertion "ProductType table exists" passed: Table ProductType exists
2013-06-09T19:49:11 - Assertion "Product table exists" failed: Table Product does not exist

Reference

Database Resources

Name Description Examples
RowExists

Asserts that a query results in exactly one row.

<assertion
    type="RowExists"
    id="c1ea9cfb-bbf5-4262-8512-4bc13ebb05a4"
    name="ProductType HW exists">
    <sql><![CDATA[
        SELECT * FROM ProductType WHERE ProductTypeCode = 'HW';
    ]]></sql>
</assertion> 
RowDoesNotExist

Asserts that a query results in zero rows.

<assertion
    type="RowDoesNotExist"
    id="c8b0941e-83bc-4151-83e3-8ca633735fbf"
    name="ProductType XY does not exist">
    <sql><![CDATA[
        SELECT * FROM ProductType WHERE ProductTypeCode = 'XY';
    ]]></sql>
</assertion>
SqlScript

Migrates a database resource from one state to another by applying a SQL script.

<migration
    type="SqlScript"
    id="8b57f16d-c690-4f10-b68f-6f1ee75fe32b"
    fromStateId="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7"
    toStateId="363568f1-aaed-4a50-bea0-9ddee713cc11">
    <sql><![CDATA[

/* ProductType */
CREATE TABLE  `ProductType` (
  `ProductTypeCode` char(2) NOT NULL,
  `Name` varchar(10) NOT NULL,
  PRIMARY KEY (`ProductTypeCode`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO
    ProductType(ProductTypeCode, Name)
VALUES
    ('HW', 'Hardware'),
    ('SW', 'Software');

</migration>

SQL Server Database Resources

To define a SQL Server database resource, set the type of the resource to SqlServerDatabase:


<?xml version="1.0"?>
<resource type="SqlServerDatabase" id="58699f8a-22fa-4784-9768-3fcc3b2619b4" name="Product Catalogue Database">

    ...

</resource>

A SQL Server database instance defines the connection settings for the host, the database name and an optional state table name.


<instance type="SqlServerDatabase">
	<hostName>127.0.0.1</hostName>
	<port>1433</port>
	<instance>MainInstance</instance>
	<adminUsername>wb</adminUsername>
	<adminPassword>password</adminPassword>
	<databaseName>ProductCatalogueStaging</databaseName>
	<stateTableName>WbState</stateTableName>
</instance>

Name Description Examples
SqlServerCreateDatabase

Typically this will be the first migration in the definition of any SQL Server database resource managed by Wildebeest. It creates a new database.

This migration takes the name of the database to create from the SqlServerDatabaseInstance it is applied to.

<migration
    type="SqlServerCreateDatabase"
    id="48e9b89a-e3be-4418-ace7-c008fcacc32f"
    toStateId="9aeeba93-6890-4690-b7b2-afa158ae6556">
</migration>
SqlServerCreateSchema

Creates a schema in a SQL Server database resource.

<migration
    type="SqlServerCreateSchema"
    id="675db93a-bd49-42a0-b36e-eef861c661f7"
    fromStateId="81806637-adbe-4123-9677-b8da2333c1a9"
    toStateId="85819eed-05d8-4cee-a34c-ff9b64f6d72b">
    <schemaName>prd</schemaName>
</migration>
SqlServerDatabaseDoesNotExist

Verifies that a SQL Server database resource does not exist.

<assertion
    type="SqlServerDatabaseDoesNotExist"
    id="4acd5c3a-4a12-4e27-91f9-56945a2e4415">
</assertion>
SqlServerDatabaseExists

Verifies that a SQL Server database resource exists.

<assertion
    type="SqlServerDatabaseExists"
    id="c5eccbf0-3d2a-4906-8f46-c380517628d7">
</assertion>
SqlServerDropSchema

Drops a schema from a SQL Server database resource.

<migration
    type="SqlServerDropSchema"
    id="ffe636f4-563f-4725-bcf2-124e7bb38d76"
    fromStateId="0cae6740-cb35-4028-af8a-14d565414078"
    toStateId="cc24394e-0f5b-42b9-8216-c95c81ff07dc">
    <schemaName>prd</schemaName>
</migration>
SqlServerSchemaDoesNotExist

Verifies that a schema does not exist in a SQL Server database resource.

<assertion
    type="SqlServerSchemaDoesNotExist"
    id="b7b780b0-ed01-4f5d-b1b1-f2f503ebfeaf"
    <schemaName>prd</schemaName>
</assertion>
SqlServerSchemaExists

Verifies that a schema exists in a SQL Server database resource.

<assertion
    type="SqlServerSchemaExists"
    id="f4bb0ea1-0913-4447-815f-bc3152da1ad1"
    <schemaName>prd</schemaName>
</assertion>
SqlServerTableDoesNotExist

Verifies that a table does not exist in a SQL Server database resource.

<assertion
    type="SqlServerTableDoesNotExist"
    id="76455254-dc96-4707-8348-41213bb4aa58"
    <tableName>product</tableName>
</assertion>
SqlServerTableExists

Verifies that a table exists in a SQL Server database resource.

<assertion
    type="SqlServerTableExists"
    id="5ad24640-3c2d-42a5-9bc2-1f49dbfced61"
    <tableName>product</tableName>
</assertion>

MySQL Database Resources

Name Description Examples
MySqlCreateDatabase

Typically this will be the first migration in the definition of any MySQL resource managed by Wildebeest. It creates a new schema.

This migration takes the name of the schema to create from the MySqlDatabaseInstance it is applied to.

<migration
    type="MySqlCreateDatabase"
    id="6b21e1e3-ff3a-44b3-84ec-e21fb01c0110"
    toStateId="199b7cc1-3cc6-48ca-b012-a70d05d5b5e7">
</migration>
MySqlDatabaseExists

Used to assert that the database described by the instance exists.

<assertion
    type="MySqlDatabaseExists"
    id="19b64fb6-f95f-4221-bc64-303f62a3eda8"
    name="Database exists" />
MySqlDatabaseDoesNotExist

Used to assert that database described by the instance does not exist.

<assertion
    type="MySqlDatabaseDoesNotExist"
    id="74204122-63fd-40f3-9b12-2b62e432a371"
    name="Database exists" />
MySqlTableExists

Used to assert that a specific table exists within the database.

<assertion
    type="MySqlTableExists"
    id="3808ba63-f055-4bf7-88fe-023546e6ed16"
    name="ProductType table exists">
        <tableName>ProductType</tableName>
</assertion>
MySqlTableDoesNotExists

Used to assert that a specific table does not exist within the database.

<assertion
    type="MySqlTableDoesNotExist"
    id="f112b36f-8312-4f8a-b7a5-e30d17be3c9b"
    name="ProductType table does not exist">
        <tableName>ProductType</tableName>
</assertion>

Releases

Versioning Scheme

Wildebeest's version number is structured as follows:

  • Release
  • Update
  • Revision (derived from the revision control commit)
  • Build (incremented for each build performed by the CI server)

Wildebeest is released four times per year in January, April, July and October. The release number in the version is incremented for each scheduled release.

From time to time we may release extra off-schedule releases with enhancements and fixes. In these cases the update number is incremented

History

Release Date Description Download
3.0.120.91 2014-01-12
  • Created the jumpstate command, to support scenarios where an existing resource needs to be placed under Wildebeest management
  • State-tracking by resource in MySql and SqlServer database resources, to support compositing separate Wildebeest resource definitions in a single instance
  • Added support for SLF4J logging to better support embedded Wildebeest scenarios
  • Fixed some minor non-blocking but irritating bugs in the command-line interface
  • Added a l33t ascii banner to the command-line interface
  • Added Releases section to the website, documenting the version numbering scheme and past releases
ZD.Wildebeest-3.0.120.91.zip
2.0.89.63 2013-10-13
  • Added first-class support for SQL-Server with a suite of plugins for managing SQL-Server databases as Wildebeest resources
  • Improvements to the web-based documentation
  • Dropped name from the assertion model as it's generally redunant and just makes work
  • Cleaned up namespaces, expanded Javadocs and general code cleanup
ZD.Wildebeest-2.0.89.63.zip
1.0.0.3 2013-08-04 First public release of Wildebeest, providing the core resource, instance, state, assertion and migration model, and with plugins for managing MySql databases as Wildebeest resources. ZD.Wildebeest-1.0.0.3.zip