Flyway Migration Sets and Spring Boot
Database migrations are a crucial aspect of modern software development, ensuring that your application database structure is consistent across environments, including local development. In this post, I show how to support migration sets - which is critical when dealing with multiple schemas, multiple databases, or even multiple database providers.
But first...
What are Database Migrations?
Database migrations, often simply referred to as migrations, are a set of structured changes made to a database schema or its data. Migrations provide a version-controlled approach to modifying database structure, ensuring that every change to the schema is tracked, reversible, and repeatable.
Migrations can be automated in a deployment pipeline. When a new version of an application is released, the associated migrations are run, ensuring the database schema matches the application expectations.
Migrations also ensure that all instances of a database - whether that be production, non-production, test-harnessed, or local development - are using the same database schema.
If you want to learn more about database migrations, this post is a good primer.
What is Flyway?
Spring Boot includes a module for the community edition of Redgate Flyway, a widely-used open-source tool that facilitates version control for databases. Flyway supports numerous databases, including PostgreSQL, MySQL, Oracle, and many, many more.
What are Migration Sets?
A migration set is a distinct collection of migration files for a given database schema. When do we need support for migration sets?
Single Database, Multiple Schemas
Your application might interact with multiple schemas in the same database.
Multiple Databases, Autonomous Domains
Your overall system may be comprised of autonomous domains, having their own database schema and thus, their own migrations.
Multiple Databases, Multiple Providers
More likely your scenario does not fit neatly into the two previous descriptions. Your organization may have numerous databases, some with multiple schemas, across numerous database providers. Some combination of SQL, NoSQL, and other database types is more common than not. This is where a custom migrator can be invaluable. More on custom migrators later.
How to Run Migrations?
Database schema or data dependencies may be altered during development cycles, and when the cycle culminates in a deployment we need to apply those alterations as one of the very first steps of deployment. In fact, in almost all cases where a development cycle includes alterations we must be able to abort deployment if the migrations fail.
Whether we are unfortunate enough to be doing manual deployment, or we employ continuous deployment, we want to execute the deployment in the following order:
- Deploy migration (e.g.,
helm upgrade -f database-migrator.yml) - Check migration
- If migration failed then compensate and abort release.
- If migration succeeded then proceed with release.
- Deploy remainder of release
How we "deploy migration" depends on the database migration tool, but in general there are two techniques: use the migration tool itself, or use a custom migrator.
Use Migration Tool
Database migration tools like Flyway bundle a command line interface (CLI) and this should be the go to technique. Using Flyway as an example, we would execute something like below to "deploy migration".
When your database migration tool CLI is insufficient for your particular situation, then you should consider using a custom migrator.
Use Custom Migrator
How we execute our custom migrator is entirely dependent on our customization, but we must be able to pass runtime arguments in a secret manner. At minimum we must be able to set environment variables as runtime arguments, but better would be to use a secrets manager to inject the runtime arguments. Whatever you do, do not pass secrets directly on the command line.
Creating a Custom Migrator
Enough theory and definitions, let's walk through creating a custom migrator using Spring Boot and Flyway. (If I get enough interest I'll write a future post using your particular stack.) For this post I have created a production-viable example that you can clone and use to follow along - or feel free to head over to Spring Initializr and create your own project.
Add Dependencies
You will need the following minimum dependencies. For this example I have used MySQL, but swap in whatever your database vendor is.
Create Migration Sets
You will need a folder structure for your migration files. Let's create a structure that supports multiple sets of migrations (migration sets).
Be sure to locate these under the resources folder.
Create Configuration
Next we configure our application.properties with the keys and values needed to support Flyway migrations (and migration sets).
The directory structure and these properties are the keys (no pun intended). Without the properties set, Flyway defaults to using db/migration for migration files - which of course means a single migration set. By setting spring.flyway.locations and spring.flyway.schemas we specify exactly which migration set to apply.
There's more to this post, but pause for a moment and just think about that. Think about the options that migration sets opens up for you in your organization.
Create Runnable
By default Spring Boot's Flyway integration will run migrations at startup. Thus the application code is dead simple.
That's pretty much it. As I mentioned, the example I created, flyway-migrator, is production-ready, so feel free to use it.
Conclusion
Using Spring Boot with Flyway makes managing and running database migrations straightforward. By supporting multiple folders of migration files, you can structure your migrations in a way that matches your needs. With this setup, you can maintain cleaner, more organized migration files that align with your development strategy.
I hope you found this post useful. Feel free to hit me up @RjaeEaston.
P.S. I have been using Spring Boot as a backdrop for several recent posts, but, with .NET 8 RC1 out, my next few posts will center around C# and .NET. Looking forward to sharing some ideas that have been rattling around in my head for some time now.
Code well, and shape the future.