Schema Migrations In HelixDB

Introduction

We are very excited to release schema migrations in Helix! All users can now lazily migrate data between versions, meaning you don’t have to do any big migrations or have any downtime when updating your schemas.

Our lazy migrations apply the migration function on your data at read time to present you the most up to date version of your data. And all new data or updates to existing data will be inserted as the newest version of your schema.

To simplify this..

Reads

The data in storage stays in the form of the old version (until the item is updated via a query)
New fields with default values display their default value.
Old redundant fields are kept until an update is made

Updates

Updating items of an older version will overwrite the old version of the item to the new one.
Old field data is hence deleted once this happens

Writes

All new data is inserted as the newest version

HQL Schema Example

Below is a standard HQL schema. This is where you define all of your node, edge and vector declarations.

N::User {
    name: String,
    age: I32,
    bio: String
}
N::Post {
    title: String,
    content: String
}
E::CreatedPost {
    From: User,
    To: Post,
    Properties: {
        since: I32,
        comment: String
    }
}

Getting Started

Here you will see how we

Update field names
Add new fields
Remove fields
Set default values
Cast between types

To start off with, you will need to wrap your original schema in schema::1 {}

Defining Version 1

schema::1 {
    N::User {
        name: String,
        age: I32,
        bio: String,
    }
    N::Post {
	    title: String,
	    content: String
    }
    E::CreatedPost {
        From: User,
        To: Post,
        Properties: {
            since: I32,
            comment: String,
        }
    }
}

Once that’s done, you can go ahead and outline your new schema.

Defining your new version (Version 2)

schema::2 {
    N::User {
        username: String,
        given_age: U32,
        bio: String,
    }
    N::Post {
        mainTitle: String,
        content: String,
        like_count: I32
    }
    
    E::CreatedPost {
        From: User,
        To: Post,
        Properties: {
            created_at: I32,
            updated_at: I32,
        }
    }
}

Note that schema versions can be in the same files (or different ones).

As you can see above, the items in schema 2 have the same type name (i.e. the type name User in N::User stays the same between schema version 1 and 2), but the fields within User change between version 1 to version 2.

To migrate data from one schema to another, we have built migration functions into HQL that allow you to define how your data should change between versions. Remember, these migrations are lazily evaluated and do not require huge data migrations or downtime.

Defining your migration

MIGRATION schema::1 => schema::2 {
    N::User => _::{
        username: name,
        given_age: age AS U32,
        ..
    }
    N::Post => _::{
        mainTitle: title,
        like_count: 0,
        ..
    }
    
     E::CreatedPost => _::{
        Properties: {
            created_at: since,
            updated_at: since,
        }
    }
}

It does the following for each type:

N::User :

It renames name to username .
It renames age to given_age and updates the type (from I32 ) to U32 .
It uses the spread symbol, .. , to say that the rest of the properties of the version 1 User items should remain the same. In this case this means all new items will keep the same bio property.

N::Post :

It renames title to mainTitle.
Adds a new field called like_count and sets all migrated Post nodes to 0 when they’re being lazily evaluated.
Again, using the spread symbol .. to keep the rest of the properties from version 1 as is. In this case the contentfield.

E::CreatedPost :

It creates creates two new fields, created_at and updated_at , using the value of the since field from version 1 as the value for both created_at and updated_at .
It also leaves out the comment field so all version 2 CreatedPost edges that are read won’t have the commentfield. In other words, the existing data will just be ignored.

Running migrations

To run your migrations, all you need to do is deploy them to your instance using helix deploy . Once the instance has been redeployed the migrations will be live! It's as easy as that.

Notes:

The migrations are still in beta, and have some limits to their functionality.

You can currently only define forward migrations. e.g. V1 ⇒ V2, but not V2 ⇒ V1. This is coming soon.
You can only modify fields between migrations. Changing type names or graph structure is not supported. This will be coming eventually but it requires a huge amount of work to complete so will take time.
You can only make migrations on nodes and edges. Vector support is coming next!

If you run in to any problems, get in touch with us in our discord server, or email us founders@helix-db.com