Feedback on Hasura migrations

[akiran]

I tried migration as mentioned in hasura docs and ran in to some issues.

1. I started creating tables from hasura dashboard with running hasura console. So migration files are not generated. I want to bring in the schema to migrations. I followed this guide to generate first migration https://docs.hasura.io/1.0/graphql/manual/migrations/database-with-migrations.html

When I tried to apply this migration on a new database, I got this error

Error: apply failed: [postgres-error] query execution failed ($.args[0].args)
File: ''
[42P06] FatalError: schema "public" already exists

I had to comment this line CREATE SCHEMA public;
from public-schema.sql which is generated with command pg_dump -O -x -h <db-host> -p <db-port> -U <db-user> -d <db-name> --schema public --schema-only > public-schema.sql

Commenting/removing CREATE SCHEMA public; from public-schema.sql is not mentioned in the documentation.

2. After generating first migration, I ran hasura console and started adding new columns from the hasura dashboard and migration are created automatically. But hasura migrate apply failed with error "table already exists".
   hasura migrate status results in

VERSION        SOURCE STATUS  DATABASE STATUS
1544721345907  Present        Not Present
1544721838932  Not Present    Present
1544721935023  Not Present    Present

I think, first migration is not added in hdb_catalog.schema_migrations.
Can you update docs about how to add first migration generated from existing database to schema_migrations

3. Currently new migration files are generated for each schema change. This could result in lot of migration file when the project is newly started. It would be better if there is an option to make few changes and save a single migration file for all these changes at once.
   For example: Create a table and add 20 columns of this table and save a migration file. (Currently system creates 21 migration files for this case)

4. Can you show hdb_catalog.schema_migrations table in hasura dashbaord. Useful when debugging issues with migrations

5. Is it possible to rollback migations ? I couldn't find docs related to rollback of migrations.

[shahidhk]

Hi @akiran, thanks for the feedback. This is very helpful 👍

1., 2.:

Based on the commands, I think you have followed this guide: https://docs.hasura.io/1.0/graphql/manual/migrations/existing-project.html. Please do correct me if I am wrong.

We have not addressed the use case for creating migrations from an existing project (db) and continuing to use that project (db) with migrations. The guide addresses the use case for creating migrations from an existing project (db) and applying it on a new project (db) and continue using that. This is also the reason for first migration not appearing in the status output.

We'll address this gap immediately.

Reg. the CREATE SCHEMA public, we have not seen this statements in any of the dumps we created. Can you let us know the Postgres version? We'll document that too.

Combining migrations files is a good suggestion. Something like a record and then save. But, this is only going to be an aesthetic change, since under the hood it'll be the same. Even if you have 800 migrations (we have users who have this many 😄), all of them will be applied at once in a single postgres transaction, when migrate apply is executed. Let us know what you think.

Showing the version status on Console is informative. @praveenweb Can you create off an issue and add this?

Yes, rolling back migrations are possible with the --down command for migrate apply command. For example, hasura migrate apply --down 12 will apply 12 down migrations from current point. hasura migrate apply --version <version> --type down can rollback a particular migration version. Note that down migrations are not generated for custom SQL executed from console.
We are also planning to add --goto <version> flag which will take you up or down the migration ladder.

[akiran]

@shahidhk Thanks for the prompt response.

I am using the Postgres from docker file given by hasura.

Regarding 3: I am interested in consolidated migration files for better maintainability of code. If we have all schema changes related to a feature in one file, it will be easy to refer back in future

[shahidhk]

Got it. I'll get back after checking a latest Postgres dump.

Yeah, it makes sense to group together migrations for maintainability. Will iterate over some ideas and get back. In the meanwhile, do let us know if you have any suggestions/ideas.

[tranqy]

For what it's worth, I followed the migration guide, but when applying to the existing database changed my initial migration to a select, then put the db dump back in the migration file.

It also looks like running a schema creates a row in hbd_catalog.schema_migrations with the form #,###,###,###,### sourced from the number leading the migration. For example I had 1553035567544_schemaname.sql, in the db the version is 1,533,035,587,544, dirty false., so inserting a row here may skip the initial migration.

[shahidhk]

@tranqy You're right. Whenever migration is applied, it's version (which is unix timestamp in nanoseconds by default) is entered into hdb_catalog.schema_migrations table. We have added a new flag, which will let you update this table and mark a migration as applied without actually executing it. It should be available in the next release.

[afreix]

I got into a similar situation following https://docs.hasura.io/1.0/graphql/manual/migrations/existing-database.html. The source was a hasura instance running in an EC2 container

hasura migrate status shows this:

VERSION        SOURCE STATUS  DATABASE STATUS
1570497949573  Not Present    Present
1570670807668  Present        Not Present
1570673089502  Present        Not Present
1570759389539  Present        Not Present

followed by a bunch of migrations that have been successfully applied on both the SOURCE and the DATABASE. I'm not quite sure how I got into this situation and since there is no migration on the SOURCE for '1570497949573' I have no idea what this migration contains that was supposedly the first migration applied to the DATABASE

At first glance it looks like the DATABASE is in the same state as what I see when running hasura console

I'd like to get this cleaned up so that I don't hit errors any time I try to run hasura migrate apply. I'm not sure if this was a bug or me not following the guidelines precisely. Are there easy steps I could take to basically start over and get the SOURCE and DATABASE consistent?

I tried running hasura migrate apply --endpoint http://localhost:8080 --version 1570497949573 --type down but that failed

I apologize if this belongs elsewhere like StackOverflow or Discord, but it seemed relevant to the original issue

[shahidhk]

1570497949573 Not Present Present -- you can get into such a state when you use the console from multiple machines (or directories) and forgot to commit/share the migration files.

Are you the only developer working on the project?

[afreix]

Thanks for the prompt reply.

There is one other developer working on the project but the server in question is a local instance of hasura running in a container, which is why I was surprised. It should only be affected by my own actions and not the other developer

[Californian]

I'm not sure what I'm doing wrong, but at the very least it seems like your documentation is out of date so I figured I'd leave a comment here for that.

What I'm doing: trying to move a heroku-hosted hasura instance to my local machine for local development and to version control migrations/schema changes.

What I'm following: https://github.com/hasura/platform-docs/blob/master/platform/manual/guides/importing-existing-database.rst

What's wrong:

• first off, the commands in that guide above (still on master so I feel reasonable in expecting them to be accurate) refer to hasura commands, but it seems as though they are now hasura-cli commands, and the syntax differs still more from there.
• the schema dump leads to the error "FatalError: schema "hdb_catalog" already exists" upon attempts to run hasura-cli migrate apply

[tirumaraiselvan]

@Californian That link is not for Hasura GraphQL Engine. Try this link: https://hasura.io/docs/1.0/graphql/manual/migrations/index.html

[ronnycoding]

I had a similar problem on an existing project, I'm using docker, so I saw that hdb_catalog already exists, so I removed docker volumes running docker-compose down -v resetting my database, and I ran again the command hasura migrate apply and it worked @akiran

[andoks]

I also had a similar problem when following the basic migrations tutorial.

My initial (first) migration created from the initializing migrations step was registered as "Not Present" in the database. I "fixed" it by running hasura migrate apply --skip-execution --version <initial version ID>. If that is supposed to be how it is done I can add it as a step top the docs and send a PR.

When it comes to the subsequent steps re squashing steps to clean up, all the intermediary migrations created by the console are still registered as "Present" in the database after the squash is done (using hasura migrate squash --name <squashed migration name> --from <first WIP migration>), but they are no longer "Present" in source (when doing hasura migrate status).

This is a bit surprising (although for the sake of data integrety, I can understand why), and I would really like it if the docs said something about why this is how it is, and how to "clean up" the right way; am I just supposed to run `hasura migrate apply --skip-execution --version ? Or should I roll the database back and apply the squashed migration? Or should I have a second database to apply the squashed migration to, and do a manual diff between the results?

I also suspect that the Add Checkpoints step should include running hasura metadata export --endpoint <endpoint>?

That was a lot of suggestions, but I also want to say that a really like the docs in general 😄

[dchoigt]

I would like to rerun my migrations or at least a specific one. I can get the db sql migration to run by removing the entry inside of schema_migrations but I cannot get my metadata to run again. If I look at the hdb_metadata table it still has the older metadata instead of my new stuff. What do I do?

[typro33333]

I have the same issue "hdb_catalog" already exists. I have run pgdump database then I restore that database and hasura connect to it. I'm using Hasura on kubernetes and it has deployed by helm chart. So what i do next to fix this error ?