Database stuck as replica after 'fly postgres backup enable' on legacy Postgres app

[sprytna]

My production database cluster is completely broken after running fly postgres backup enable.

App: sprytna-cards-db
Region: ams

Issue:

• Ran fly postgres backup enable on unmanaged postgres
• Now cluster cannot register primary: “failed to re-register existing primary: failed to
  register primary: exit status 1”
• All connections show pg_is_in_recovery() = true (all replicas)
• Production app cannot write data for 2+ hours

Status shows:

• 1 primary (17814501c41208)
• 2 replicas (784961b2223698, 2871339a429e08)

But primary cannot register and all connections route to read replicas.

This appears to be a bug where backup enable command is not compatible with unmanaged postgres clusters and should have never been allowed to run via fly cli.

Need urgent help to either:

1. Fix the primary registration
2. Guidance on safely restoring from backup

[sprytna]

And I can’t even upgrade my plan to add service because I have to wait until the end of the billing cycle

[mayailurus]

Hi… You might try creating a new, copied database cluster using the volume-forking technique (with an explicit volume ID).

(You may also need an explicit --image-ref flag, if the original app is older.)

It’s a little inelegant, but it’s the simplest to describe here in the community forum.

@sprytna

I can’t even upgrade my plan to add service because I have to wait until the end of the billing cycle

Fly Support doesn’t cover Legacy Postgres, anyway, only Managed Postgres. (The fact that the latter is supported is one of its primary advantages.)

In other contexts, though, you could email [email protected] to get your plan activated earlier.

Hope this helps!

[sprytna]

Thanks! Yeah, migrating to managed cluster solved everything,