I'm getting divergent versions when deploying to my production environment

[thomas-tribus]

Hi,

I just wanted to do a quick check to see if I'm missing something super obvious here. I'm having some issues running my migrations locally vs in a GKE cluster.

I'm not sure if it is a bug, or if I'm simply doing something strange.

My app is containerized and connects through postgres through a proxy (GCP Cloud SQL Proxy). I can run the same setup connecting to the same database from my laptop and from GKE. On my laptop I can run it directly, or through a local k8s cluster (just to keep everything the same).

I always use the same proxy image and always target the same database.

On my local machine, regardless of the deployment scenario the refinery always correctly detects that the migrations have been applied already. However, every time I run on GKE the migration fails because it has detected a divergent migration.

I think it must somehow related to the checksum field, and some differences between my local and GKE setups that causes the checksums to diverge...

I can work around the issue by set_abort_divergent(false), but that doesn't feel really safe either.

I've run out of things to test, so I'm really hoping this sounds familiar to somebody. Thanks!

[jxs]

Hi, and thanks for the interest!
So yeah divergent migrations are verified by comparing the name and the checksum of the migrations applied vs the ones on the filesystem. I don't know what might be causing your problem but I would suggest comparing the mentioned migration checksum on both environments, If you want we can add the checksums of the divergent migrations to the error log here

[thomas-tribus]

Thanks for the reply! I’ll look at those checksums when I get a chance, is there any way I can compute those checksums myself? Looking though the source it seems it is just based on the sql (in my case) of the migration. So hypothetically running that computation should / could return something different on both systems? Adding the checksum to the log would be helpful for me. I’ll let you know what I find out, probably by Friday.

On Wed 21. Dec 2022 at 18:46, João Oliveira ***@***.***> wrote: Hi, and thanks for the interest! So yeah divergent migrations are verified by comparing the name and the checksum of the migrations applied vs the ones on the filesystem <https://github.com/rust-db/refinery/blob/main/refinery_core/src/traits/mod.rs#L19-L47>. I don't know what might be causing your problem but I would suggest comparing the mentioned migration checksum on both environments, If you want we can add the checksums of the divergent migrations to the error log here <https://github.com/rust-db/refinery/blob/main/refinery_core/src/traits/mod.rs#L39> — Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASHQ42WEGYHVKX6NHUB3D3TWOM66VANCNFSM6AAAAAATF2F3TA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Thomas

[jxs]

I’ll look at those checksums when I get a chance, is there any way I can
compute those checksums myself?

yeah, see here how we do it

[thomas-tribus]

Cool, I’ll let you know how it goes 👍

On Thu 22. Dec 2022 at 13:11, João Oliveira ***@***.***> wrote: I’ll look at those checksums when I get a chance, is there any way I can compute those checksums myself? yeah see here <https://github.com/rust-db/refinery/blob/main/refinery_core/src/runner.rs#L102-L124> how we do it — Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASHQ42ROUJAITRJHTDTMCGDWORAQZANCNFSM6AAAAAATF2F3TA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Thomas

[thomas-tribus]

Ok, quick update: I have confirmed that the GKE deployed version computes a different hash vs the local version (even if run through docker). We expected that, but it's still good to have confirmation. I'll spare you the hashes themselves, as they are not useful here I think? Next step would be to log all three inputs of the hash and to double check the exact version of siphasher. I'm out of time for now, but I'll pick it up again next week. In the meantime brilliant ideas are of course welcome ;-).

…

On Thu, Dec 22, 2022 at 1:18 PM Thomas Duindam ***@***.***> wrote: Cool, I’ll let you know how it goes 👍 On Thu 22. Dec 2022 at 13:11, João Oliveira ***@***.***> wrote: > I’ll look at those checksums when I get a chance, is there any way I can > compute those checksums myself? > > yeah see here > <https://github.com/rust-db/refinery/blob/main/refinery_core/src/runner.rs#L102-L124> > how we do it > > — > Reply to this email directly, view it on GitHub > <#265 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ASHQ42ROUJAITRJHTDTMCGDWORAQZANCNFSM6AAAAAATF2F3TA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Thomas

[jxs]

are you using the same refinery version in both places? We updated it to specifically use SipHasher13 whereas before it was using rust's Hasher. Also for the sake of discarding hypothesis can you list the local operating system?

[thomas-tribus]

Having had 30 more minutes to look at this, I'm thinking 'probably not'. I can reproduce the hashing issue locally using my production docker image, before I was testing using a newly build image. I had assumed both to be the same, but probably something went wrong with caching. I have a hunch it might be related to cargo chef thinking that two version are compatible when they are not. But I would have to look into it a bit more to be sure. Do you know in what version you changed the hashing algorithm?

…

On Fri, Dec 23, 2022 at 7:35 PM João Oliveira ***@***.***> wrote: are you using the same refinery version in both places? We updated it to specifically use SipHasher13 whereas before it was using rust's Hasher. Also for the sake of discarding hypothesis can you list the local operating system? — Reply to this email directly, view it on GitHub <#265 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASHQ42US632LIYQP5BAMBPDWOXWFZANCNFSM6AAAAAATF2F3TA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jxs]

yeah, see here

[thomas-tribus]

Thanks! Now I understand it even less 😢

I target the latest version explicitly:

refinery = {version="0.8.7", features=["tokio-postgres"]}
siphasher = "0.3.10"

I created a test function to just generate the hash. Running it locally it produces the same checksum as refinery does (also locally).

However, when running this function in my container it produces the same hash as it does locally, however refinery comes up with a different hash.

I appreciate your replies, and sorry for being slow to respond (holiday season). I have the feeling I'm missing something obvious somewhere.... But I really can't see it.

Some snippets:
Test function:

ub fn dump_hash() -> Result<u64, Error> {
    let input_name = "V1__workup.sql";
    let mut file = File::open("./migrations/V1__workup.sql")?;
    let mut sql = String::new();
    file.read_to_string(&mut sql)?;
    let captures = RE
        .captures(input_name)
        .filter(|caps| caps.len() == 4)
        .ok_or_else(|| Error::ConnectionError("Invalid name".to_string()))?;
    let version: i32 = captures[2]
        .parse()
        .map_err(|_| Error::ConnectionError("Invalid version".to_string()))?;

    let name: String = (&captures[3]).into();

    // Previously, `std::collections::hash_map::DefaultHasher` was used
    // to calculate the checksum and the implementation at that time
    // was SipHasher13. However, that implementation is not guaranteed:
    // > The internal algorithm is not specified, and so it and its
    // > hashes should not be relied upon over releases.
    // We now explicitly use SipHasher13 to both remain compatible with
    // existing migrations and prevent breaking from possible future
    // changes to `DefaultHasher`.
    let mut hasher = siphasher::sip::SipHasher13::new();
    name.hash(&mut hasher);
    version.hash(&mut hasher);
    sql.hash(&mut hasher);
    let checksum = hasher.finish();
    Ok(checksum)
}

Migration (V1__workup.sql)

 CREATE TABLE workup
 (
     id          SERIAL PRIMARY KEY,
     user_id     VARCHAR(50)              NOT NULL,
     session_id  UUID UNIQUE,
     series_id   UUID                     NOT NULL,
     last_update timestamp with time zone NOT NULL,
     data        JSONB                    NOT NULL
 )

Output in container:

2023-01-03T16:30:06.030Z WARN  database::adapter > Computed migration hash 11227837385157790651
 2023-01-03T16:30:06.074Z INFO  tokio_postgres::connection > NOTICE: relation "refinery_schema_history" already exists, skipping
 2023-01-03T16:30:06.192Z ERROR viewer_service             > Could not connect to backend database, check connection string in env TMB_DB_CONNECTION_STRING MigrationError(Error { kind: DivergentVersion(Migration { state: Applied, name: "workup", checksum: 10155571909293699195, version: 1, prefix: Versioned, sql: None, applied_on: Some(OffsetDateTime { local_datetime: PrimitiveDateTime { date: Date { year: 2022, ordinal: 362 }, time: Time { hour: 15, minute: 47, second: 30, nanosecond: 425080977 } }, offset: UtcOffset { hours: 0, minutes: 0, seconds: 0 } }) }, Migration { state: Unapplied, name: "workup", checksum: 11227837385157790651, version: 1, prefix: Versioned, sql: Some(" CREATE TABLE workup\r\n (\r\n     id          SERIAL PRIMARY KEY,\r\n     user_id     VARCHAR(50)              NOT NULL,\r\n     session_id  UUID UNIQUE,\r\n     series_id   UUID
      NOT NULL,\r\n     last_update timestamp with time zone NOT NULL,\r\n     data        JSONB                    NOT NULL\r\n )"), applied_on: None }), report: None })

For clarity, the output of the test function locally Dumping hash: Ok(11227837385157790651)

[thomas-tribus]

I have tried different things over the past few weeks (on and off) but I still can't get a handle on this.

If we could add the checksum logging? I have a hunch that maybe the process of embedding the SQL might trigger differences in the checksum computation: I get consistent checksums when I compute them manually, but in some cases the manually computed checksum differs from the checksum computed by refinery.

That would indicate that one of the inputs is different, which leads me to my hunch. Hopefully with logging in place we can clear that out a bit more.

[jxs]

yeah you can do that, you can do that locally and then install with cargo install