Proxy to split writes and reads

[gedw99]

Is there a way to force writes to one db and read to another ?

this would enable scaling patterns where writes go to origin db and reads go to slave dbs.

it’s definitiv possible since you have a http proxy .

i guess the thing is that the concept of many db instances of the same db might need to be added.

Basically a map of db end points with a tag of what’s origin and what’s slave.

hope this is not out of scope of your projects.

Background:

Under the hood it’s common to setup using CDC based replication of db so that you can scale out your db.
The classic way is to do the old master / slave pattern.
if origin falls over then an election happens ( using some control plane like NATS agent ) and one of the slaves is voted to be the new master.
with worum of 3 db , then your down to 2 and have time ( hopefully ) to get your cloud to create another one off the latest snapshot off s3.

[gedw99]

For example

https://github.com/maxpert/marmot

It just demonstrates a control plane to watch may db instances .,

this ones a little different in that it’s multi master. So your http proxy would I guess just load balance in that can and watch health data and then remove db instances.

[gedw99]

Btw, not sure yet if this is applicable to duckdb since that uses s3 which is by nature clustered. your duckdb instance can of course also die, so I guess it is. But not sure yet . I don’t actually know if I can have 3 duckdb I stances pointing to 1 s3. Do you know ?

[flarco]

Hi, interesting idea. This is pretty out of scope for me :). Seems like something you could set up with caddy?
Basically run multiple instances of db-rest, or even just one since it can handle multiple databases, and have caddy forward your requests to multiple endpoints.

[flarco]

For duckdb, all S3 data sourcing would be handled internally (inside duckdb). But I suppose you could have 3 duckdb instances reading from the same S3, don't see why not. https://duckdb.org/docs/guides/import/s3_import.html