How to handle CouchDB deleted documents with replications

[obautista]

Using v.3.1.1 CouchDB. We are replicating a database to many clients. We are deleting many documents on the server. Is it suggested following be run on some kind of schedule (daily maybe):

• https://{database_url}/{database}/_compact
• https://{database_url}/{database}/_compact/{design_document}
• https://{database_url}/{database}/_view_cleanup

I understand we can create a filter and replication document to ignore deleted documents. Is this needed when compacting and cleaning up?

Alternatively, can purge be run instead or in addition to compact/cleanup?

• https://{database_url}/{database}/_purge

Should we consider upgrading CouchDB version in the event new features were introduced?

I am trying to be mindful of deletions and replications.

[nickva]

You can configure the auto-compaction service for dbs and views called "smoosh": https://docs.couchdb.org/en/stable/maintenance/compaction.html#compaction

In the latest version, 3.3.2, the auto-compaction service will also clean up old index files: https://docs.couchdb.org/en/stable/maintenance/compaction.html#channel-configuration. The index_cleanup channel effectively calls https://{database_url}/{database}/_view_cleanup. The trigger for it is when the view is compacted by smoosh.

I understand we can create a filter and replication document to ignore deleted documents. Is this needed when compacting and cleaning up?

You can filter out deleted documents when you replicate. This guide shows how to do it: https://blog.cloudant.com/2021/05/21/Removing-Tombstones.html. But be careful, if your client also needs know if a document has been deleted, then you'd want to allow that deletion to go through. It depends on your application logic.

After documents are deleted on the server, if you configured your auto-compactor it will trigger based on the ratio or the amount of disk space to be recovered (called slack in the docs and the channel configuration). Or like you suggested you can run the compaction manually, too.

Alternatively, can purge be run instead or in addition to compact/cleanup?

After some time, you can choose to _purge your deleted document. You'd make requests to the _purge endpoint with batches of doc_id and their revisions. This doesn't happen automatically as we don't your clients had replicated that particular document update yet. You could insert a timestamp in your deleted documents (delete using an update and remove the document body, except for {"_deleted": true, "ts": $some-kind-of-a-timestmap}. Then after some long enough period of time you can go through your server's deleted documents and if they are old enough, you can start purging batches of them.

Another approach I have seen often is to use a time-based database, say one per month or year. Then the client code would be need to transition between them (know which one is current), and you can either delete or archive the previous db instance.

Some of the auto-compactor feature have been improved since 3.1.1. So I'd also advise you to upgrade. There also were some security fixes we have made since.