configured HISTORY_RETENTION_COUNT but disk size is growing

[saravadeanil]

What version are you using?

Horizon: 2.28.3-fa5404bc79ba72ec9311e063e1a27cd5712941be
Core: 20.2.0
go: go1.20.14

What did you do?

I am running a Stellar Horizon node. My goal is to ensure that the old historical ledgers are pruned to avoid disk growth and consumption. I have configured the environment variable HISTORY_RETENTION_COUNT=518400 (for a 30-day retention period) Reference.

What did you expect to see?

Only latest 518400 blocks are available on node, and disk consumption should stop growing.

What did you see instead?

Based on my investigation, it seems the db reap is working after setting the HISTORY_RETENTION_COUNT=518400. I checked and the value inside the history_ledgers table was dropped to 518413 (nearby value of HISTORY_RETENTION_COUNT). I monitored the horizon database after 1h frequency and the db reap functionality seems to be affecting the history_* tables.

Additionally, the history_elder_ledger ledger became unavailable after a while.

  "ingest_latest_ledger": 50783570,
  "history_latest_ledger": 50783570,
  "history_latest_ledger_closed_at": "2024-03-14T07:02:21Z",
  "history_elder_ledger": 50265033,

{
  "type": "https://stellar.org/horizon-errors/before_history",
  "title": "Data Requested Is Before Recorded History",
  "status": 410,
  "detail": "This horizon instance is configured to only track a portion of the stellar network's latest history. This request is asking for results prior to the recorded history known to this horizon instance."
}

However, I also did a manual db reap but still the disk size is growing.

stellar-horizon db reap

2024/03/14 09:18:38 Checking DB migrations...
2024/03/14 09:18:38 Preparing captive core...
INFO[2024-03-14T09:18:38.942Z] Initializing horizon...                       pid=12040
INFO[2024-03-14T09:18:38.943Z] Initializing database...                      pid=12040
INFO[2024-03-14T09:18:38.943Z] Establishing database session for history     pid=12040
INFO[2024-03-14T09:18:38.947Z] Establishing database session for ingest      pid=12040
INFO[2024-03-14T09:18:42.411Z] reaper: clearing                              end_ledger=50266521 pid=12040 start_ledger=50266233
INFO[2024-03-14T09:18:46.766Z] reaper succeeded                              new_elder=50266522 pid=12040

[sokiaoba]

+1

[mollykarcher]

@saravadeanil can you provide more detail on how you know the disk size is growing, and how fast it's growing?

Note that after the reaping commands are run by Horizon, postgres will need to autovacuum the relevant tables to remove the tombstones of the deleted entries and for you to fully realize the effects of the reaping. Also note that, despite reaping, the ledger density has always trended upwards over time, and network activity can vary quite a bit. So it's definitely possible/likely that you could see small increases over time, but it shouldn't be that significant.

[mollykarcher]

@saravadeanil please reopen with the requested info if your issue has persisted.