Make backfill behavior more explicit #700

dermanyang · 2025-01-29T18:29:57Z

Purpose

Make restart behavior more explicit through the introduction of TestingConfig, BootstrapConfig and ProcessorMode on IndexerProcessorConfig. The starting_version on transaction_stream_config should no longer be used, instead there are new fields initial_starting_version on the backfill + bootstrap configs, and override_starting_version on testing config.

Behavior

The current live behavior of taking the higher value of transaction_stream_config.starting_version is moved to bootstrap_config.initial_starting_version. This is the default behavior of the processor if ProcessorMode is not specified. Just as before, if the respective starting version is not provided, the processor will begin from genesis.

BackfillConfig.intial_starting_version will be respected when any of the following are true:

there is no checkpoint in backfill_processor_status
the last backfill has been complete (last_successful_version >= backfill_config.ending_version)
initial_starting_version > last_successful_version.
backfill_config.overwrite_checkpoint is set to true
Otherwise, the processor will select last_successful_version.

Technical Bits

Using pattern matching on IndexerProcessorConfig.processor_mode for different logic in starting_version.rs. All new fields are Optional using Serde to default processor_mode to default when not provided. Serde will throw if mode == testing or mode == backfill and the respective sub-config isn't provided.

Make start and end version on backfill_config required. Remove backfill_alias field in favor of backfill_id. This will be appended onto the processor type as a PK in the backfill_processor_status table.

Testing

Added tests for new behavior

Mode not set, no checkpoint

Mode not set, checkpoint higher than bootstrap start ver (ran after the above run)

mode = backfill with no checkpoint

mode = backfill, checkpoint higher than `backfill_config.intial_starting_version`

mode = testing, checkpoint higher than `testing_config.override_starting_version`. latter is still picked

rust/sdk-processor/src/utils/starting_version.rs

rust/sdk-processor/src/config/indexer_processor_config.rs

rust/sdk-processor/src/db/common/models/backfill_processor_status.rs

rust/sdk-processor/src/utils/starting_version.rs

yuunlimm · 2025-01-30T18:07:58Z

rust/sdk-processor/src/utils/starting_version.rs

+}
+
+fn log_ascii_warning(version: u64) {
+    println!(r#"


this was fun haha

yuunlimm · 2025-01-30T18:08:45Z

rust/sdk-processor/src/utils/starting_version.rs

-/// If it is a regular processor, this will return the higher of the checkpointed version,
-/// or `staring_version` from the config, or 0 if not set.
+/// If it is a regular (`mode == default`) processor, this will return the higher of the (checkpointed version,
+/// `initial_starting_version` || 0) from the bootstrap config.


q: what purpose does intial_starting_version serve?

it's the same purpose as it is today, i just prepended it with initial to make it explicit that it'll only be respected on initial startup and get_starting_version() will take the larger of the config's starting version and the checkpointed version

rust/sdk-processor/src/config/indexer_processor_config.rs

rust/sdk-processor/src/utils/starting_version.rs

rust/processor/parser.yaml

bowenyang007

Where is the updated readme?

had some questions. once confirmed should be g2g. thanks for addressing this!

rust/sdk-processor/src/config/indexer_processor_config.rs

rust/sdk-processor/src/utils/starting_version.rs

rtso

left a few comments but devex lgtm! would you mind also updating the README.md?

rust/sdk-processor/README.md

bowenyang007 · 2025-02-04T06:28:40Z

rust/sdk-processor/README.md

+    - `backfill_id`: appended to `processor_type` for a unique backfill identifier
+    - `initial_starting_version`: processor starts here unless there is a greater checkpointed version
+    - `ending_version`: ending version of the backfill
+    - `overwrite_checkpoint`: overwrite checkpoints if it exists, restarting the backfill from `initial_starting_version`.


wouldn't this lead to crashlooping behavior?

we either have crash looping when someone sets overwrite_checkpoint = true, or we don't have a way to in progress backfills. I think if someone needs to restart an in-progress BF, they're able to pay attention and fix the wrong behavior.

What do you think?

yuunlimm · 2025-02-04T19:27:55Z

rust/sdk-processor/src/utils/starting_version.rs

 /// the checkpointed version + 1.
 ///
 /// If this is a backfill processor and there is not an in-progress backfill (i.e., no checkpoint or
-/// backfill status is COMPLETE), this will return `starting_version` from the config, or 0 if not set.
+/// backfill status is COMPLETE), this will return `intial_starting_version` from the backfill config, or 0 if not set.


return intial_starting_version from the backfill config, or 0 if not set. isn't intial_starting_version field required?

rust/sdk-processor/src/utils/starting_version.rs

yuunlimm

left minor questions, overall lgtm!

rtso · 2025-02-06T16:48:12Z

rust/sdk-processor/README.md

+        backfill_config:
+            backfill_id: "123"
+            initial_starting_version: 42
+            ending_version: 100042
+            overwrite_checkpoint: false
+        testing_config:
+            override_starting_version: 111111
+            ending_version: 222222
+        bootstrap_config:
+            initial_starting_version: 123456789


are only 1 of these configs set at a time? if so, might be confusing to include all three of these in the template. could we also add comments here for when to use each mode?

updated to not include these in the example config but mention them in the explanation section:

backfill_config (optional)

backfill_id: appended to processor_type for a unique backfill identifier

initial_starting_version: processor starts here unless there is a greater checkpointed version

ending_version: ending version of the backfill

overwrite_checkpoint: overwrite checkpoints if it exists, restarting the backfill from initial_starting_version.

testing_config (optional)

override_starting_version: starting version of the testing. always starts from this version

ending_version: ending version of the testing

bootstrap_config (optional) used for regular, non-backfill, processors

initial_starting_version: processor starts here unless there is a greater checkpointed version.

mode: (optional) default, testing or backfill. Set to default if no mode specified. If backfill/testing/bootstrap
configs are not specified, processor will start from 0 or the last successfully processed version.

codecov · 2025-02-08T06:42:19Z

Codecov Report

Attention: Patch coverage is 59.90338% with 249 lines in your changes missing coverage. Please review.

Project coverage is 50.4%. Comparing base (e943d0f) to head (c6c5ad8).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...k-processor/src/config/indexer_processor_config.rs	0.0%	31 Missing ⚠️
...ocessor/src/steps/common/processor_status_saver.rs	26.9%	19 Missing ⚠️
...ocessors/parquet_account_transactions_processor.rs	0.0%	12 Missing ⚠️
...or/src/parquet_processors/parquet_ans_processor.rs	0.0%	12 Missing ⚠️
...rc/parquet_processors/parquet_default_processor.rs	0.0%	12 Missing ⚠️
...src/parquet_processors/parquet_events_processor.rs	0.0%	12 Missing ⚠️
...uet_processors/parquet_fungible_asset_processor.rs	0.0%	12 Missing ⚠️
...rc/parquet_processors/parquet_objects_processor.rs	0.0%	12 Missing ⚠️
.../src/parquet_processors/parquet_stake_processor.rs	0.0%	12 Missing ⚠️
...c/parquet_processors/parquet_token_v2_processor.rs	0.0%	12 Missing ⚠️
... and 14 more

Additional details and impacted files

@@          Coverage Diff           @@
##            main    #700    +/-   ##
======================================
  Coverage   50.3%   50.4%            
======================================
  Files        253     253            
  Lines      28643   29128   +485     
======================================
+ Hits       14432   14699   +267     
- Misses     14211   14429   +218

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dermanyang added 7 commits January 27, 2025 15:54

Add new configs and update starting_version.rs logic

39fdfa3

make tests pass

574fec1

fix tests

c5cebf0

remove accidentally included files

c67f07d

add processor mode to processor config

1cbb15f

Fix tests, add ascii warning if backfill restart starts at checkpoint

0333e25

Fix ascii message by using println instead of log.warn

ad8e4d1

dermanyang changed the title ~~Sdy/improve backfill~~ Make backfill behavior more explicit Jan 30, 2025

undo change to old parser.yml

cd3b4ce

dermanyang commented Jan 30, 2025

View reviewed changes

rust/sdk-processor/src/utils/starting_version.rs Show resolved Hide resolved

dermanyang marked this pull request as ready for review January 30, 2025 02:13

Update documentation

9a5052e

dermanyang requested review from yuunlimm and rtso January 30, 2025 02:14

yuunlimm reviewed Jan 30, 2025

View reviewed changes

dermanyang added 2 commits January 30, 2025 22:18

add docstrings to different sub configs

23b1897

Merge branch 'main' into sdy/improve_backfill

f6da19e

bowenyang007 reviewed Jan 31, 2025

View reviewed changes

rust/processor/parser.yaml Outdated Show resolved Hide resolved

bowenyang007 reviewed Jan 31, 2025

View reviewed changes

rtso reviewed Feb 3, 2025

View reviewed changes

rust/sdk-processor/src/config/indexer_processor_config.rs Show resolved Hide resolved

rtso reviewed Feb 3, 2025

View reviewed changes

rust/sdk-processor/src/utils/starting_version.rs Show resolved Hide resolved

rtso reviewed Feb 3, 2025

View reviewed changes

dermanyang added 3 commits February 3, 2025 22:00

overwrite_checkpoint -> overwrite_in_progress_checkpoint

0c46c2e

separate case for COMPLETE and overwrit_checkpoint=false

1ffae8a

update readme

f53f867

bowenyang007 reviewed Feb 4, 2025

View reviewed changes

rust/sdk-processor/README.md Show resolved Hide resolved

bowenyang007 reviewed Feb 4, 2025

View reviewed changes

yuunlimm reviewed Feb 4, 2025

View reviewed changes

yuunlimm approved these changes Feb 4, 2025

View reviewed changes

rtso reviewed Feb 6, 2025

View reviewed changes

dermanyang added 7 commits February 6, 2025 20:51

update readme and add ending version to txn stream

1911097

remove newline

ca61095

format

895aced

update tests to pass starting_ver to processor config

485d805

Merge branch 'main' into sdy/improve_backfill

6b94d0e

add ending_verison for sdk processor diff tests

3151105

lint

c6c5ad8

dermanyang merged commit bd642dd into main Feb 8, 2025
11 checks passed

dermanyang deleted the sdy/improve_backfill branch February 8, 2025 06:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make backfill behavior more explicit #700

Make backfill behavior more explicit #700

dermanyang commented Jan 29, 2025 •

edited

Loading

yuunlimm Jan 30, 2025

dermanyang Jan 31, 2025

yuunlimm Jan 30, 2025

dermanyang Jan 31, 2025

bowenyang007 left a comment

rtso left a comment

bowenyang007 Feb 4, 2025

dermanyang Feb 4, 2025

yuunlimm Feb 4, 2025

yuunlimm left a comment

rtso Feb 6, 2025

dermanyang Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 8, 2025 •

edited

Loading

Make backfill behavior more explicit #700

Make backfill behavior more explicit #700

Conversation

dermanyang commented Jan 29, 2025 • edited Loading

Purpose

Behavior

Technical Bits

Testing

Mode not set, no checkpoint

Mode not set, checkpoint higher than bootstrap start ver (ran after the above run)

mode = backfill with no checkpoint

mode = backfill, checkpoint higher than backfill_config.intial_starting_version

mode = testing, checkpoint higher than testing_config.override_starting_version. latter is still picked

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenyang007 left a comment

Choose a reason for hiding this comment

rtso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuunlimm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dermanyang Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Feb 8, 2025 • edited Loading

Codecov Report

dermanyang commented Jan 29, 2025 •

edited

Loading

mode = backfill, checkpoint higher than `backfill_config.intial_starting_version`

mode = testing, checkpoint higher than `testing_config.override_starting_version`. latter is still picked

dermanyang Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 8, 2025 •

edited

Loading