Add predownload functionality to Pinot

[lnbest0707-uber]

feature
performance
#14592

Add a pre-download feature to enable "graceful" node replacement on Pinot. With this feature, during node replacement, admins would replace the old node (OD) to new node (NN) with the same instance id. Instead of bringing down the ON before starting up NN (which is required because there cannot be 2 nodes taking same helix id at the same time), admins could:

1. Start NN in "pre-download" mode by adding one more parameter to StartServerCommand, like:

PropertiesConfiguration properties = CommonsConfigurationUtils.fromPath(<config_path>);
PredownloadScheduler predownloadScheduler = new PredownloadScheduler(properties);
predownloadScheduler.start();

2. Waiting for NN "pre-download" complete with one of following conditions:
   - pre-download fully succeed
   - pre-download partially succeed but have retried enough times
   - pre-download failed in non-retriable mode
   - already waited for a max time period
3. Bring down the ON
4. Start NN in normal mode

With a successful pre-download, the Pinot node replacement could perform in the same way as a node restart. We could observe the downtime (referring to helix pending message values decrease to 0) reduce significantly.
(1h -> 10min in the attached case. Sometimes, it could be fully transparent if node load is not high.)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 82.91747% with 89 lines in your changes missing coverage. Please review.

Project coverage is 64.01%. Comparing base (59551e4) to head (83afb4c).
Report is 1547 commits behind head on master.

Files with missing lines	Patch %	Lines
...pinot/server/predownload/PredownloadScheduler.java	82.88%	30 Missing and 8 partials ⚠️
...pache/pinot/server/predownload/StatusRecorder.java	64.28%	31 Missing and 4 partials ⚠️
...g/apache/pinot/server/predownload/SegmentInfo.java	90.32%	2 Missing and 4 partials ⚠️
...e/pinot/server/predownload/PredownloadMetrics.java	73.33%	4 Missing ⚠️
...pinot/server/predownload/PredownloadException.java	50.00%	2 Missing ⚠️
.../org/apache/pinot/server/predownload/ZKClient.java	97.40%	0 Missing and 2 partials ⚠️
...t/local/segment/store/SegmentLocalFSDirectory.java	0.00%	0 Missing and 1 partial ⚠️
...org/apache/pinot/server/predownload/TableInfo.java	96.87%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14686      +/-   ##
============================================
+ Coverage     61.75%   64.01%   +2.26%     
- Complexity      207     1608    +1401     
============================================
  Files          2436     2711     +275     
  Lines        133233   151443   +18210     
  Branches      20636    23353    +2717     
============================================
+ Hits          82274    96952   +14678     
- Misses        44911    47262    +2351     
- Partials       6048     7229    +1181     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.96% <82.14%> (+2.25%)	⬆️
java-21	63.86% <82.91%> (+2.23%)	⬆️
skip-bytebuffers-false	64.00% <82.91%> (+2.26%)	⬆️
skip-bytebuffers-true	63.81% <82.91%> (+36.09%)	⬆️
temurin	64.01% <82.91%> (+2.26%)	⬆️
unittests	64.01% <82.91%> (+2.26%)	⬆️
unittests1	56.33% <90.90%> (+9.44%)	⬆️
unittests2	34.39% <82.91%> (+6.66%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ankitsultana]

If the goal is to use this predownload feature in large scale deployments then I don't think putting this in pinot-tools is the right approach. Ideally such deployments should not have a dependency on pinot-tools. pinot-tools mainly contains administrative and utility functions, and brings in a ton of dependencies.

[lnbest0707-uber]

Thanks for the suggestions. Do you have any recommended destinations? How about putting directly into pinot-server pkg?