Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update load_from_private_log of duplication task priority from LOW to COMMON #2183

Open
ninsmiracle opened this issue Jan 20, 2025 · 0 comments
Labels
type/enhancement Indicates new feature requests

Comments

@ninsmiracle
Copy link
Contributor

ninsmiracle commented Jan 20, 2025

Feature Request

Is your feature request related to a problem? Please describe:
When I test the performance bottleneck of duplication, I feel strange about the task priority of load_from_private_log stage.
load_from_private_log is the main time-consuming stage that generates IO, and is handled by the THREAD_POOL_REPLICATION_LONG thread. The priority of the dup-related tasks is also defined as LOW.
We can see all the RPC heandled by THREAD_POOL_REPLICATION_LONG:

// THREAD_POOL_REPLICATION_LONG
#define CURRENT_THREAD_POOL THREAD_POOL_REPLICATION_LONG
MAKE_EVENT_CODE(LPC_LEARN_REMOTE_DELTA_FILES, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE_AIO(LPC_REPLICATION_COPY_REMOTE_FILES, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_GARBAGE_COLLECT_LOGS_AND_REPLICAS, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_OPEN_REPLICA, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_CLOSE_REPLICA, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_CHECKPOINT_REPLICA, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_CATCHUP_WITH_PRIVATE_LOGS, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_DISK_STAT, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_BACKGROUND_COLD_BACKUP, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_PARTITION_SPLIT_ASYNC_LEARN, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_REPLICATION_LONG_LOW, TASK_PRIORITY_LOW)
MAKE_EVENT_CODE(LPC_REPLICATION_LONG_COMMON, TASK_PRIORITY_COMMON)
MAKE_EVENT_CODE(LPC_REPLICATION_LONG_HIGH, TASK_PRIORITY_HIGH)
#undef CURRENT_THREAD_POOL

In fact, the Pegasus users always requires real-time dup. If the delay of the dup process of a piece of data is too high, when the master cluster down, the backup cluster will be unable to completely replace all the data of the master cluster to provide services.
And I do not think LPC_DISK_STAT, LPC_GARBAGE_COLLECT_LOGS_AND_REPLICAS task is importanter than duplicate.

Describe the feature you'd like:
I'd like to raise the priority of load_from_private_log when doing duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

1 participant