migrate experimental-memory-mlock flag to memory-mlock #19282

jmao-dd · 2025-01-27T14:04:03Z

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

This fixes #19061

k8s-ci-robot · 2025-01-27T14:04:13Z

Hi @jmao-dd. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-01-27T14:22:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.87%. Comparing base (dd60559) to head (0795f6b).
Report is 37 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
server/config/config.go	`80.23% <ø> (ø)`
server/embed/config.go	`79.72% <100.00%> (+0.06%)`	⬆️
server/embed/etcd.go	`76.79% <100.00%> (+0.30%)`	⬆️
server/etcdmain/config.go	`73.49% <100.00%> (+0.32%)`	⬆️
server/storage/backend.go	`80.00% <100.00%> (+7.27%)`	⬆️

... and 24 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19282      +/-   ##
==========================================
+ Coverage   68.84%   68.87%   +0.03%     
==========================================
  Files         420      420              
  Lines       35693    35705      +12     
==========================================
+ Hits        24573    24592      +19     
+ Misses       9698     9693       -5     
+ Partials     1422     1420       -2

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dd60559...0795f6b. Read the comment docs.

ahrtr · 2025-01-27T20:58:31Z

/ok-to-test

jmao-dd · 2025-01-28T12:39:46Z

server/etcdmain/help.go

@@ -75,6 +75,8 @@ Member:
    Maximum number of snapshot files to retain (0 is unlimited). Deprecated in v3.6 and will be decommissioned in v3.7.
  --max-wals '` + strconv.Itoa(embed.DefaultMaxWALs) + `'
    Maximum number of wal files to retain (0 is unlimited).
+  --memory-mlock
+    Enable to enforce etcd pages (in particular bbolt) to stay in RAM.


I am not sure if Member is the right category to add --memory-mlock. Please let me know if it isn't.

jmao-dd · 2025-01-29T03:34:33Z

/retest

ahrtr

LGTM

Thanks

ahrtr · 2025-01-31T09:44:25Z

cc @fuweid @ivanvc @jmhbnz @serathius @siyuanfoundation

siyuanfoundation · 2025-01-31T16:59:48Z

server/embed/config.go

 	fs.BoolVar(&cfg.ExperimentalMemoryMlock, "experimental-memory-mlock", cfg.ExperimentalMemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")
+	fs.BoolVar(&cfg.MemoryMlock, "memory-mlock", cfg.MemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")


can you add the new flag to experimentalNonBoolFlagMigrationMap and rename the map to experimentalFlagMigrationMap?

This is actually a boolean flag. Based on the name I think experimentalNonBoolFlagMigrationMap is supposed to track non-boolean flags and alert users that they specified contradicting values. For boolean flags if either flag is set it is treated as set, otherwise as unset?

the user might specify conflicting values like --experimental-memory-mlock=true --memory-mlock=false, which could be rare but should still be disallowed.
experimentalNonBoolFlagMigrationMap just makes sure the two flags are not both explicitly set. You can rename it to experimentalFlagMigrationMap

I see. I can change to forbid both being set.

siyuanfoundation · 2025-01-31T17:01:34Z

server/etcdmain/config_test.go

+
+			cfgFromCmdLine, errFromCmdLine, cfgFromFile, errFromFile := generateCfgsFromFileAndCmdLine(t, yc, cmdLineArgs)
+
+			if errFromCmdLine != nil || errFromFile != nil {


can you add a test case to show you can not set both flags at the same time?

Similar as above, I think we want to allow both boolean flags to be set?

Signed-off-by: Jiayin Mao <jiayin.mao@datadoghq.com>

jmao-dd · 2025-02-02T14:08:16Z

/retest

fuweid

LGTM

siyuanfoundation · 2025-02-03T17:54:12Z

server/embed/config.go

 	fs.BoolVar(&cfg.ExperimentalMemoryMlock, "experimental-memory-mlock", cfg.ExperimentalMemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")
+	fs.BoolVar(&cfg.MemoryMlock, "memory-mlock", cfg.MemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")


k8s-ci-robot · 2025-02-03T17:54:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, fuweid, jmao-dd, siyuanfoundation

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

siyuanfoundation · 2025-02-03T17:54:40Z

/lgtm

Thank you @jmao-dd !

k8s-ci-robot added the needs-ok-to-test label Jan 27, 2025

k8s-ci-robot added the size/M label Jan 27, 2025

jmao-dd mentioned this pull request Jan 27, 2025

migrate to use memory-mlock flag #19277

Closed

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Jan 27, 2025

jmao-dd commented Jan 28, 2025

View reviewed changes

jmao-dd force-pushed the jmao/19061-migrate-mlock3 branch from 544b275 to e658c89 Compare January 29, 2025 02:20

jmao-dd force-pushed the jmao/19061-migrate-mlock3 branch from e658c89 to 52a91e2 Compare January 29, 2025 14:20

k8s-ci-robot added the needs-rebase label Jan 29, 2025

jmao-dd force-pushed the jmao/19061-migrate-mlock3 branch from 52a91e2 to 8d5be6e Compare January 30, 2025 01:00

k8s-ci-robot removed the needs-rebase label Jan 30, 2025

ahrtr approved these changes Jan 30, 2025

View reviewed changes

k8s-ci-robot added the approved label Jan 30, 2025

ahrtr mentioned this pull request Jan 30, 2025

migrate experimental-peer-skip-client-san-verification flag to feature gate #19225

Open

siyuanfoundation reviewed Jan 31, 2025

View reviewed changes

k8s-ci-robot added size/L and removed size/M labels Feb 1, 2025

jmao-dd requested a review from siyuanfoundation February 1, 2025 22:57

migrate experimental-memory-mlock flag to memory-mlock

0795f6b

Signed-off-by: Jiayin Mao <jiayin.mao@datadoghq.com>

jmao-dd force-pushed the jmao/19061-migrate-mlock3 branch from f541d78 to 0795f6b Compare February 1, 2025 22:58

fuweid approved these changes Feb 3, 2025

View reviewed changes

siyuanfoundation approved these changes Feb 3, 2025

View reviewed changes

k8s-ci-robot assigned siyuanfoundation Feb 3, 2025

k8s-ci-robot added the lgtm label Feb 3, 2025

ahrtr merged commit 3cc3daf into etcd-io:main Feb 3, 2025
36 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate experimental-memory-mlock flag to memory-mlock #19282

migrate experimental-memory-mlock flag to memory-mlock #19282

jmao-dd commented Jan 27, 2025

k8s-ci-robot commented Jan 27, 2025

codecov bot commented Jan 27, 2025 •

edited

Loading

ahrtr commented Jan 27, 2025

jmao-dd Jan 28, 2025

jmao-dd commented Jan 29, 2025

ahrtr left a comment

ahrtr commented Jan 31, 2025

siyuanfoundation Jan 31, 2025

jmao-dd Jan 31, 2025

siyuanfoundation Feb 1, 2025

jmao-dd Feb 1, 2025

jmao-dd Feb 1, 2025

siyuanfoundation Feb 3, 2025

siyuanfoundation Jan 31, 2025

jmao-dd Jan 31, 2025

jmao-dd Feb 1, 2025

jmao-dd commented Feb 2, 2025

fuweid left a comment

siyuanfoundation Feb 3, 2025

k8s-ci-robot commented Feb 3, 2025

siyuanfoundation commented Feb 3, 2025

		fs.BoolVar(&cfg.ExperimentalMemoryMlock, "experimental-memory-mlock", cfg.ExperimentalMemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")
		fs.BoolVar(&cfg.MemoryMlock, "memory-mlock", cfg.MemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")


		cfgFromCmdLine, errFromCmdLine, cfgFromFile, errFromFile := generateCfgsFromFileAndCmdLine(t, yc, cmdLineArgs)

		if errFromCmdLine != nil \|\| errFromFile != nil {

migrate experimental-memory-mlock flag to memory-mlock #19282

migrate experimental-memory-mlock flag to memory-mlock #19282

Conversation

jmao-dd commented Jan 27, 2025

k8s-ci-robot commented Jan 27, 2025

codecov bot commented Jan 27, 2025 • edited Loading

Codecov Report

ahrtr commented Jan 27, 2025

Choose a reason for hiding this comment

jmao-dd commented Jan 29, 2025

ahrtr left a comment

Choose a reason for hiding this comment

ahrtr commented Jan 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmao-dd commented Feb 2, 2025

fuweid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 3, 2025

siyuanfoundation commented Feb 3, 2025

codecov bot commented Jan 27, 2025 •

edited

Loading