Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Size tracking component #423

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Conversation

jfantinhardesty
Copy link
Contributor

What type of Pull Request is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Describe your changes in brief

This adds a new size tracker component which tracks the size of the mounted directory to prevent calls to the cloud to get the size of the mount. This in particular is useful when mounting subdirectories as there is no call in S3 to retrieve the size of the storage. This component goes in between the file_cache and attr_cache to track write information.

Checklist

  • Tested locally
  • Added new dependencies
  • Updated documentation
  • Added tests

Related Issues

  • Related Issue #
  • Closes #

@jfantinhardesty jfantinhardesty marked this pull request as ready for review January 24, 2025 18:21
@jfantinhardesty jfantinhardesty marked this pull request as draft January 24, 2025 18:27
@jfantinhardesty jfantinhardesty marked this pull request as ready for review January 24, 2025 20:48
Copy link
Contributor

@foodprocessor foodprocessor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's quite the testing infrastructure! Good work!

component/size_tracker/size_tracker.go Outdated Show resolved Hide resolved
}

var newSize int64
attr, getAttrErr2 := st.NextComponent().GetAttr(internal.GetAttrOptions{Name: options.Handle.Path})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the WriteFileOptions to figure out the new size without calling GetAttr again?
I think the new size would be whichever is larger: the old size or options.Offset + len(options.Data), right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good idea to avoid an extra call. I will make that change.

jfantinhardesty and others added 4 commits January 27, 2025 08:23
Co-authored-by: Michael Habinsky <foodprocessor@users.noreply.github.com>
Signed-off-by: James Fantin-Hardesty <24646452+jfantinhardesty@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants