tbv2: calculate total memory footprint #448
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JakeHillion
force-pushed
the
pr448
branch
3 times, most recently
from
8d1b505 to
9a0c672
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #448 +/- ##
==========================================
+ Coverage 44.63% 44.95% +0.31%
==========================================
Files 121 123 +2
Lines 11905 11975 +70
Branches 1947 1953 +6
==========================================
+ Hits 5314 5383 +69
+ Misses 5738 5737 -1
- Partials 853 855 +2 ☔ View full report in Codecov by Sentry. |
This was referenced Jan 3, 2024
Add the option to calculate total size (inclusive size) by wrapping the existing iterator. This change provides a new iterator, `SizedIterator`, which wraps an existing iterator and adds a new field `size` to the output element. This is achieved with a two pass algorithm on the existing iterator: 1. Gather metadata for each element. This includes the total size up until that element and the range of elements that should be included in the size. 2. Return the result from the underlying iterator with the additional field. This algorithm is `O(N)` time on the number of elements in the iterator and `O(N)` time, storing 16 bytes per element. This isn't super expensive but is a lot more than the current algorithm which requires close to constant space. Because of this I've implemented it as a wrapper on the iterator rather than on by default, though it is now on in every one of our integration test cases. Test plan: - Added to the integration tests for full coverage.
arsarwade
approved these changes
Jan 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tbv2: calculate total memory footprint
Add the option to calculate total size (inclusive size) by wrapping the
existing iterator. This change provides a new iterator,
SizedIterator, whichwraps an existing iterator and adds a new field
sizeto the output element.This is achieved with a two pass algorithm on the existing iterator:
element and the range of elements that should be included in the size.
field.
This algorithm is
O(N)time on the number of elements in the iterator andO(N)time, storing 16 bytes per element. This isn't super expensive but is alot more than the current algorithm which requires close to constant space.
Because of this I've implemented it as a wrapper on the iterator rather than on
by default, though it is now on in every one of our integration test cases.
Test plan:
Stack created with Sapling. Best reviewed with ReviewStack.