♻️ Refactor SQL chunk processing to reduce memory errors #699
+292
−52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: a387527 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
CI Feedback 🧐(Feedback updated until commit 7e3a65d)A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
hoshinotsuyoshi
had a problem deploying
to
preview - @liam-hq/erd-sample
GitHub Actions
Failure
hoshinotsuyoshi
had a problem deploying
to
preview - erd-web
GitHub Actions
Failure
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
c749b2a to
0badc26
Compare
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
0badc26 to
f08c5ee
Compare
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
f08c5ee to
d714b0f
Compare
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
e13604a to
de1bc5d
Compare
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
hoshinotsuyoshi
changed the title
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
de1bc5d to
7e3a65d
Compare
hoshinotsuyoshi
changed the title
This refactor increases the likelihood of processing larger `.sql` files without encountering memory errors. - Updated `processor` to improve error handling and prevent unnecessary semicolon-related logic. - Modified `processSQLInChunks` to track read offsets and adjust chunk sizes dynamically. - Improved test cases to ensure SQL chunks are processed correctly, even when split mid-statement.
hoshinotsuyoshi
force-pushed
the
avoid-semicolon-logic
branch
from
7e3a65d to
fa395d0
Compare
- Added test cases to verify correct handling of `readOffset` when processing SQL chunks with incomplete statements. - Added test cases for `errorOffset` handling, including shrinking and expanding modes when parsing SQL statements larger than the chunk size.
- Introduced a new test case to verify that SQL statements exceeding 500 lines (surpassing CHUNK_SIZE) are parsed correctly without errors. - Added a `normal` test case in `createParserTestCases` for basic column validation. - Updated `index.test.ts` to include a test for handling long SQL statements.
hoshinotsuyoshi
changed the title
MH4GF
approved these changes
Feb 20, 2025
| 'CREATE TABLE t1 (\n c1 int,\n c2 int,\n c3 int,\n c4 int\n);', | ||
| [null, null, []], | ||
| ], | ||
| ] |
There was a problem hiding this comment.
Test case is easy to understand 👍🏻
MH4GF
reviewed
Feb 20, 2025
Co-authored-by: Hirotaka Miyagi <31152321+MH4GF@users.noreply.github.com>
|
Thanks, I'll merge now |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User description
closes #410
closes #688
♻️ Refactor SQL chunk processing to reduce memory errors
This refactor increases the likelihood of processing larger
.sqlfiles without encountering memory errors.processorto improve error handling and prevent unnecessary semicolon-related logic.processSQLInChunksto track read offsets and adjust chunk sizes dynamically.Cons:
The processing time for
mastodon db/structure.sql liam-hq's pg_dumpincreases 😭( 0.84s -> 4.54s ). (It has not only DDLs, but also a little DMLs.) Reducing the chunk size has resulted in more frequent calls to theparsefunction. This might lead to more.sqlfiles timing out in erd-web. However, since it reduces the likelihood of errors, I still want to proceed with this change.Related Issue
Testing
result:
main branch ( on 0cda4f4 ) :
this PR branch ( on 7e3a65d ) :
Other Information
PR Type
Enhancement, Tests
Description
Refactored SQL chunk processing to reduce memory errors.
Enhanced test cases for SQL chunk processing.
Added utility function to determine line numbers for error handling.
Changes walkthrough 📝
index.ts
Refactor `processor` for better SQL chunk handlingfrontend/packages/db-structure/src/parser/sql/postgresql/index.ts
processorto handle incomplete SQL statements.processSQLInChunks.ts
Refactor `processSQLInChunks` for dynamic chunk handlingfrontend/packages/db-structure/src/parser/sql/postgresql/processSQLInChunks.ts
processSQLInChunks.test.ts
Enhance tests for SQL chunk processingfrontend/packages/db-structure/src/parser/sql/postgresql/processSQLInChunks.test.ts
cyan-crabs-kiss.md
Add changeset for SQL chunk processing refactor.changeset/cyan-crabs-kiss.md
Footnotes
supabase.sql.txt ↩ ↩2
mastodon-structure.sql.txt ↩ ↩2