Feat: Use duckdb as backend instead of sqlite #59
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Resolves issue #58 |
|
It looks like duckdb isn't consistent about the ordering even when it's called the same way, I think that's going to cause bugs in peoples code. I'd suggest that we order the results before returning them. I also want to discuss scope of this a bit, but will continue that in #58 |
ivirshup
approved these changes
Apr 3, 2024
| @@ -295,7 +308,7 @@ def _build_query( | |||
| else: | |||
| query = self.db.table(table) | |||
| # add filter | |||
| query = query.filter(filter.convert()).select(cols) | |||
| query = query.filter(filter.convert()).select(cols).order_by(cols) | |||
There was a problem hiding this comment.
@nvictus does this ordering look reasonable? Or is there a specific thing we should be ordering by?
|
I'm going to merge this an open an issue to discuss ordering |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a
backend(default: sqlite, optional: duckdb). Also it forces ordering by columns requested in order.Benchmarking:
Running tests using DuckDB backend
pytest 40.40s user 7.76s system 196% cpu 24.544 total
Running tests using Sqlite3
pytest 39.02s user 10.77s system 97% cpu 51.043 total
DuckDB is twice as fast because it used two cores, but about the same number of cpu cycles.