Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: csv support for conversiont to json #48

Merged
merged 46 commits into from
Dec 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
a18d2cf
move json loaders to loaders module
shoshta73 Nov 23, 2024
59e23a8
implement loader class
shoshta73 Nov 23, 2024
fb76b5f
implement load_as support for csv
shoshta73 Nov 23, 2024
ce39df9
cleanup load_as function
shoshta73 Nov 23, 2024
ecfc151
rename loaders -> loader
shoshta73 Nov 27, 2024
a1f9a6e
remove usages of load_json and load_jsonc
shoshta73 Nov 27, 2024
e7af520
add test for loading csv
shoshta73 Nov 28, 2024
3ea755e
fix file closing issue
shoshta73 Nov 28, 2024
8b8a10c
implement simple csv to json conversion
shoshta73 Nov 28, 2024
8e7d0f5
implement loading data from filepath in converter
shoshta73 Nov 28, 2024
cbb2641
implement loading str_data in converter class
shoshta73 Nov 28, 2024
bf687a1
fix variable naming
shoshta73 Nov 28, 2024
ce4334f
move main.py to app/cli.py
shoshta73 Nov 29, 2024
41bc158
add pyi file for converter
shoshta73 Nov 29, 2024
1cbe2d9
add example app that reads stdin
shoshta73 Nov 29, 2024
42ee6d0
add test for converter class
shoshta73 Nov 29, 2024
b491eba
add convertable method on converter class
shoshta73 Nov 29, 2024
9400e62
add ability to convert json back to csv
shoshta73 Nov 29, 2024
6298646
add python 3.13 to the ci
shoshta73 Nov 30, 2024
1aa80ec
finish up converter app
shoshta73 Nov 30, 2024
c98cf1a
add converter app to the pyproject.toml
shoshta73 Nov 30, 2024
67363c9
add can_use_match helper function
shoshta73 Nov 30, 2024
fbd74b6
use match statement in logger
shoshta73 Dec 1, 2024
74c7b93
make local imports not depend on package
shoshta73 Dec 1, 2024
b93aaf1
switch order of calling in _logger_print
shoshta73 Dec 1, 2024
fe4abfa
add optional 3.10 support
shoshta73 Dec 1, 2024
39c3125
make tests not depend on installed module
shoshta73 Dec 1, 2024
31c6880
remove not needed "pip install ." commands from ci
shoshta73 Dec 1, 2024
f138e63
multithread converter app
shoshta73 Dec 1, 2024
fca65a6
log time needed to convert
shoshta73 Dec 1, 2024
1900110
add ~5mb json file for testing
shoshta73 Dec 1, 2024
a767d9e
add large csv file for testing
shoshta73 Dec 1, 2024
9feddfb
add jsonc to csv conversion
shoshta73 Dec 1, 2024
5d1df98
add large files to test input
shoshta73 Dec 1, 2024
e55f91c
fix random fileIO errors
shoshta73 Dec 1, 2024
867f533
add windows docs scripts
shoshta73 Dec 1, 2024
37a6366
add docs for the converter app
shoshta73 Dec 1, 2024
4fea59f
add ability for converter app to read directly from a file
shoshta73 Dec 1, 2024
135c278
fix blocking, and incorect main parameters
shoshta73 Dec 1, 2024
3057029
fix test_main not passing stdin
shoshta73 Dec 1, 2024
f54a552
show all durations on tests, clean stdout of running tests
shoshta73 Dec 1, 2024
fdfdba2
combine helpers example
shoshta73 Dec 10, 2024
514875c
combine json maestro examples
shoshta73 Dec 10, 2024
fe46cc4
add loader documentation
shoshta73 Dec 10, 2024
de1f8ef
add converter documentation
shoshta73 Dec 10, 2024
9868369
remove assertions in source
shoshta73 Dec 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 63 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,11 @@ jobs:
source ./venv/bin/activate
pip install -r dev.requirements
pip install -r requirements.txt
pip install .

- name: Run tests
run: |
source ./venv/bin/activate
pytest
pytest -vv

test-3-10:
needs: test-3-9
Expand All @@ -82,12 +81,11 @@ jobs:
source ./venv/bin/activate
pip install -r dev.requirements
pip install -r requirements.txt
pip install .

- name: Run tests
run: |
source ./venv/bin/activate
pytest
pytest -vv

test-3-11:
needs: test-3-10
Expand All @@ -113,12 +111,11 @@ jobs:
source ./venv/bin/activate
pip install -r dev.requirements
pip install -r requirements.txt
pip install .

- name: Run tests
run: |
source ./venv/bin/activate
pytest
pytest -vv

test-3-12:
needs: test-3-11
Expand All @@ -144,12 +141,41 @@ jobs:
source ./venv/bin/activate
pip install -r dev.requirements
pip install -r requirements.txt
pip install .

- name: Run tests
run: |
source ./venv/bin/activate
pytest
pytest -vv

test-3-13:
needs: test-3-12

strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
fail-fast: true

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.13
uses: actions/setup-python@v3
with:
python-version: "3.13"
- name: Setup Venv
run: |
python -m venv ./venv

- name: Install packages
run: |
source ./venv/bin/activate
pip install -r dev.requirements
pip install -r requirements.txt

- name: Run tests
run: |
source ./venv/bin/activate
pytest -vv

pip-install-3-9:
needs: test-3-9
Expand Down Expand Up @@ -264,3 +290,32 @@ jobs:
run: python3 -m build
- name: Install
run: python3 -m pip install .

pip-install-3-13:
needs:
- test-3-13
- pip-install-3-12

strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
fail-fast: true

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.13
uses: actions/setup-python@v3
with:
python-version: "3.13"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r dev.requirements
- name: Install build deps
run: python3 -m pip install --upgrade build
- name: Build
run: python3 -m build
- name: Install
run: python3 -m pip install .
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -163,4 +163,5 @@ pyrightconfig.json
.debugpy

# docs build directory
book
book
docs/LICENSE
5 changes: 5 additions & 0 deletions data/example.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
name,age,text_editor
John,30,Sublime Text
Jane,25,Visual Studio Code
Michael,35,Neovim
Noah,40,Atom
18 changes: 18 additions & 0 deletions data/example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"John": {
"age": "30",
"text_editor": "Sublime Text"
},
"Jane": {
"age": "25",
"text_editor": "Visual Studio Code"
},
"Michael": {
"age": "35",
"text_editor": "Neovim"
},
"Noah": {
"age": "40",
"text_editor": "Atom"
}
}
22 changes: 22 additions & 0 deletions data/example_invalid_for_csv_conversion.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"John": {
"age": "30",
"text_editor": "Sublime Text",
"invalid_key": "invalid_value"
},
"Jane": {
"age": "25",
"text_editor": "Visual Studio Code",
"place_of_birth": "New York"
},
"Michael": {
"age": "35",
"text_editor": "Neovim",
"job": "Software Engineer"
},
"Noah": {
"age": "40",
"text_editor": "Atom",
"favorite_color": "Blue"
}
}
21 changes: 21 additions & 0 deletions data/really_large.csv

Large diffs are not rendered by default.

Loading
Loading