Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Branch][DeepSparse Evaluation API] Update lm-eval, perplexity, additional datasets #1580

Merged
merged 36 commits into from
Feb 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6035536
initial implementation
dbogunowicz Jan 29, 2024
53cb9ec
initial commit
dbogunowicz Jan 30, 2024
6599f41
add some more tests for hardening
dbogunowicz Jan 30, 2024
4721c1f
Update src/deepsparse/evaluation/cli.py
dbogunowicz Jan 30, 2024
1247794
Update src/deepsparse/transformers/pipelines/text_generation/pipeline.py
dbogunowicz Jan 30, 2024
9e88f89
Apply suggestions from code review
dbogunowicz Jan 30, 2024
fdb21c6
quality
dbogunowicz Jan 30, 2024
be80132
Merge branch 'main' into feature/damian/ui_improvements
dbogunowicz Jan 30, 2024
5d40b8d
Merge remote-tracking branch 'origin/main' into feature/damian/fix_lm…
dbogunowicz Jan 31, 2024
3e5b7a8
fix the UI, implement loglikelihood function
dbogunowicz Feb 1, 2024
ff0944b
Merge branch 'main' into feature/damian/fix_lm_eval
dbogunowicz Feb 1, 2024
f38f0db
remove unneccessary file
dbogunowicz Feb 1, 2024
dd45493
Merge branch 'feature/damian/fix_lm_eval' of github.com:neuralmagic/d…
dbogunowicz Feb 1, 2024
cd10b92
Merge branch 'main' into feature/damian/ui_improvements
dbogunowicz Feb 1, 2024
b2aad17
initial commit
dbogunowicz Feb 2, 2024
35454a1
tests passing, refactor time!
dbogunowicz Feb 2, 2024
d3b84f8
cleanup
dbogunowicz Feb 2, 2024
e7d8c31
Update test_evaluator.py
dbogunowicz Feb 5, 2024
a148fc5
finished
dbogunowicz Feb 5, 2024
3b5977b
rebase
dbogunowicz Feb 5, 2024
a9e9847
quality
dbogunowicz Feb 5, 2024
787ee45
rebase
dbogunowicz Feb 5, 2024
b5a6d6d
manual testing
dbogunowicz Feb 5, 2024
d0698e7
Merge remote-tracking branch 'origin/main' into feature/damian/genera…
dbogunowicz Feb 5, 2024
e10f0c9
UI improvements
dbogunowicz Feb 5, 2024
48a5900
new UI adaptations
dbogunowicz Feb 6, 2024
44e3e6e
make test more lightweight
dbogunowicz Feb 6, 2024
abb6ab8
fix tests 2
dbogunowicz Feb 6, 2024
79fd7e0
Merge branch 'main' into feature/damian/generate_until
dbogunowicz Feb 7, 2024
e5aad65
good point Michael
dbogunowicz Feb 7, 2024
06302dc
Merge branch 'main' into feature/damian/generate_until
dbogunowicz Feb 8, 2024
d65cac6
Return to the name `lm-evaluation-harness` but add alias `lm-eval-har…
dbogunowicz Feb 8, 2024
e0b4f36
Merge branch 'main' into feature/damian/generate_until
dbogunowicz Feb 9, 2024
b82b49b
[DeepSparse Evaluation API] Perplexity (#1555)
dbogunowicz Feb 9, 2024
d4cdd98
Merge branch 'main' into feature/damian/generate_until
dbogunowicz Feb 9, 2024
7a3ad2f
move the registration of the perplexity eval function where it belongs
dbogunowicz Feb 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'origin/main' into feature/damian/genera…
…te_until
  • Loading branch information
dbogunowicz committed Feb 5, 2024
commit d0698e7d3f7e97a5d99e0f9ee413bc3fdaf032aa
2 changes: 2 additions & 0 deletions src/deepsparse/evaluation/cli.py
Original file line number Diff line number Diff line change
@@ -92,6 +92,8 @@
"model_path",
type=click.Path(dir_okay=True, file_okay=True),
required=True,
help="A path to a remote or local directory containing ONNX/torch model "
"(including all the auxiliary files) or a SparseZoo stub",
)
@click.option(
"-d",
Original file line number Diff line number Diff line change
@@ -12,8 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from transformers import AutoModelForCausalLM

import pytest
from deepsparse.evaluation.integrations import try_import_lm_evaluation_harness
from deepsparse.evaluation.utils import create_pipeline
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.