Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: dfencoder import missing from prod DFP example #1407

Closed
2 tasks done
pdmack opened this issue Dec 5, 2023 · 1 comment
Closed
2 tasks done

[BUG]: dfencoder import missing from prod DFP example #1407

pdmack opened this issue Dec 5, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@pdmack
Copy link
Contributor

pdmack commented Dec 5, 2023

Version

23.11

Which installation method(s) does this occur on?

Docker, Conda, Source

Describe the bug.

Last step of production DFP example fails with 'dfencoder' import missing.

Minimum reproducible example

python dfp_azure_pipeline.py --train_users none  --start_time "2022-08-30" --input_file="../../../data/dfp/azure-inference-data/*.json"

Relevant log output

Click here to see error details

[Downloading artifacts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 20.53it/s]
2023/12/05 18:28:35 WARNING mlflow.pytorch: Stored model version '1.13.1+cu116' does not match installed PyTorch version '2.0.1'
Error trying to get model
Traceback (most recent call last):
File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/stages/dfp_inference_stage.py", line 101, in on_data
loaded_model = model_cache.load_model(self._client)
File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/utils/model_cache.py", line 109, in load_model
self._model = mlflow.pytorch.load_model(model_uri=self._model_uri)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/pytorch/init.py", line 713, in load_model
return _load_model(path=torch_model_artifacts_path, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/pytorch/init.py", line 628, in _load_model
return torch.load(model_path, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/torch/serialization.py", line 809, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/torch/serialization.py", line 1172, in _load
result = unpickler.load()
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/torch/serialization.py", line 1165, in find_class
return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'dfencoder'

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@pdmack pdmack added the bug Something isn't working label Dec 5, 2023
@dagardner-nv dagardner-nv self-assigned this Dec 5, 2023
@pdmack
Copy link
Contributor Author

pdmack commented Dec 5, 2023

closing, was due to old mlflow docker volumes from previous test runs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

2 participants