[BUG]: error in completion pipeline - LLM example

[aadesoba-nv]

Version

25.02

Which installation method(s) does this occur on?

Docker

Describe the bug.

The completion pipeline under the LLM example doesn't work with running with --llm_service=NemoLLM.

Minimum reproducible example

python examples/llm/main.py completion pipeline --llm_service=NemoLLM

Relevant log output

Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/pipeline/pipeline.py", line 408, in post_start
await executor.join_async()
File "/opt/conda/envs/morpheus/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/opt/conda/envs/morpheus/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.await())
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus_llm/llm/nodes/llm_generate_node.py", line 55, in execute
results = await self._llm_client.generate_batch_async(inputs, return_exceptions=self._return_exceptions)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus_llm/llm/services/nemo_llm_service.py", line 190, in generate_batch_async
results = await asyncio.gather(*futures, return_exceptions=return_exceptions)
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus_llm/llm/services/nemo_llm_service.py", line 153, in _process_one_async
raise RuntimeError(
RuntimeError: Failed to generate response for prompt 'What is the capital of France?' after 5 attempts. Errors: ['\n401\n', '\n401\n', '\n401\n', '\n401\n', '\n401\n']

Full env printout

Click here to see environment details

 **git***
 commit 8cc9ab7440aafd000badf8e90918cff7b962b784 (HEAD -> branch-25.02)
 Author: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
 Date:   Mon Jan 27 11:50:22 2025 -0800
 
 Document known ARM64 issues (#2128)
 
 * Document examples which don't work on Arm64
 * Lazily import `milvus` libs allowing users to run the LLM `completion` and `agent` pipelines on Arm64 systems.
 * Fix handling of `DOCKER_TARGET_ARCH` in docker build scripts.
 
 Closes #2096
 
 
 ## By Submitting this PR I confirm:
 - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
 - When the PR is ready for review, new or existing tests cover these changes.
 - When the PR is ready for review, the documentation is up to date with these changes.
 
 Authors:
 - David Gardner (https://github.com/dagardner-nv)
 
 Approvers:
 - Yuchen Zhang (https://github.com/yczhang-nv)
 
 URL: https://github.com/nv-morpheus/Morpheus/pull/2128
 **git submodules***
 c83e22fc0be11a522d51ee79eb64b2d94d55ae2c external/morpheus-visualizations (v25.02.00a-1-gc83e22f)
 59ed80833bc9cb28ea6b137ba63313eab94c552c external/utilities (v25.02.00a-13-g59ed808)
 
 ***OS Information***
 DGX_NAME="DGX Station A100"
 DGX_PRETTY_NAME="NVIDIA DGX Station A100"
 DGX_SWBUILD_DATE="2024-03-15-14-25-26"
 DGX_SWBUILD_VERSION="6.2.0"
 DGX_COMMIT_ID="9f30f3a"
 DGX_PLATFORM="DGX Station A100"
 DGX_SERIAL_NUMBER="1635220000041"
 
 DGX_OTA_VERSION="6.2.1"
 DGX_OTA_DATE="Tue Jun 11 08:57:13 AM PDT 2024"
 
 DGX_OTA_VERSION="6.3.0"
 DGX_OTA_DATE="Thu Aug 29 09:35:42 AM PDT 2024"
 
 DGX_OTA_VERSION="6.3.1"
 DGX_OTA_DATE="Thu Oct 24 09:29:39 AM PDT 2024"
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=22.04
 DISTRIB_CODENAME=jammy
 DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
 PRETTY_NAME="Ubuntu 22.04.5 LTS"
 NAME="Ubuntu"
 VERSION_ID="22.04"
 VERSION="22.04.5 LTS (Jammy Jellyfish)"
 VERSION_CODENAME=jammy
 ID=ubuntu
 ID_LIKE=debian
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 UBUNTU_CODENAME=jammy
 Linux saeA-dgx-a100-80gb 5.15.0-1058-nvidia #59-Ubuntu SMP Tue Jun 4 01:39:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Tue Jan 28 14:34:09 2025
 +-----------------------------------------------------------------------------------------+
 | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
 |-----------------------------------------+------------------------+----------------------+
 | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
 |                                         |                        |               MIG M. |
 |=========================================+========================+======================|
 |   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
 | N/A   40C    P0             68W /  275W |   45282MiB /  81920MiB |      0%      Default |
 |                                         |                        |             Disabled |
 +-----------------------------------------+------------------------+----------------------+
 |   1  NVIDIA A100-SXM4-80GB          On  |   00000000:47:00.0 Off |                    0 |
 | N/A   40C    P0             70W /  275W |   74880MiB /  81920MiB |      0%      Default |
 |                                         |                        |             Disabled |
 +-----------------------------------------+------------------------+----------------------+
 |   2  NVIDIA A100-SXM4-80GB          On  |   00000000:81:00.0 Off |                    0 |
 | N/A   39C    P0             61W /  275W |       6MiB /  81920MiB |      0%      Default |
 |                                         |                        |             Disabled |
 +-----------------------------------------+------------------------+----------------------+
 |   3  NVIDIA DGX Display             On  |   00000000:C1:00.0 Off |                  N/A |
 | 33%   45C    P8             N/A /   50W |       0MiB /   4096MiB |      0%      Default |
 |                                         |                        |                  N/A |
 +-----------------------------------------+------------------------+----------------------+
 |   4  NVIDIA A100-SXM4-80GB          On  |   00000000:C2:00.0 Off |                    0 |
 | N/A   39C    P0             65W /  275W |       0MiB /  81920MiB |      0%      Default |
 |                                         |                        |             Disabled |
 +-----------------------------------------+------------------------+----------------------+
 
 +-----------------------------------------------------------------------------------------+
 | Processes:                                                                              |
 |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
 |        ID   ID                                                               Usage      |
 |=========================================================================================|
 |    0   N/A  N/A    479991      C   tritonserver                                 7774MiB |
 |    0   N/A  N/A   2010507      C   tritonserver                                14012MiB |
 |    0   N/A  N/A   3275832      C   python                                      22986MiB |
 |    0   N/A  N/A   3286839      C   ...wsj25/tritonserver/bin/tritonserver        478MiB |
 |    1   N/A  N/A    669756      C   /opt/nim/llm/.venv/bin/python3              74868MiB |
 +-----------------------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:                       x86_64
 CPU op-mode(s):                     32-bit, 64-bit
 Address sizes:                      43 bits physical, 48 bits virtual
 Byte Order:                         Little Endian
 CPU(s):                             128
 On-line CPU(s) list:                0-127
 Vendor ID:                          AuthenticAMD
 Model name:                         AMD EPYC 7742 64-Core Processor
 CPU family:                         23
 Model:                              49
 Thread(s) per core:                 2
 Core(s) per socket:                 64
 Socket(s):                          1
 Stepping:                           0
 Frequency boost:                    enabled
 CPU max MHz:                        2250.0000
 CPU min MHz:                        1500.0000
 BogoMIPS:                           4492.01
 Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
 Virtualization:                     AMD-V
 L1d cache:                          2 MiB (64 instances)
 L1i cache:                          2 MiB (64 instances)
 L2 cache:                           32 MiB (64 instances)
 L3 cache:                           256 MiB (16 instances)
 NUMA node(s):                       4
 NUMA node0 CPU(s):                  0-15,64-79
 NUMA node1 CPU(s):                  16-31,80-95
 NUMA node2 CPU(s):                  32-47,96-111
 NUMA node3 CPU(s):                  48-63,112-127
 Vulnerability Gather data sampling: Not affected
 Vulnerability Itlb multihit:        Not affected
 Vulnerability L1tf:                 Not affected
 Vulnerability Mds:                  Not affected
 Vulnerability Meltdown:             Not affected
 Vulnerability Mmio stale data:      Not affected
 Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
 Vulnerability Spec rstack overflow: Mitigation; safe RET
 Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
 Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
 Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
 Vulnerability Srbds:                Not affected
 Vulnerability Tsx async abort:      Not affected
 
 ***CMake***
 /usr/bin/cmake
 cmake version 3.22.1
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /usr/bin/g++
 g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
 Copyright (C) 2021 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /usr/local/cuda/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2023 NVIDIA Corporation
 Built on Tue_Aug_15_22:02:13_PDT_2023
 Cuda compilation tools, release 12.2, V12.2.140
 Build cuda_12.2.r12.2/compiler.33191640_0
 
 ***Python***
 
 ***Environment Variables***
 PATH                            : /home/aadesoba/.vscode-server/cli/servers/Stable-fabdb6a30b49f79a7aba0f2ad9df9b399473380f/server/bin/remote-cli:/home/aadesoba/.local/bin:/usr/local/cuda/bin:/opt/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
 LD_LIBRARY_PATH                 :
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    :
 PYTHON_PATH                     :
 
 conda not found
 ***pip packages***
 /usr/bin/pip
 Package                  Version
 ------------------------ ----------------
 aiohttp                  3.9.5
 aiosignal                1.3.1
 annotated-types          0.7.0
 anyio                    4.4.0
 apturl                   0.5.2
 async-timeout            4.0.3
 attrs                    21.2.0
 Automat                  20.2.0
 Babel                    2.8.0
 bcrypt                   3.2.0
 blinker                  1.4
 Brlapi                   0.8.3
 certifi                  2020.6.20
 cffi                     1.16.0
 cfgv                     3.3.1
 chardet                  4.0.0
 charset-normalizer       3.3.2
 click                    8.0.3
 cloud-init               24.3.1
 colorama                 0.4.4
 command-not-found        0.3
 configobj                5.0.6
 constantly               15.1.0
 cryptography             3.4.8
 cupshelpers              1.0
 dbus-python              1.2.18
 defer                    1.0.6
 distlib                  0.3.4
 distro                   1.7.0
 distro-info              1.1+ubuntu0.2
 duplicity                0.8.21
 evdev                    1.6.1
 exceptiongroup           1.2.2
 fail2ban                 0.11.2
 fasteners                0.14.1
 filelock                 3.15.4
 frozenlist               1.4.1
 fsspec                   2024.6.1
 future                   0.18.2
 git-filter-repo          2.34.0
 greenlet                 3.0.3
 grpcio                   1.65.1
 grpcio-tools             1.48.2
 h11                      0.14.0
 httpcore                 1.0.5
 httplib2                 0.20.2
 httpx                    0.27.0
 huggingface-hub          0.24.6
 hyperlink                21.0.0
 identify                 2.4.10
 idna                     3.3
 importlib-metadata       4.6.4
 incremental              21.3.0
 iotop                    0.6
 ipython_genutils         0.2.0
 jeepney                  0.7.1
 Jinja2                   3.0.3
 jsonpatch                1.33
 jsonpointer              2.0
 jsonschema               3.2.0
 jupyter-core             4.9.1
 keyboard                 0.13.5
 keyring                  23.5.0
 langchain                0.2.11
 langchain-core           0.2.23
 langchain-openai         0.1.17
 langchain-text-splitters 0.2.2
 langsmith                0.1.93
 language-selector        0.1
 launchpadlib             1.10.16
 lazr.restfulclient       0.14.4
 lazr.uri                 1.0.6
 lockfile                 0.12.2
 louis                    3.20.0
 macaroonbakery           1.3.1
 Mako                     1.1.3
 MarkupSafe               2.0.1
 monotonic                1.6
 more-itertools           8.10.0
 multidict                6.0.5
 netifaces                0.11.0
 nodeenv                  0.13.4
 numpy                    1.26.4
 nvidia-riva-client       2.16.0
 nvsm                     24.6.2
 nvsm-dshm                24.6.2
 nvsm-health              24.6.2
 nvsm-sdk                 1.0.0
 oauthlib                 3.2.0
 olefile                  0.46
 openai                   1.37.1
 orjson                   3.10.6
 packaging                24.1
 paho-mqtt                1.5.1
 pandas                   2.2.2
 paramiko                 2.9.3
 pexpect                  4.8.0
 Pillow                   9.0.1
 pip                      22.0.2
 platformdirs             2.5.1
 pre-commit               2.17.0
 protobuf                 3.20.3
 psutil                   5.9.0
 ptyprocess               0.7.0
 pyasn1                   0.4.8
 pyasn1-modules           0.2.1
 pycairo                  1.20.1
 pycparser                2.22
 pycups                   2.0.1
 pydantic                 2.8.2
 pydantic_core            2.20.1
 PyGObject                3.42.1
 PyHamcrest               2.0.2
 PyJWT                    2.3.0
 pymacaroons              0.13.0
 PyNaCl                   1.5.0
 pynput                   1.7.7
 pyOpenSSL                21.0.0
 pyparsing                2.4.7
 pyRFC3339                1.1
 pyrsistent               0.18.1
 pyserial                 3.5
 python-apt               2.4.0+ubuntu4
 python-dateutil          2.9.0.post0
 python-debian            0.1.43+ubuntu1.1
 python-magic             0.4.24
 python-xlib              0.33
 pytz                     2022.1
 pyxdg                    0.27
 PyYAML                   5.4.1
 regex                    2024.7.24
 reportlab                3.6.8
 requests                 2.32.3
 scipy                    1.14.0
 screen-resolution-extra  0.0.0
 SecretStorage            3.3.1
 service-identity         18.1.0
 setuptools               71.1.0
 six                      1.16.0
 sniffio                  1.3.1
 sos                      4.5.6
 sounddevice              0.4.7
 soundfile                0.12.1
 SQLAlchemy               2.0.31
 ssh-import-id            5.11
 systemd-python           234
 tenacity                 8.5.0
 tiktoken                 0.7.0
 toml                     0.10.2
 tqdm                     4.66.4
 traitlets                5.1.1
 Twisted                  22.1.0
 typing_extensions        4.12.2
 tzdata                   2024.1
 ubuntu-drivers-common    0.0.0
 ubuntu-pro-client        8001
 ufw                      0.36.1
 urllib3                  1.26.5
 usb-creator              0.3.7
 uv                       0.5.15
 virtualenv               20.13.0+ds
 wadllib                  1.3.6
 wheel                    0.37.1
 xdg                      5
 xkit                     0.0.0
 xlrd                     1.2.0
 yarl                     1.9.4
 zipp                     1.0.0
 zope.interface           5.4.0

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[morpheus-bot-test]

Hi @aadesoba-nv!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can!
In the meantime, feel free to add any relevant information to this issue.

[aadesoba-nv]

Resolved!

[dagardner-nv]

This is being replaced by #2153