Stuck in "Update Autograder" -- process very slow?

[trgardos]

Is there a way to kill the docker image build process after you click "Update Autograder"?

Also, is anyone finding the update process extremely slow?

I initially tried to load an autograder.zip that did some machine learning training in the solutions and tests and that was taking forever to build. Are the tests executed during the autograder image build process? Then I ended up deleting that assignment and starting a new one with the machine learning training removed but that was also taking a very long time. I killed it after ~12 hours.

When you delete an assignment, does it kill the autograder build processes? Or do they continue somehow as ghost processes and would that impact my subsequent attempts to create new programming assignments?

Any insights into how the build process gets queued, etc., would be helpful.

[erickim555]

I'm also seeing this from my end. Interestingly, I was able to successfully upgrade the autograder + build the docker image two days ago (around 11 pm PST of Sun Aug 24th), and building the docker image took like <15 minutes (as usual).

Now, I'm seeing that building the docker image seemingly gets stuck, I waited up to 45 minutes with it hanging at this part:

Solving environment: ...working... done

Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/root/anaconda/envs/cs182_hw4/bin/python', '-m', 'pip', 'install', '-U', '-r', '/autograder/source/condaenv._8btmqo_.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting gradescope-utils==0.4.0
  Downloading gradescope_utils-0.4.0-py2.py3-none-any.whl (5.6 kB)
Installing collected packages: gradescope-utils
Successfully installed gradescope-utils-0.4.0

done
#
# To activate this environment, use
#
#     $ conda activate cs182_hw4
#
# To deactivate an active environment, use
#
#     $ conda deactivate

+ apt-get clean
+ rm -rf /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-backports_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-backports_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-backports_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_focal-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_focal-security_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_focal-security_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_focal-security_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_focal-security_universe_binary-amd64_Packages.lz4 /tmp/autograder.zip /tmp/dumb-init_1.2.0_amd64.deb '/var/tmp/*'

This is blocking us from updating our assignment autograder, if we can get help with this issue that'd be great. Thanks!

[erickim555]

Update: perhaps the issue is resolved, my docker image just finished building. Perhaps it was a transient issue?

[erickim555]

Update: I'm seeing this again from my end, the exact same hang. It's been building for nearly 2 hours, when it should finish in ~10 minutes normally:

Successfully built pandas matplotlib numpy
Installing collected packages: pytz, mpmath, gradescope-utils, tzdata, typing-extensions, tqdm, threadpoolctl, sympy, six, pyparsing, pillow, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, kiwisolver, joblib, fsspec, fonttools, filelock, cycler, scipy, python-dateutil, nvidia-cusparse-cu12, nvidia-cudnn-cu12, jinja2, contourpy, scikit-learn, pandas, nvidia-cusolver-cu12, matplotlib, torch
Successfully installed MarkupSafe-3.0.2 contourpy-1.3.1 cycler-0.12.1 filelock-3.16.1 fonttools-4.55.0 fsspec-2024.10.0 gradescope-utils-0.4.0 jinja2-3.1.4 joblib-1.4.2 kiwisolver-1.4.7 matplotlib-3.8.0 mpmath-1.3.0 networkx-3.4.2 numpy-1.26.4 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-nccl-cu12-2.21.5 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.4.127 packaging-24.2 pandas-2.2.2 pillow-11.0.0 pyparsing-3.2.0 python-dateutil-2.9.0.post0 pytz-2024.2 scikit-learn-1.5.2 scipy-1.14.1 six-1.16.0 sympy-1.13.1 threadpoolctl-3.5.0 torch-2.5.1 tqdm-4.66.6 typing-extensions-4.12.2 tzdata-2024.2

done
#
# To activate this environment, use
#
#     $ conda activate datac182_final_proj_faulty_commit
#
# To deactivate an active environment, use
#
#     $ conda deactivate

+ apt-get clean
+ rm -rf /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_universe_binary-amd64_Packages.lz4 /tmp/autograder.zip /tmp/dumb-init_1.2.0_amd64.deb '/var/tmp/*'

I'm not sure what's going wrong, or even if it's an issue from our end (eg setup.sh is incorrect), or if it's a Gradescope-end issue. I have a feeling it's Gradescope-end, as our setup.sh has worked correctly in the past, so it feels like a transient issue on Gradescope's side, as the issue seems to come and go.

For reference, here's our Gradescope page: https://www.gradescope.com/courses/837491/assignments/5393019/configure_autograder

[erickim555]

For what it's worth: here is what a "normal" run output looks like (without the hang), this finished within 15 mins:

done
#
# To activate this environment, use
#
#     $ conda activate datac182_final_proj_faulty_commit
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Done creating conda env!
+ apt-get clean
+ rm -rf /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_universe_binary-amd64_Packages.lz4 /tmp/autograder.zip /tmp/dumb-init_1.2.0_amd64.deb '/var/tmp/*'
 ---> Removed intermediate container ec4e8e221513
 ---> dc89a85b7d95
Successfully built dc89a85b7d95
Successfully tagged gradescope/autograders:us-prod-docker_image-431660

Note that the hang starts before the ---> Removed intermediate container ec4e8e221513 line. I wonder if there is some Docker-related issue/hang on the Gradescope servers? I don't think it's an issue with my setup.sh. Thanks!