Conflicts with latest Docker refactor and documentation

[ahankinson]

The latest round of refactoring on the Docker files has meant that all of the existing documentation for how to run Qlever with docker is out of date.

Some examples:

• Previously the documentation specified "/index", but now the script seems to fail if the indexes are not found in "/data"? And this seems to be hardcoded? https://github.com/ad-freiburg/qlever/blob/master/docker-entrypoint.sh#L40
• The instructions for how to set users are not at all clear. There is a comment in the docker entrypoint script that says the use of -u shouldn't be used, but then several examples that do use it.
• There are no instructions on the expected form of the host system or how docker is executed. Should the docker command be run as root? Or should there be a qlever user on the host system? What privileges should this user have?

After working through these I am trying to run it with the following command, and it still is not working. It's failing with a usermod: no changes error, and just keeps restarting. The basics of this command worked prior to the latest Dockerfile update, but I've made changes to the -v argument (to point it to "/data"), added the -w argument, and the two -e arguments for UID / GID, after unsuccessfully trying to use -u (until I saw the note in the bash file).

docker run -d --restart unless-stopped -v /opt/qlever/qlever-indices/myindex:/data -w /data -e UID=$(id -u qlever) -e GID=$(id -g qlever) -p 7001:7001 -e INDEX_PREFIX=myindex -e MEMORY_FOR_QUERIES=4GB -e CACHE_MAX_SIZE_GB=30GB -e CACHE_MAX_SIZE_GB_SINGLE_ENTRY=5GB --name qlever.myindex adfreiburg/qlever

I would be grateful for any help in updating the command to work with the latest changes!

[ahankinson]

Looking through #1439 perhaps also @ludovicm67 can help understand the new setup?

If anyone else is currently struggling, for now I have reverted to the last Docker release before the change:

docker pull adfreiburg/qlever:commit-44e2ba8

I think I understand that now the ENTRYPOINT is no longer set to automatically start the server, but to run the docker entrypoint script, and then we have to now use the qlever script to start the server, i.e., `[docker command] -c "qlever start ..."

However, this gives me an error,

Invoking command `start` without a Qleverfile. You have to specify all required arguments on the command line. This is possible, but not recommended.

Command: start

docker run -d --restart=unless-stopped -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -p None:None -w /index --init --entrypoint bash --name qlever.server.rism docker.io/adfreiburg/qlever -c 'ServerMain -i rism -j 8 -p None -m 5G -c 2G -e 1G -k 200 -s 30s > rism.server-log.txt 2>&1'

So it looks like the qlever start command is trying to start docker within docker?

[hannahbast]

@ahankinson Sorry to hear about those problems + I have a few questions:

1. You mention that "previously the documentation specified". Which documentation are you referring to? For over two years now, the QLever README.md says: "Use QLever via the qlever script, following the instructions on https://github.com/ad-freiburg/qlever-control ."

2. The qlever script (which you can install via the respective repo or even simpler via pip install qlever) should work with the new Dockerfile just as it worked with the old Dockerfile.

3. You have posted a Docker command line. Where does that command line come from? There is no scenario where you should have to write such a command line yourself.

4. If you don't want the goodies of the new entrypoint script, you can simply call docker with --entrypoint bash and then do whatever you want in the container, including calling the QLever binaries IndexBuilderMain and ServerMain directly (which you shouldn't, but you can).

Overall, my impression is that you have been using QLever in a deprecated way for a long time and that deprecated way was broken by the recent changes to the Dockerfile.

[ahankinson]

Hi @hannahbast , and thanks for you response!

I was looking at this documentation, particularly the "Start the engine" section: https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md

With the old Dockerfile I didn't need to use the qlever command, since the old entrypoint in the Dockerfile ran the server directly.

qlever/Dockerfile

Line 51 in 80938b7

ENTRYPOINT ["/bin/sh", "-c", "exec ServerMain -i \"/index/${INDEX_PREFIX}\" -j 8 -m ${MEMORY_FOR_QUERIES} -c ${CACHE_MAX_SIZE_GB} -e ${CACHE_MAX_SIZE_GB_SINGLE_ENTRY} -k ${CACHE_MAX_NUM_ENTRIES} -p 7001 \"$@\"", "--"]

I pasted a couple docker command lines. The one in the original post comes from the Quickstart documentation above (with some modification and re-ordering).

The docker command in in my second message comes from the qlever script in the new setup, when I try to use the new setup. I'm obviously not doing something right.

It's not that I don't want the new "goodies" -- far from it! I like to keep our installation up-to-date to take advantage of the new developments. But this particular change could use a bit more clarification on the setup and state of the machine.

I didn't think I was using it in a deprecated way... There's even a "quickstart.md.DEPRECATED" file which I was not using... 😁