HIVE-28437: Add documentation for initializing the system schemas for HiveServer2 for Docker Image

[linghengqian]

What changes were proposed in this pull request?

• Add documentation for initializing the system schemas for HiveServer2 for Docker Image. See https://hive.apache.org/development/quickstart/ .

Why are the changes needed?

• The result of my 6-month investigation on https://issues.apache.org/jira/browse/HIVE-28437 is that HiveServer2 definitely supports the use of the INFORMATION_SCHEMA database, but it is not enabled by default, which causes a lot of trouble for unit testing scenarios of third-party libraries such as testcontainers-java .
• Early investigations came from Add unit tests for information_schema database linghengqian/hive-server2-jdbc-driver#22 , Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests shardingsphere#29052 . It seems that the Hive documentation https://hive.apache.org/docs/latest/hive-schema-tool_34835119/ does not support viewing images. I will still refer to the documentation at https://cwiki.apache.org/confluence/display/hive/hive+schema+tool .
• Without the current PR, it would be very troublesome to use Hive's Docker Image to retrieve the information_schema database in unit tests. Because /opt/hive/bin/schematool is not part of the SQL command, /opt/hive/bin/schematool is part of the shell command, and using /opt/hive/bin/schematool requires entering the Hive Docker Image.

import org.junit.jupiter.api.Test;
import org.testcontainers.containers.Container.ExecResult;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.sql.*;
import java.time.Duration;
import java.time.temporal.ChronoUnit;
import static org.awaitility.Awaitility.await;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.junit.jupiter.api.Assertions.assertThrows;
@SuppressWarnings({"SqlNoDataSourceInspection", "resource"})
@Testcontainers
public class InformationSchemaTest {
    @Container
    public static final GenericContainer<?> CONTAINER = new GenericContainer<>(DockerImageName.parse("apache/hive:4.0.1"))
            .withEnv("SERVICE_NAME", "hiveserver2")
            .withExposedPorts(10000);

    @Test
    void test() throws SQLException, IOException, InterruptedException {
        String jdbcUrlPrefix = "jdbc:hive2://" + CONTAINER.getHost() + ":" + CONTAINER.getMappedPort(10000);
        await().atMost(Duration.of(30L, ChronoUnit.SECONDS)).ignoreExceptions().until(() -> {
            DriverManager.getConnection(jdbcUrlPrefix).close();
            return true;
        });
        try (Connection connection = DriverManager.getConnection(jdbcUrlPrefix);
             Statement statement = connection.createStatement()) {
            statement.execute("CREATE DATABASE demo_ds_0");
        }
        try (Connection connection = DriverManager.getConnection(jdbcUrlPrefix + "/demo_ds_0");
             Statement statement = connection.createStatement()) {
            statement.execute("CREATE TABLE IF NOT EXISTS t_order (\n" +
                    "    order_id   BIGINT NOT NULL,\n" +
                    "    order_type INT,\n" +
                    "    user_id    INT    NOT NULL,\n" +
                    "    address_id BIGINT NOT NULL,\n" +
                    "    status     string,\n" +
                    "    PRIMARY KEY (order_id) disable novalidate\n" +
                    ") STORED BY ICEBERG STORED AS ORC TBLPROPERTIES ('format-version' = '2')");
            statement.execute("TRUNCATE TABLE t_order");
            statement.executeUpdate("INSERT INTO t_order (order_id, user_id, order_type, address_id, status) VALUES (1, 1, 1, 1, 'INSERT_TEST')");
            ResultSet resultSet = statement.executeQuery("select * from t_order");
            assertThat(resultSet.next(), is(true));
        }
        assertThrows(SQLException.class, () -> DriverManager.getConnection(jdbcUrlPrefix + "/information_schema").close());
        ExecResult infoResult = CONTAINER.execInContainer(
                "/opt/hive/bin/schematool",
                "-info",
                "-dbType", "hive",
                "-metaDbType", "derby",
                "-url", "jdbc:hive2://localhost:10000/default"
        );
        assertThat(infoResult.getStdout(), is("Metastore connection URL:\t jdbc:hive2://localhost:10000/default\n" +
                "Metastore connection Driver :\t org.apache.hive.jdbc.HiveDriver\n" +
                "Metastore connection User:\t APP\n"));
        ExecResult initResult = CONTAINER.execInContainer(
                "/opt/hive/bin/schematool",
                "-initSchema",
                "-dbType", "hive",
                "-metaDbType", "derby",
                "-url", "jdbc:hive2://localhost:10000/default"
        );
        assertThat(initResult.getStdout(), is("Initializing the schema to: 4.0.0\n" +
                "Metastore connection URL:\t jdbc:hive2://localhost:10000/default\n" +
                "Metastore connection Driver :\t org.apache.hive.jdbc.HiveDriver\n" +
                "Metastore connection User:\t APP\n" +
                "Starting metastore schema initialization to 4.0.0\n" +
                "Initialization script hive-schema-4.0.0.hive.sql\n" +
                "Initialization script completed\n"));
        try (Connection connection = DriverManager.getConnection(jdbcUrlPrefix + "/information_schema");
             Statement statement = connection.createStatement()) {
            ResultSet resultSet = statement.executeQuery("select * from information_schema.COLUMNS limit 100");
            assertThat(resultSet.next(), is(true));
        }
    }
}

Does this PR introduce any user-facing change?

• This will affect all users of the Hive Docker Image.

Is the change a dependency upgrade?

• No.

How was this patch tested?

• CI tests the Docker Image for the master branch. Add unit tests for information_schema database linghengqian/hive-server2-jdbc-driver#22 tests the Docker Image for Hive 4.0.1 .

[dengzhhu653]

Hi @linghengqian, Thank you for the contribution! could we add the details to the quickstart as well? the link: https://github.com/apache/hive-site/blob/main/content/Development/quickStart.md

[linghengqian]

• I found a potential drawback of the current PR is that due to the use of remote HMS, the startup of HS2 is particularly long and requires at least 2 minutes of waiting. According to my test results at Add unit tests for information_schema database with custom tables linghengqian/hive-server2-jdbc-driver#23, should I adjust compose.yaml to the following?

services:
  some-postgres:
    image: postgres:17.2-bookworm
    environment:
      POSTGRES_PASSWORD: "example"
  hiveserver2-standalone:
    image: apache/hive:4.0.1
    depends_on:
      - some-postgres
    environment:
      SERVICE_NAME: hiveserver2
      DB_DRIVER: postgres
      SERVICE_OPTS: >-
        -Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driver
        -Djavax.jdo.option.ConnectionURL=jdbc:postgresql://some-postgres:5432/postgres
        -Djavax.jdo.option.ConnectionUserName=postgres
        -Djavax.jdo.option.ConnectionPassword=example
    volumes:
      - ~/.m2/repository/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar:/opt/hive/lib/postgres.jar

mvn dependency:get -Dartifact=org.postgresql:postgresql:42.7.5
docker compose up -d
docker compose exec hiveserver2-standalone /bin/bash
/opt/hive/bin/schematool -initSchema -dbType hive -metaDbType postgres -url jdbc:hive2://localhost:10000/default
exit

[dengzhhu653]

• I found a potential drawback of the current PR is that due to the use of remote HMS, the startup of HS2 is particularly long and requires at least 2 minutes of waiting. According to my test results at Add unit tests for information_schema database with custom tables linghengqian/hive-server2-jdbc-driver#23, should I adjust compose.yaml to the following?

services:
  some-postgres:
    image: postgres:17.2-bookworm
    environment:
      POSTGRES_PASSWORD: "example"
  hiveserver2-standalone:
    image: apache/hive:4.0.1
    depends_on:
      - some-postgres
    environment:
      SERVICE_NAME: hiveserver2
      DB_DRIVER: postgres
      SERVICE_OPTS: >-
        -Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driver
        -Djavax.jdo.option.ConnectionURL=jdbc:postgresql://some-postgres:5432/postgres
        -Djavax.jdo.option.ConnectionUserName=postgres
        -Djavax.jdo.option.ConnectionPassword=example
    volumes:
      - ~/.m2/repository/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar:/opt/hive/lib/postgres.jar

mvn dependency:get -Dartifact=org.postgresql:postgresql:42.7.5
docker compose up -d
docker compose exec hiveserver2-standalone /bin/bash
/opt/hive/bin/schematool -initSchema -dbType hive -metaDbType postgres -url jdbc:hive2://localhost:10000/default
exit

It also works

[linghengqian]

It also works

• My understanding is that I should document both ways of writing compose.yaml.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[dengzhhu653]

This PR doesn't change any code, so I merge this without a green build to free the resources.