Skip to content

Commit

Permalink
[#26066] YSQL: pg_upgrade --check improvements
Browse files Browse the repository at this point in the history
Summary:
When `pg_upgrade --check` fails, it writes out the list of incompatible objects in files in the data directory. This is not ideal for users, since they will need to retrieve and analyze those files. It also makes the use of the check via `yugabyted`, `YBA` and `YBM` less user friendly. This change send logs those objects to the console output, in addition to the files.
Additionally, instead of exiting the check on the first error, the tool now lists all the errors. In the actual upgrade, it continues to exit on the first error like before.

- Cleaned up some print formatting errors with yb specific checks.
- Disabling the call to `check_new_cluster`. Since all the related checks (check_new_cluster_is_empty, check_databases_are_compatible, check_loadable_libraries, installed user, check_for_prepared_transactions and check_for_new_tablespace_dir) are not applicable for yb.
- Disabled `check_for_tables_with_oids` since it is not relevant in yb.
- Migrated all shell upgrade tests to c++.
- Added `TEST_SETUP_SUPER` macro that will call `SetUp` on the super class, and exit if the test has been marked as skipped.
- Moved `CheckUpgradeCompatibilityGuc` and `UsersAndRoles` to the new `ysql_major_upgrade_check-test.cc` file

Fixes #26066
Jira: DB-15392

Test Plan:
Pg15UpgradeTest.BasicTablespace
Pg15UpgradeTest.DroppedColumnTest
YsqlMajorUpgradeCheckTest

**Failed check:**

```
yugabyte=# ALTER DATABASE postgres WITH allow_connections FALSE;
ALTER DATABASE
yugabyte=# CREATE TABLE system_composite_test (id int primary key, authid pg_authid);
CREATE TABLE
yugabyte=# CREATE TABLE reg_check (a int, b regproc);
CREATE TABLE
yugabyte=# CREATE FUNCTION ident(integer)
yugabyte-# RETURNS integer
yugabyte-# AS $$
yugabyte$# BEGIN
yugabyte$#     RETURN $1;
yugabyte$# END;
yugabyte$# $$ LANGUAGE plpgsql;
CREATE FUNCTION
yugabyte=# CREATE OPERATOR !!! (LEFTARG = integer, PROCEDURE = ident);
CREATE OPERATOR
yugabyte=# CREATE AGGREGATE array_accum (ANYELEMENT)(sfunc = array_append, stype =
yugabyte(# ANYARRAY, initcond = '{}');
CREATE AGGREGATE
yugabyte=# CREATE TABLE sql_identifier_test (id int primary key, d information_schema.sql_identifier);
CREATE TABLE
```

```
$ ./build/latest/postgres/bin/pg_upgrade --check  -U yugabyte --old-socketdir /tmp/.yb.127.0.0.1:5433 --old-port 5433 --old-datadir ~/yugabyte-data/node-1/disk-1/pg_data_11
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking attributes of the 'yugabyte' user                  ok
Checking for all 3 system databases                         ok
Checking database connection settings                       fatal

postgres

All non-template0 databases must allow connections, i.e. their
pg_database.datallowconn must be true.  Your installation contains
non-template0 databases with their pg_database.datallowconn set to
false.  Consider allowing connection for all non-template0 databases
or drop the databases which do not allow connections.  A list of
databases with the problem is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/databases_with_datallowconn_false.txt

Checking for system-defined composite types in user tables  fatal

In database: yugabyte
  public.system_composite_test.authid

Your installation contains system-defined composite type(s) in user tables.
These type OIDs are not stable across PostgreSQL versions,
so this cluster cannot currently be upgraded.  You can
drop the problem columns and restart the upgrade.
A list of the problem columns is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/tables_using_composite.txt

Checking for reg* data types in user tables                 fatal

In database: yugabyte
  public.reg_check.b

Your installation contains one of the reg* data types in user tables.
These data types reference system OIDs that are not preserved by
pg_upgrade, so this cluster cannot currently be upgraded.  You can
drop the problem columns and restart the upgrade.
A list of the problem columns is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/tables_using_reg.txt

Checking for user-defined postfix operators                 fatal

In database: yugabyte
  (oid=16393) public.!!! (pg_catalog.int4, NONE)

Your installation contains user-defined postfix operators, which are not
supported anymore.  Consider dropping the postfix operators and replacing
them with prefix operators or function calls.
A list of user-defined postfix operators is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/postfix_ops.txt

Checking for incompatible polymorphic functions             fatal

In database: yugabyte
  aggregate: public.array_accum(anyelement)

Your installation contains user-defined objects that refer to internal
polymorphic functions with arguments of type "anyarray" or "anyelement".
These user-defined objects must be dropped before upgrading and restored
afterwards, changing them to refer to the new corresponding functions with
arguments of type "anycompatiblearray" and "anycompatible".
A list of the problematic objects is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/incompatible_polymorphics.txt

Checking for invalid "sql_identifier" user columns          fatal

In database: yugabyte
  public.sql_identifier_test.d

Your installation contains the "sql_identifier" data type in user tables.
The on-disk format for this data type has changed, so this
cluster cannot currently be upgraded.  You can
drop the problem columns and restart the upgrade.
A list of the problem columns is printed above and in the file:
    /Users/hsunder/yugabyte-data/node-1/disk-1/pg_data_11/pg_upgrade_output.d/20250218T134505.203/tables_using_sql_identifier.txt

Failure, exiting
```

**Successful check:**

```
yugabyte=# DROP TABLE tbl2;
DROP TABLE
yugabyte=# DROP TABLE system_composite_test;
DROP TABLE
yugabyte=# ALTER DATABASE postgres WITH allow_connections TRUE;
ALTER DATABASE
```

```
$ ./build/latest/postgres/bin/pg_upgrade --check  -U yugabyte --old-socketdir /tmp/.yb.127.0.0.1:5433 --old-port 5433 --old-datadir ~/yugabyte-data/node-1/disk-1/pg_data_11
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking attributes of the 'yugabyte' user                  ok
Checking for all 3 system databases                         ok
Checking database connection settings                       ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for user-defined postfix operators                 ok
Checking for incompatible polymorphic functions             ok
Checking for invalid "sql_identifier" user columns          ok

*Clusters are compatible*
```

Reviewers: telgersma, smishra

Reviewed By: telgersma

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D41932
  • Loading branch information
hari90 committed Feb 20, 2025
1 parent f307315 commit bad1b7a
Show file tree
Hide file tree
Showing 19 changed files with 483 additions and 887 deletions.
172 changes: 0 additions & 172 deletions pg15_tests/common_upgrade.sh

This file was deleted.

138 changes: 0 additions & 138 deletions pg15_tests/test_upgrade.sh

This file was deleted.

Loading

0 comments on commit bad1b7a

Please sign in to comment.