GA release v1.1.7

gh-ost v1.1.7 includes performance improvements, new features, and several bug fixes 👻
Full Changelog: v1.1.6...v1.1.7

What's Changed

• Add cpu-profile interactive command by @timvaillancourt in #1358
  • Use the cpu-profile command to get a base64-encoded runtime/pprof report
• Add --mysql-wait-timeout flag by @timvaillancourt in #1401 and #1406
  • --mysql-wait-timeout sets the wait_timeout at cutover time
• Support to config charset and collation by @shaohk in #1368
  • --charset sets the character set on gh-ost's MySQL connections
• Use query builders for DML queries, reducing time and allocations by @danieljoos in #1459
  • improves throughput 5-10%
• Avoid causing deadlocks when copying rows on busy tables by @arthurschreiber in #1455
  • adds FOR SHARE NO WAIT on the copy row queries for MySQL 8 to avoid deadlocks
• Use multiStatement to apply DML by @meiji163 in #1462
  • batches DML queries into multi-statements to reduce network overhead
• Use testcontainers to for unit tests. by @arthurschreiber in #1458 and #1460
• Use last second copy rate to project ETA by @morgo in #1231
• Docker setup for localtests by @meiji163 in #1479
• Ensure replication running after replication restart by @andyedison in #1422
• Force index in unique key range queries by @timvaillancourt in #1237
• Fix: ensure -attempt-instant-ddl respects -execute flag by @timvaillancourt in #1440
• Fix: use cut-over-lock-timeout for instant DDL by @artemvovk in #1468
• Fix: onSuccess hook to run after INSTANT by @artemvovk in #1463
• Fix: successive failures error in streamer by @timvaillancourt in #1371
• Fix: binlog end log pos over lost data by @shaohk in #1367
• Fix: generated column in unique key error by @meiji163 in #1461
• Fix: tokens in inspector logging by @timvaillancourt in #1226
• Fix: Print status after escaping % character by @icyflame in #1374

New Contributors

• @testwill made their first contribution in #1331
• @icyflame made their first contribution in #1374
• @sentinel made their first contribution in #1213
• @dependabot made their first contribution in #1391
• @dontstopbelieveing made their first contribution in #1388
• @mountcount made their first contribution in #1405
• @overallteach made their first contribution in #1417
• @danieljoos made their first contribution in #1459
• @artemvovk made their first contribution in #1463
• @whhe made their first contribution in #1472
• @andyedison made their first contribution in #1422

Thanks to all the contributors!

$ sha256sum *
5bb2ed9ece5db42ae747b28585572afd9d123a45e3e916f4d93601dbf5f1d430  gh-ost
9e7c91d07ccae51c653252b8c58c148032f3785223bfa8e531eba81aa912b71a  gh-ost-1.1.7-1.x86_64.rpm
bf99e7df791e08642fbfbfefc8bd63cad55c563cd15c0741c996bb11b0f0d392  gh-ost_1.1.7_amd64.deb
71e6277957e52efb7c050f2ef0fac3c4f71a1045c9af809cf10a58b69959faae  gh-ost-binary-linux-amd64-20241219160321.tar.gz
1afffa95bb3e1f62d47d82b517e58ce9a8f81d80c9369ec798274359a2b9b7c8  gh-ost-binary-linux-arm64-20241219160321.tar.gz
dacfa1b516d4558d8db90c73ffac749733ea64d355cb8a53f6ea8db47769fe2b  gh-ost-binary-osx-amd64-20241219160321.tar.gz
769d8290245d2e7d9ccc1c7c8256617c2f757a9810bba4a539581c43d3ba3731  gh-ost-binary-osx-arm64-20241219160321.tar.gz

GA release v1.1.6

Changes since 1.1.5: v1.1.5...v1.1.6

gh-ost v1.1.6 is based on the master branch, which is now considered stable.

What's Changed

• Attempt INSTANT DDL first for MySQL 8.0 by @morgo in #1201
  • disabled by default; enable with the flag --attempt-instant-ddl
• Add rocksdb as transactional engine by @wangzihuacool in #1190
  • RocksDB support is experimental. Enable with --storage-engine=rocksdb.
• Allow zero in dates by @timvaillancourt in #1161
  • enable with the flag --allow-zero-in-date
• Set binlogsyncer max reconnect attempts by @shaohk in #1279
  • set with the flag --binlogsyncer-max-reconnect-attempts
• Set a transaction isolation level for MySQL connections by @timvaillancourt in #1156
  • gh-ost will always use REPEATABLE_READ for InnoDB and READ_COMMITED for RocksDB
• Print status to migrationContext logger by @SocalNick in #1194
• Fix: parse binlog timestamp in UTC by @meiji163 in #1322
• Fix: Convert column value in binlog events to bytes instead of utf8 encoded unicode by @wangzihuacool in #1158
• Fix: because lock is not release, drop cutover sentry table is hanged by @lmtwga in #1180
• Default to go1.17.11 by @timvaillancourt in #1136
• Improve applier .ReadMigrationRangeValues() func accuracy by @timvaillancourt in #1164
• Add basic tests by @timvaillancourt #1165, #1166, #1168, #1179
• Upgrade go standard library for security patches by @meiji163 in #1332

Deprecation Notice

gh-ost officially supports MySQL 5.7 and newer.

• Remove MySQL 5.5 version (end-of-life Dec 2018) by @timvaillancourt in #1097
• Remove MySQL 5.6 version (end-of-life Feb 2021) by @dm-2 in #1101

Thanks to all contributors!

---

Edit Dec 7, 2023: Binaries updated to be statically linked

$ sha256sum *
0058c44e1a3ddba6fb6ac0b3132de5b266d57fdd059b756730d562307ffada6d  gh-ost
5e389eeeaf3fded040e61f5be6f8ab282af866f33a94ba7d34fccf9e52a8ec44  gh-ost-1.1.6-1.x86_64.rpm
5d15547f207e72591fd3a55c9cbea275396880a65a290742287a1ac84d0f4977  gh-ost-binary-linux-amd64-20231207144046.tar.gz
12f9d91a77774e85073fdea6bfb26f457424bf65b12043cb330e288231aa3465  gh-ost-binary-linux-arm64-20231207144046.tar.gz
dbb4005dee04ae441ef5aba0c4d35052374ad852c87637f374b3990f36339cdb  gh-ost-binary-osx-amd64-20231207144046.tar.gz
48fc179f7f91efb317c13c16bdc0d817bff0a4141b2732b40a7cbbe6ff211c20  gh-ost-binary-osx-arm64-20231207144046.tar.gz
f6ef9c83db73079c0c8395a85520b492551b9331c6063ce2f8996080aee51a3e  gh-ost_1.1.6_amd64.deb

GA release v1.1.5

Changes since 1.1.4: v1.1.4...v1.1.5

gh-ost v1.1.5 is based on v1.1.4, plus commits cherry-picked from master branch.

Notable:

• Cancel any row count queries before attempting to cut over, #846, thanks @ajm188
• Add context/timeout to HTTP throttle check, #1131, thanks @timvaillancourt
  • Throttler will now throttle migration on HTTP timeouts to throttler source. This may cause a change in behaviour in rare cases (more throttling)
  • New command line flags:
    1. --throttle-http-interval-millis - Number of milliseconds to wait between HTTP throttle checks (default 100)
    2. --throttle-http-timeout-millis - Number of milliseconds to use as an HTTP throttle check timeout (default 1000)
  • HTTP throttler calls now use gh-ost/VERSION-HERE as a User-Agent header
• Use switch statements for readability, simplify .NewGoMySQLReader(), #1135, thanks @timvaillancourt
• Ensure mysql rows responses are closed, #1132, thanks @timvaillancourt
• compound pk tests, #387, thanks @shlomi-noach
• Fix integer divide by zero panic in migrator, #1140, thanks @timvaillancourt for the PR and @ethankhall for reporting
• fix: lost data in mysql two-phase commit, #1141, thanks @shaohk
• Add golangci-lint CI action, fix gosimple, govet + unused lint errors, #1127, thanks @timvaillancourt
• Check RowsAffected when applying DML events to get more accurate statistics, #844, thanks @ajm188

Deprecation Notice:

• --replication-lag-query command line flag will be deprecated in gh-ost release 1.1.7

This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

Thank you to all contributors!

Updated 20th July 2022: RPM and deb packages have been updated to correctly include x86-64 binaries instead of arm64 binaries (thanks @Cartman75 for the issue report!)

$ sha256sum *
6a9dff3594b60a26422f7192ff7a79a85684b477d453431bedf025bc4023c389  gh-ost
236a1b9fb8fbb38bb6e27159377bd5af276ba09c3382724b5d88e9da6424dcbd  gh-ost-1.1.5-1.x86_64.rpm
791ab7fe0e8fe9e754480f4decdba8380c87d5ce70648f8a549060d81f889116  gh-ost-binary-linux-amd64-20220707162303.tar.gz
48f53683e3c6bf52957ff985c8ccc2c09f8f2484b44da4346f659e019ef69be9  gh-ost-binary-linux-arm64-20220707162303.tar.gz
1b0d992d03eeba7aa919ca9b40019b92af319dcb3668e0a447db21b35016c5ee  gh-ost-binary-osx-amd64-20220707162303.tar.gz
16fa1e09d2981e8a9312961337e41693a2e1396fe11abfd5b6a60781f3433795  gh-ost-binary-osx-arm64-20220707162303.tar.gz
2574da3ffe60f9a662366ca4f6a46edd90b8885acee8047ba0b11544561eb28d  gh-ost_1.1.5_amd64.deb

GA release v1.1.4

Changes since 1.1.3: v1.1.3...v1.1.4

gh-ost v1.1.4 is based on v1.1.3, plus commits cherry-picked from master branch.

Notable:

• Add flag to customize the interval which the onStatus hook is called, #1083, thanks @tknodell-recurly
• Add docs for hooks-status-interval, #1090
• Reduce the minimal chunk size from 100 to 10, #1087, thanks @arthurschreiber

This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

Thank you to all contributors!

Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

Updated 11th Mar 2022: Added macOS arm64 binary to support M1 Macs (thanks @jecepeda for adding support in #1108!)

$ sha256sum *
da2b2a943aaf18fb849b52e19b0828c9878b180546d1adb31ad03b3cf7c50d53  gh-ost
d626f6ae582ca15c58a387c819c410788a78f64dd7ac7ec946df07f0bf66f7a0  gh-ost-1.1.4-1.x86_64.rpm
ccc055a9fa2ca080b99aeb58a6aacf557582c317b8f36ee13c551d87cd5eeb30  gh-ost-binary-linux-20220225143506.tar.gz
9feebe056914335d23127db10de8f20631e0df448b1382f30366a462ab5d5473  gh-ost-binary-osx-20220225143506.tar.gz
66619ad49fb7a2a13edfe08aee110a554f7174b44453451fee2e532661f1f254  gh-ost-binary-osx-arm64-20220311164158.tar.gz
1dead2c07b73a8730ec4b923433be143dcb4dc18719e2ad640962e389168a22d  gh-ost_1.1.4_amd64.deb

GA release v1.1.3

Changes since 1.1.2: v1.1.2...v1.1.3

gh-ost v1.1.3 is comprised of v1.1.2 with a fix for security vulnerability GHSA-rrp4-2xx3-mv29, plus low-risk documentation updates cherry-picked from master branch.

This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

Notable:

• Security vulnerability: Improper Input Validation of -database Parameter, GHSA-rrp4-2xx3-mv29, thanks @dwisiswant0
• doc/triggerless-design.md: typo, #1011, thanks @eddroid
• Document the flag critical-load-hibernate-seconds, improve grammar of flags documentation, #1066, thanks @EagleEyeJohn
• Update hook environment variable docs, #1076
• Update build script to generate releases, #1078
• Remove CGO dependency in go-mysql in Gh-ost v1.1.3, #1079

Thank you to all contributors!

Updated 8th Feb 2022: Binaries have been re-uploaded with the correct version number; previously they were incorrectly labelled as v1.1.2 instead of v1.1.3. (thanks @dharnett for notifying us!)

Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

$ sha256sum *
10255fa8690001d7eb37f51c6d41aba0e55ee9ef4c2b949ca4c0604011b74d71  gh-ost
496ac03a36738e752ad98668a3bb97eca913039431fdff80732fbb4acfac8991  gh-ost-1.1.3-1.x86_64.rpm
c432e7d711268117bc8c13cebcf3e3cf3c986557a205cc041e0ea63937700ef5  gh-ost-binary-linux-20220225143057.tar.gz
e55b3c28c655eab28617a47eefa88fb6a508af506652bafdd226293968bebd8e  gh-ost-binary-osx-20220225143057.tar.gz
116db1f8f5e4bff6a8903f057dad07e14ad36eea0f3b462a16342b6d3b447d1a  gh-ost_1.1.3_amd64.deb

GA release v1.1.2

Changes since 1.1.1: v1.1.1...v1.1.2

Notable:

• Copy auto increment, #911, thanks @shlomi-noach
• Cut-over should wait for heartbeat lag to be low enough to succeed, #921, thanks @ccoffey
• All MySQL DBs limited to max 3 concurrent/idle connections, #931, thanks @shlomi-noach
• Report GH_OST_ETA_SECONDS ETA as part of migration context, #936, thanks @shlomi-noach
• Remove build_id files from rpm, #954, thanks @yakirgb
• Fix bug with -exact-rowcount on MySQL 8.0.2+, #962, thanks @Fanduzi
• Support enum to varchar, #963, thanks @shlomi-noach
• Set the ServerName for TLS configuration, #988, thanks @dbussink
• Use Golang 1.16 and temporarily disable go modules in builds, #966
• Use matrix build for replica test CI, #968

Thank you to all contributors!

$ sha256sum *
342a7f4760d51dc05154dd1e6663f701bb024868a7f9b0804e8bf38afe67e0ff  gh-ost
255dbaed5de7c20b1f11c82ed0275e0ecfaa8f8afa4f38fd36705cfac4b6207f  gh-ost-1.1.2-1.x86_64.rpm
98ebff834bf930a198d3506cda0ee1a6a3ea67f09e9112ca8a2534ef4f74288c  gh-ost-binary-linux-20210617134741.tar.gz
0b000805894ae5cc6b5ff0aacb67e4273328354f5e559a9ad39b1e7cc617b316  gh-ost-binary-osx-20210617155835.tar.gz
afaf95ea239b5e473080d0b58592e6449cfc48d936396a8540722c4772741893  gh-ost_1.1.2_amd64.deb

GA release v1.1.1

Changes since 1.1.0: v1.1.0...v1.1.1

Notable:

• Use sync.Once to avoid dropping cutover sentry table twice, #755, thanks @MOON-CLJ
• Parsing IPV6 addresses, #640, thanks @shlomi-noach
• Support Azure Database for MySQL, #807, thanks @elemount
• Use Golang 1.15 and remove Debian Jessie (end-of-life) tarball build, #906
• Improved connection type logging, #515, thanks @akshaychhajed
• Fix: Data loss on tables with binary primary key, #915, thanks @cenkore
• Add applier and inspector interactive commands to socket/TCP server, #951

Thank you to all contributors!

$ sha256sum *
8e802e8685d4b7d757ae47513b918d6de1d0402511cc3fadb2d961713beb6d89  gh-ost
6b49bbcf1a985d96fa98954d6aba5b2e8fa9f8b0ac9aee2c10bd47755f986254  gh-ost-1.1.1-1.x86_64.rpm
1188bb2d4539d25fc8a4b8a96cccb9b56a217124a9d4c8b9faab4534acd9258b  gh-ost-binary-linux-20210503150942.tar.gz
1ef5585f926af0d93af53007b563c86a34d52bd4c05e1be5c6afaf8064199d3e  gh-ost-binary-osx-20210503171035.tar.gz
02d39afd96e0b5d69baf6a7259871b664445eaa811ccb6c3928376793b56f997  gh-ost_1.1.1_amd64.deb

GA release v1.1.0

Changes since 1.0.49: v1.0.49...v1.1.0

Notable:

• Support a complete ALTER TABLE statement in --alter, #865 / #878, thanks @shlomi-noach
• Support --mysql-timeout flag, #824, thanks @shlomi-noach
• Add a check to rows.Err after processing all rows, #835, thanks @ajm188
• Update go-sql-driver to latest, #823, thanks @shlomi-noach
• Update build to Golang 1.14, #861 / #876, thanks @shlomi-noach
• Add error checking for an err variable that was left unchecked, #810, thanks @yaserazfar
• Implement a logging interface, #789 / #864, thanks @abeyum and @jfudally
• Throttle on HTTP error to throttling API, #833, thanks @jfudally
• Additional testing:
  • Add latin1 tests with TEXT columns, #291 / #455, thanks @shlomi-noach
  • Add BINARY / VARBINARY tests, #655, thanks @shlomi-noach
  • Add NULL-able INT tests, #692, thanks @shlomi-noach

Thank you to all contributors!

$ sha256sum *
e6de44ed76cd7834d7dac5ee4a53459ce18e16fbb482e9ff5d898f7d83444632  gh-ost-1.1.0-1.x86_64.rpm
2c29184d5b7f1340999680c06ae6796639f195d00907214a9baddc9ca9391ca9  gh-ost-binary-linux-20200828140552.tar.gz
c1dc2716dd5b843b4ef11b3fa601885f079363ca5545799bdac7037fd132e723  gh-ost-binary-osx-20200828160625.tar.gz
6ca0c5f0a79e51035e380a00350a535e04381b29ac9bbb847acf31d3b8985390  gh-ost_1.1.0_amd64.deb

GA release v1.0.49

Changes since 1.0.48: v1.0.48...v1.0.49

Notable:

• --force-named-panic (on panic command, force table name), named throttle, named no-throttle, #695
• add --skip-strict-mode option, #712, thank @babinomec
• Improve SSL support #716, thanks @RafeKettler
• context, status and hooks: add progressPct and CurrentLag, #788
• Always use NO_AUTO_VALUE_ON_ZERO #723
• tests: test on MySQL 8.0
• tests: Bump default-retries to 3 #754 , thanks @zmoazeni
• Support and require go1.12
• doc updates/fixes, thanks @mohitm15, Tobias Johansson, @jchambers

$ sha256sum *
628ad01f4b327ee81f01aee1e4e5d2a966ed681fa391d064424270dbd7b0b0b1  gh-ost-1.0.49-1.x86_64.rpm
a1d7f72e1119bb8a939204a56acbee09eb52c769183a4649e56d6b3b524cb774  gh-ost-binary-linux-20200209110835.tar.gz
667f8800654410749b1af6053f26109769992adb813eb9edaa154d537cc86ef3  gh-ost_1.0.49_amd64.deb
95b3ed74f70ffea9f11f4af06fb6e312f0a857bf4fd3c3ad2da31fc7b3359dd3  gh-ost-binary-osx-20200209114711.tar.gz

GA release v1.0.48

Changes since 1.0.47: v1.0.47...v1.0.48

• Update to latest go-mysql library, which solved multiple issues
• Fix inspector column types #661, thanks @esnunes
• inspect: remove redundant join conditions #699, thanks @twotwotwo
• maintainer email address in .deb package
• added: --hooks-hint-owner, --hooks-hint-token #703
• Initial SSL Connection Support #705, thanks @brandonbodnar-wk
• Fixed DECIMAL datatype #684 (the trigger for updating go-mysql)
• more... thank you to all contributors!

$ sha256sum *
cf1f0999c81f194472b9d3f154c21e0848bd1d61697ce1ca9123368b31d718b2  gh-ost-1.0.48-1.x86_64.rpm
8d0a518064c5803e1d5e8676031e5a5b5b180e67700d476e19eaf68e4a4a583a  gh-ost-binary-linux-20190214020851.tar.gz
f57a1361315adba33b51ecd9bf8863722385dbcff741f0a4d467087aaded9bc4  gh-ost-binary-osx-20190214120940.tar.gz
67bf74f1af6f968a59a2725398ccb5167d97e9b0cd1ab00c867425bf86a0d124  gh-ost_1.0.48_amd64.deb