[HUDI-7430] Fix empty schema issue for compactor #10718
Conversation
| try (SparkRDDWriteClient client = | ||
| UtilHelpers.createHoodieClient(jsc, cfg.basePath, "", cfg.parallelism, Option.of(cfg.strategyClassName), props)) { | ||
| UtilHelpers.createHoodieClient(jsc, cfg.basePath, getSchema(), cfg.parallelism, Option.of(cfg.strategyClassName), props)) { |
There was a problem hiding this comment.
Does this fail with empty schema string? If so, could you construct a test case that fails without the fix and passes with the fix?
There was a problem hiding this comment.
@jonvex @yihua, to trigger the failures, I probably need to write a functional test for that since I need to use "hoodie.schema.add.field.ids = true". CC @nsivabalan
There was a problem hiding this comment.
@jonvex, do you know how to trigger the failures by setting "hoodie.schema.add.field.ids = true"?
There was a problem hiding this comment.
As discussed offline, a test implementation of HoodieAvroWriteSupport needs to be added, that does schema parsing to reproduce the failure. This is related to the custom write support as a Hudi extension in XTable: https://github.com/apache/incubator-xtable/blob/main/hudi-support/extensions/README.md.
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieCompactor.java
Show resolved
Hide resolved
linliu-code
force-pushed
the
HUDI-7430_fix_empty_schema
branch
2 times, most recently
from
892125e to
82ab336
Compare
linliu-code
force-pushed
the
HUDI-7430_fix_empty_schema
branch
from
82ab336 to
d6fccb7
Compare
This is to avoid Mockito failure.
|
we can close it then. |
Change Logs
The input schema string is empty.
We try to get it form schema file or instants.
Impact
Fixed the bug.
Risk level (write none, low medium or high below)
None.
Contributor's checklist