[SPARK-51740][SQL] Allow get_json_object to consider leading spaces in paths

[harshmotw-db]

What changes were proposed in this pull request?

This PR allows the get_json_object to consider leading whitespaces/tabs in paths. Earlier, these spaces would just get ignored as follows:

scala> spark.sql("""select get_json_object('{" a b c ": " leading space present", "a b c ": "leading space absent"}', "$[' a b c ']")""").show()
+------------------------------------------------------------------------------------------------------+
|get_json_object({" a b c ": " leading space present", "a b c ": "leading space absent"}, $[' a b c '])|
+------------------------------------------------------------------------------------------------------+
|                                                                                  leading space absent|
+------------------------------------------------------------------------------------------------------+

Why are the changes needed?

JSON keys could have leading spaces. However, there is currently no way to extract these keys using get_json_object. Also, it is possible for users to extract the wrong key if there is another nearly identical key without the leading whitespaces.

Does this PR introduce any user-facing change?

Yes. This is a behavioral change guarded by a feature flag.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[gene-db]

@harshmotw-db Thanks for finding and fixing this bug!

[gene-db]

What is a scenario where we would want to skip leading whitespace? Wouldn't that just be incorrect behavior?

[harshmotw-db]

Yeah, but considering this is a very old expression and this is a behavioral difference, some workloads might be based on the old behavior. I could remove the flag if accommodating it is not important.