Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

Closed

Conversation

sercanCyberVision
Copy link
Contributor

@sercanCyberVision sercanCyberVision commented May 5, 2023

What changes were proposed in this pull request?

@JsonProperty annotation has been added to model classes for mapping the properties with their corresponding getter/setter methods. This will not effect Ranger's functionality directly, but it will provide consistency in case there is Jackson jar conflict (or when/if Jackson is upgraded to version-2).
Please see the Jira for more detailed analysis of the issue https://issues.apache.org/jira/browse/RANGER-4225

Basically, some of the model classes do not comply with JavaBean naming conventions, therefore it is possible that Ranger may hit this issue https://stackoverflow.com/questions/30205006/why-does-jackson-2-not-recognize-the-first-capital-letter-if-the-leading-camel-c

As a reference of the issue, please see http://futuretask.blogspot.com/2005/01/java-tip-6-dont-capitalize-first-two.html

How was this patch tested?

Built the project and made sure that deserialized responses on UI side have correct property names even though Jackson-2 jars are in the classpath, hence the corresponding UI components work as expected.

@sercanCyberVision sercanCyberVision changed the title RANGER-4225: Possible Jackson serialization due to not complain to Java bean standards RANGER-4225: Possible Jackson serialization issue due to not comply to Java bean standards May 5, 2023
@sercanCyberVision sercanCyberVision changed the title RANGER-4225: Possible Jackson serialization issue due to not comply to Java bean standards RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards May 5, 2023
@sercanCyberVision sercanCyberVision force-pushed the RANGER-4225 branch 2 times, most recently from 602a07d to 77fac4d Compare May 5, 2023 12:08
@sercanCyberVision
Copy link
Contributor Author

@kumaab could you please take a look at this?

@sercanCyberVision
Copy link
Contributor Author

@mneethiraj could you please take a look at this? I hit this issue earlier, and the solution here can save some grief for the end-users for the future.

@bhavikpatel9977
Copy link
Contributor

LGTM.

Kindly resolve the conflict.

@sercanCyberVision
Copy link
Contributor Author

@bhavikpatel9977, thank you for reviewing the PR. I have resolved the conflict and quickly tested it again.
So, two actions have been taken here:

  1. org.codehaus.jackson has been replaced with com.fasterxml.jackson.
  2. @JsonProperty has been added to view classes to help jackson recognize java bean properties.

@bhavikpatel9977
Copy link
Contributor

@mneethiraj Can you kindly review once

Copy link
Contributor

@mneethiraj mneethiraj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sercanCyberVision - thank you for taking up this critical update to switch to fasterxml.jackson. In addition to the couple of review comments, can you please verify that the final packaging does not include codehaus jackson library? It seems Ranger admin continues to use codehaus jackson library to serialize REST API responses. For example, the response includes fields having null and empty value.

pom.xml Outdated
@@ -91,7 +91,7 @@
<cglib.version>2.2.0-b23</cglib.version>
<checkstyle.plugin.version>3.1.0</checkstyle.plugin.version>
<checkstyle.version>8.29</checkstyle.version>
<codehaus.jackson.version>1.9.13</codehaus.jackson.version>
<com.fasterxml.jackson.version>2.17.0</com.fasterxml.jackson.version> <!--here-->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using existing property fasterxml.jackson.version (line # 215) instead of introducing new one com.fasterxml.jackson.version. Also, update the version in #215 and #216 from 2.14.0 to 2.17.0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @mneethiraj for your review.

I have implemented above requested change and pushed it as a separate commit for now as it is easier for you to see the difference, the commit is cccbabf.

If needed, when we reach the final state, I can squash the commits or change the names.

import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;


@JsonAutoDetect(getterVisibility = Visibility.NONE, setterVisibility = Visibility.NONE, fieldVisibility = Visibility.ANY)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the move to fasterxml.jackson 2.x, consider replacing deprecated include = JsonSerialize.Inclusion.NON_EMPTY, with @JsonInclude(JsonInclude.Include.NON_EMPTY)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As stated above, the changes have been implemented with cccbabf commit.

@sercanCyberVision
Copy link
Contributor Author

@sercanCyberVision - thank you for taking up this critical update to switch to fasterxml.jackson. In addition to the couple of review comments, can you please verify that the final packaging does not include codehaus jackson library? It seems Ranger admin continues to use codehaus jackson library to serialize REST API responses. For example, the response includes fields having null and empty value.

Thank you @mneethiraj for detailed review. You are right; codehaus.jackson was present in the admin library. I have checked the dependency tree and found that they were transitive. I have fixed the issue with this separate commit: 7eb06a5. If needed, we can squash it once the review is completed. I kept it separate so you can see the changes easily. The latest state is as follows:

ranger@ranger:/opt/ranger/ranger-3.0.0-SNAPSHOT-admin$ find . -name "*jackson*"
./ews/lib/jackson-core-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-smile-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-cbor-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-databind-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-core-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-module-jaxb-annotations-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-yaml-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-annotations-2.17.0.jar
./ews/webapp/WEB-INF/classes/ranger-plugins/knox/jackson-databind-2.17.0.jar
./ews/webapp/WEB-INF/classes/ranger-plugins/knox/jackson-core-2.17.0.jar
ranger@ranger:/opt/ranger/ranger-3.0.0-SNAPSHOT-admin$

I have one concern: when I deploy the project locally with Docker, I see the following issue on some endpoints after removing transitive dependencies:
image

However, I'm not sure if this is a real issue because when I saw the issue previously, others did not see it with the same codebase. For example, when I mentioned the issue here, in the next comment, the same codebase worked on another machine. Could you please check on your end?

@mneethiraj
Copy link
Contributor

@sercanCyberVision - the error is likely due to missing Json provider in Ranger admin server. A provider should be implemented in Ranger servers like admin/KMS. In addition, Ranger client libraries should be updated to replace calls to ClientResponse.getEntity(cls) with JsonUtilsV2.jsonToObj(jsonStr, cls). I added these fixes on top of the changes in this PR, and created a .patch file that applies on the latest master branch. I will attach this file in RANGER-4225 shortly. Can you please review/verify these updates and create a new PR? Thank you again for driving this critical update.

@sercanCyberVision
Copy link
Contributor Author

sercanCyberVision commented Jul 11, 2024

@mneethiraj, thank you for providing the fix. I have modified this PR for the complete solution.

At the moment, I see that everything is working fine on my end. However, I still see some ClientResponse.getEntity(cls):
https://github.com/sercanCyberVision/ranger/blob/f84dd021c4c77666e1995535449f10b272ccad3b/intg/src/main/java/org/apache/ranger/RangerClient.java#L567
or
https://github.com/sercanCyberVision/ranger/blob/71809108fd106b664b6f9d53e0efd86d4c5cd039/plugin-yarn/src/main/java/org/apache/ranger/services/yarn/client/YarnClient.java#L148
or
https://github.com/sercanCyberVision/ranger/blob/f84dd021c4c77666e1995535449f10b272ccad3b/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L736
There are some more. Are they left as they are on purpose? Or, should we replace them with JsonUtilsV2.jsonToObj(jsonStr, cls)

Checks on UI:

home

settings

policy

@mneethiraj
Copy link
Contributor

@sercanCyberVision - good catch on use of readEntity() in RangerClient and RangerAdminRESTClient. I updated these classes to deserialize the response using JsonUtilV2 methods. YarnClient uses readEntity() to read the response as String; such usage doesn't require using Jackson Json conversion.

One more issue I noticed was that the patch replaced NOT_NULL to NOT_EMPTY in several classes; this caused errors in UI as the response from server didn't include empty collection values - [] or {}.

I updated the patch to address above issues and will attach in RANGER-4225.

@mneethiraj
Copy link
Contributor

The final patch is in the review board at https://reviews.apache.org/r/75082/. The patch is merged in master and ranger-2.5 branches.

@mneethiraj mneethiraj closed this Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants