[MIN-2432] Enum schema evolution Python fork #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
antongrbin
changed the title
…terface # Note to reviewer * The baseline here is a fork, please review documentation on #1 and [tech doc section](https://docs.google.com/document/d/1p5LUSTWrVBcT9F2wXBgHBDT3OJyOm5BpXAyTpwwDVts/edit#heading=h.b8v2uff99l8v). # Motivation Reducing the dependencies that `json_format.py` is making is helping us create a more stable fork of that file in noom-contracts. See concrete problem [here](https://github.com/noom/noom-contracts/runs/6651503347?check_suite_focus=true), where `mypy` fails because it can't find type hints on the `google.protobuf.internal` library. See code comment above `TruncateToFourByteFloat` # Testing completed Ran python unit tests as described in test plan in #1.
# Motivation Tech doc: https://docs.google.com/document/d/1p5LUSTWrVBcT9F2wXBgHBDT3OJyOm5BpXAyTpwwDVts/edit# # Changes In this PR we actually fix the Python protobuf library not to break when unknown enum value is encountered if ignoreUnknownFields is true. The design principles for the change were: * make the change as small as possible, so it's easy to keep up to date with the upstream * if one element is unknown in an array of elements, other elements should still parse (see unit tests) Changes: * Change `_ConvertScalarFieldValue` so it throws a special exception when unknown enum string value is encountered. * Wrap every invocation of `_ConvertScalarFieldValue` to optionally ignore the new exception thrown. * Add unit tests To write this PR, I reached out for input on #python-club [on slack](https://noom.slack.com/archives/C020854DJDU/p1655740069429799) # Tested Executed the test plan from here: #1 to test this within the `protobuf` repo. Also tested as part of `noom-contracts` repo: https://github.com/noom/noom-contracts/pull/293
antongrbin
pushed a commit
that referenced
this pull request
Jan 10, 2024
PiperOrigin-RevId: 573311752
antongrbin
pushed a commit
that referenced
this pull request
Jan 10, 2024
Before, every charAt would emit (on android):
```
0x00002104 adrp x17, #+0x1000 (addr 0x3000)
0x00002108 ldr w17, [x17, protocolbuffers#20]
0x0000210c ldr x0, [x0, protocolbuffers#128]
0x00002110 ldr x0, [x0, protocolbuffers#328]
0x00002114 ldr lr, [x0, protocolbuffers#24]
0x00002118 blr lr <-- Call into String.charAt(int)
```
Now, it emits the inlined implementation of charAt (branch is for possibly compressed strings):
```
0x000020b4 ldur w16, [x4, #-8]
0x000020b8 tbnz w16, #0, #+0xc (addr 0x20c4)
0x000020bc ldrb w4, [x4, x0]
0x000020c0 b #+0x8 (addr 0x20c8)
0x000020c4 ldrh w4, [x4, x0, lsl #1]
```
PiperOrigin-RevId: 591147406
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DON'T MERGE THIS PR
This is PR is just a visual representation of our JSON enum fork. Learn more in the original Tech Doc.
Motivation
This PR helps us visualize code changes that we've made to protobuf library wrt JSON schema evolution.
Tech Doc:
https://docs.google.com/document/d/1p5LUSTWrVBcT9F2wXBgHBDT3OJyOm5BpXAyTpwwDVts/edit#
Setup
The baseline branch
json_unknown_enum_fork_baselineis pinned to a version of the upstream repository on the version specified by:https://github.com/noom/noom-contracts/blob/master/PROTOBUF_VERSION
This can be checked by running a command similar to
git diff json_unknown_enum_fork_baseline v3.18.1, which should always produce empty results.The head branch for this PR
json_unknown_enum_fork, shows all changes that we've made to the upstream repository.Changes
If you want to make changes to our fork, create a PR that merges to
json_unknown_enum_fork.Install
If you want to install or update the fork to our
noom-contractsrepo, run the following script:./lib/python/fetch_forked_json_format.sh, create a PR, review it and merge it.Testing in
protobufrepoWhen does the fork end?
Once protobuf team decides what to do with JSON enum schema evolution, we will end the fork and implement whatever they decide upstream. They will announce it here:
protocolbuffers#7392