Andrew7234/consensus tx body

[Andrew7234]

Task

api: Expose consensus tx body as JSON

We currently expose it as base64(cbor(actual_contents))

See e.g. https://nexus.oasis.io/v1/consensus/transactions/2a59b2765df414e0ac56af24f1a4769b9f559d92ace6c65eafcdff98e76a837f -- it's a staking.Transfer tx so we'd expect to see amount more readily exposed.

Reported by @buberdds

This PR

We currently expose the consensus tx body as a base64(cbor-encode(body)). This PR updates the consensus analyzer to now store the tx body as a json object, and return it in the same manner via the api.

This PR includes a breaking API change by changing the type of (consensus)Transaction.body from string -> object

The current migration is a breaking migration and drops the existing column and re-adds it as a json type since postgres cannot automatically convert existing bytea values to json. Existing consensus tx body data will be lost, and ideally this change should only be applied prior to a full reindex.

Alternatives considered:

• Migrate existing values to some sort of 'temporary' json object like

{ cborBody: "omJ0b1UAGFXafz9TY1hQE753oi4R3pU4/HpmYW1vdW50RAL68IA="}

This seems alright but I'm not sure that it's worth it compared to just having the existing consensus body be empty for past txs. Open to suggestions!

• Use a different column. This creates lots of tech debt (esp with the api/queries) and isn't worth it.

Note: In order to çbor-unmarshal the tx.Body into the proper structs, I had to re-add some Method variables to the vendored oasis-core types. See coreapi/v22.2.11/consensus/api/transaction/transaction.go. Methods unique to cobalt/eden were linked to the damask registeredMethods in their respective nodeapi convert.go files. All changes to the vendored damask oasis-core are simply re-adding code that was originally omitted for brevity.

Note: There is a SetEpoch method defined in a cometbft(formerly tendermint) app but this is only used for testing, so it was omitted.

Testing:

• Currently just the e2e_regression tests.
• todo: some unit tests for the consensus analyzer as well

Sample output:

 » curl "http://localhost:8008/v1/consensus/transactions?method=staking.AddEscrow" | jq             andrewlow@Beleriand
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1849  100  1849    0     0   157k      0 --:--:-- --:--:-- --:--:--  300k
{
  "is_total_count_clipped": false,
  "total_count": 5,
  "transactions": [
    {
      "block": 8049363,
      "body": {
        "account": "oasis1qq3xrq0urs8qcffhvmhfhz4p0mu7ewc8rscnlwxe",
        "amount": "184700000000"
      },
      "fee": "0",
      "hash": "fdabfae72be8dd582e7250323a0269bea59018e976521df690f904e8c1e97552",
      "index": 0,
      "method": "staking.AddEscrow",
      "nonce": 2,
      "sender": "oasis1qp0nyn6fcl6dm4a6h7cvsfketcgxl8838s22xqvt",
      "success": true,
      "timestamp": "2022-04-11T04:46:01-07:00"
    },

For anyone curious about our consensus tx distribution:

oasisindexer=> select *, count(*) as cnt from (select method from chain.transactions) as x group by 1 order by cnt desc;
              method               |    cnt
-----------------------------------+-----------
 roothash.ExecutorCommit           | 116611434
 registry.RegisterNode             |   9465357
 beacon.VRFProve                   |   3574383
 staking.Transfer                  |   1586824
 consensus.Meta                    |   1442163
 roothash.ExecutorProposerTimeout  |    226040
 staking.AddEscrow                 |    188549
 beacon.PVSSCommit                 |    167048
 beacon.PVSSReveal                 |    167011
 staking.Allow                     |    126821
 staking.ReclaimEscrow             |     88278
 registry.ProveFreshness           |     84922
 keymanager.PublishEphemeralSecret |      2381
 registry.RegisterEntity           |       635
 staking.AmendCommissionSchedule   |       390
 governance.CastVote               |       327
 registry.RegisterRuntime          |        50
 keymanager.UpdatePolicy           |        10
 staking.Withdraw                  |         7
 governance.SubmitProposal         |         3
 registry.UnfreezeNode             |         2
 registry.DeregisterEntity         |         1
(22 rows)

[pro-wh]

ya good way to do it

[mitjat]

Thank you, Andy! This brings a ton of value to the consensus layer, IMO.

I do have a request for change that involves a good number of lines of code, but is mostly mechanical now that you've done the hard work of finding all the relevant method names across all the coreapi versions.

[Andrew7234]

unit tests incoming

[mitjat]

🙏 thank you for the refactor!

Thank you also for the upcoming tests. IMO unit tests are less valuable in this particular case because it's mostly about decoding undecipherable binary blobs into something readable. So it's hard to set up a unit test that's much more than a copy of (a segment of) an e2e test. Will be nice to test out Cobalt and the failure case though.