How to add an object to the end of an array with unknown length

[estigma88]

Hi everyone, your help:

Request:

[
  {
    "id": "123",
    "stages": [
      {
        "id": "stage1",
        "stageType": "TRANSCODER"
      },
      {
        "id": "stage2",
        "stageType": "STREAM_TARGET_HLS"
      }
    ],
    "version": "1.0"
  }
]

Expected result:

[
  {
    "id": "123",
    "stages": [
      {
        "id": "stage1",
        "stageType": "TRANSCODER"
      },
      {
        "id": "stage2",
        "stageType": "STREAM_TARGET_HLS"
      },
      {
        "id": "stage3",
        "stageType": "STREAM_TARGET_RTMP"
      }
    ],
    "version": "1.0"
  }
]

Notice how we want to add the stage3 object to the end of the array. We are aware that we can use the following transformation:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "stages": {
          "[2]": {
            "id": "stage3",
            "stageType": "STREAM_TARGET_RTMP"
          }
        }
      }
    }
  }
]

However, it implies we must know the index where we want to add the new element, and in our use case, the index is unknown, so, ideally, we just want to add at the end of the array.

Any help would be appreciated.

[LucaBiscotti]

Hi @estigma88, this problem can be easily fixed withe the =size() function of the modify-overwrite-beta.
You can extract the size to then use it as the index, so then the jolt would be something like this:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "s": "=size(@(1,stages))",
        "a": "=intSum(@(1,s),1)",
        "addon": {
          "id": "=concat('stage',@(2,a))",   //this one put the size+1 in the value (like stage3 if size is 2)
          "stageType": "STREAM_TARGET_RTMP"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "@0,id": "[0].id",
        "@(0,stages)": "[0].stages",
        "@(0,addon)": "[0].stages[@(0,s)]",
        "@version": "[0].version"
      }
    }
  }
]

Hope i helped

[estigma88]

Hi @LucaBiscotti , thanks for the help, however, the solution has some drawbacks for our use case:

1. Our JSON is far longer than the sample I shared, it might have dozens of properties, objects and arrays, that some might be optionals, so, it seems no viable to add each of them to the shift operation.
2. The transformation spec is handled by our support team, with low knowledge in JSON and the internals of JOLT, so, understanding that spec will be challenging.

I was hoping for something more simple, as follows:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "stages": {
          "[]": {
            "id": "stage3",
            "stageType": "STREAM_TARGET_RTMP"
          }
        }
      }
    }
  }
]

Where [] tells JOLT to add the object to the end.

If something similar is not supported, I am open to contribute the change to JOLT, if you think it makes sense and give me some clues about where to start..

[LucaBiscotti]

Hi @estigma88, unfortunately there is no possible way of doing what you say by only using the shift operation (as far as I know).
The only way I found to solve your problem is to put all the values you want in the addon in the first operation, and then add that later in the shift.
The JOLT is not very powerful, so sometimes you have to put more operations to reach your goal.
Maybe a less dynamic version of the solution can be this:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "s": "=size(@(1,stages))",
        "addon": {      //inside here you can put every value you want to insert
          "id": "stage3",
          "stageType": "STREAM_TARGET_RTMP"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "@0,id": "[0].id",                    //copying the id outside the list stages
        "@(0,stages)": "[0].stages",          //copying the stages list
        "@(0,addon)": "[0].stages[@(0,s)]",  //adding in the last position of the stages list the "addon" created in the other spec
        "@version": "[0].version"            //copying the version outside the list stages
      }
    }
  }
]

I wrote a guide you can use to better understand what's going on with those operations, you can find it here

[janvi04]

Hi @estigma88
I tried the following spec. Just that you will have to use an extra remove and shift.

[
   {
      "operation":"modify-default-beta",
      "spec":{
         "*":{
            "temp":{
               "id":"stage3",
               "stageType":"STREAM_TARGET_RTMP"
            }
         }
      }
   },
   {
      "operation":"shift",
      "spec":{
         "*":{
            "temp":"stages",
            "@":""
         }
      }
   },
   {
      "operation":"remove",
      "spec":{
         "temp":""
      }
   },
   {
      "operation":"shift",
      "spec":{
         "@":"[]"
      }
   }
]

I hope this helps!

[estigma88]

Thank you all for the help, however, the complexity is still hight, so, we are going to evaluate a switch from JOLT to JsonPatch, as seems like JOLT addresses a lot of features we don't truly need