CMA output name changes for an array

[hailiangzhang]

We ran into a situation where different lambda functions from the same workflow coming from different component versions with different schema.

One solution is to change the names using CMA as following,

"task_config": {
  "cumulus_message": {
      "outputs": [
        {
          "source": "{$.payload.granules[0].files[0].filepath}",
          "destination": "{$.payload.granules[0].files[0].key}}"
        },
        {
          "source": "{$.payload.granules[0].files[1].filepath}",
          "destination": "{$.payload.granules[0].files[1].key}}"
        },
        {
          "source": "{$.payload.granules[1].files[0].filepath}",
          "destination": "{$.payload.granules[1].files[0].key}}"
        },
        {
          "source": "{$.payload.granules[1].files[1].filepath}",
          "destination": "{$.payload.granules[1].files[1].key}}"
        }
      ]
    }
}

However, as shown above, since this name change involves multiple array items, we have to write a long list to iterate through them. More importantly, the array size which can only be determined at run-time, and the above solution will not work.

Regarding this, I noticed that array support is indeed implemented in CMA input:

  "task_config": {
    "inlinestr": "prefix{meta.foo}suffix",
    "array": "{[$.meta.foo]}",
    "object": "{$.meta}"
  },

and by looking into the CMA source code, I found that actually the array is resolved based on jsonpath, and therefore the following will also work:

    "array": "{[$.meta.array[*].foo]}",

however, it's only implemented in the input, but not for output name changes.

So, it will be nice (and symmetric) if the jsonpath-based name changes can be implemented in output, so that the above long task config can be written as following to make it short and dynamic:

"task_config": {
  "cumulus_message": {
      "outputs": [
        {
          "source": "{[$.payload.granules[*].files[*].filepath]}",
          "destination": "{[$.payload.granules[*].files[*].key}]}"
        }
      ]
    }
}

[hailiangzhang]

I have implemented this in my fork from CMA, and will send a PR soon.

[hailiangzhang]

Sorry forgot to mention that PR was sent out.