[jmaeng72]

Overview

What is the objective?

To let large collections be deleted properly without timeout or server errors

What are the changes?

• Created new index delete queue
• Created new ingest delete exchange
• Collection deletes (to delete its granules) routed to new exchange and queue (other concept deletes will remain going to original index queue and ingest exchange)
• Collection cascade delete in DB is now async and runs in background
• Race condition of users re-creating collection and grans while a delete in happening background is addressed by tracking timestamp of delete request and re-created coll and grans will not be deleted
• Collection cascade delete in ES is now async and runs in background with longer SQS timeout and will only run one delete task at a time

What areas of the application does this impact?

Cascade collection delete - ingest, indexer, metadata-db

Required Checklist

• New and existing unit and int tests pass locally and remotely
• clj-kondo has been run locally and all errors in changed files are corrected
• I have commented my code, particularly in hard-to-understand areas
• I have made changes to the documentation (if necessary)
• My changes generate no new warnings

Additional Checklist

• I have removed unnecessary/dead code and imports in files I have changed
• I have cleaned up integration tests by doing one or more of the following:
  • migrated any are2 tests to are3 in files I have changed
  • de-duped, consolidated, removed dead int tests
  • transformed applicable int tests into unit tests
  • reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

[daniel-zamora]

hardcoded name.

[jmaeng72]

So this value "cmr_index_delete.queue" comes from indexer.config and this method is in message-queue-lib. Indexer already pulls in message-queue-lib as a dependency. If this func (from message-queue-lib) pulls in indexer to get the queue name, then it will create a circular dependency.

The options are keep the hardcoded name as it.

Or do the pattern established like how provider got its queue name with a string match func above

Or refactor queue names from indexer to all live in message-queue-lib

I was hesitant about the string match because I didn't want it to accidentally match some other very closely named queue that would be created elsewhere in the future.

I can refactor the queue names to live in message-queue-lib instead of indexer, but this template will have to be updated for the other apps (like metadata-db) at some point too to make it clean (which we can put in a diff ticket)

LMK your thoughts

[daniel-zamora]

I think there is an option that avoids both problems, :queues-to-policies is used by bootstrap for its 12h timeout.

:queues-to-policies {(bootstrap-queue-name) {:max-tries 1
                                             ;; max 12 hours visibility timeout
                                             :visibility-timeout-secs 43200}}

you can probably do:

:queues-to-policies
{(index-delete-queue-name) {:visibility-timeout-secs (index-delete-queue-visibility-timeout-secs)}}

then the sqs.clj changes can be reverted.

[jmaeng72]

but that is bootstrap, how will that change message-queue-lib? That is where the index-delete queue is created

[jmaeng72]

ok got your message and made the change

[jmaeng72]

I do realize I have a lot of debug logs I created, I can remove the ones you believe are unnecessary. Some I left in there for testing so that the dev can see that the old concepts are being sent to the right queues vs the new cascade coll deletes are being sent to the new queue