As a developer,
I want to know if the IATI applications communicating via the MQ get out of sync
so that I can make the system more robust and debug problems.
Think about adding observability / alerts when data consistency issues arise, for example, when the BDS receives a message to update or delete a dataset or reporting org but it doesn't yet have this dataset or reporting org in its database. The update case could be handled by simply creating it. But the delete may indicate something has gone wrong.
In theory, delete messages should only be received for known datasets. But there are two scenarios where this may not hold.
-
The Registry may fail to send a CREATED message, perhaps due to a temporary failure to contact Azure Service Bus.
-
Azure Service Bus does not guarantee that messages are always delivered in the order in which they are received. Thus, it is possible given normal operation of the Registry that a user may add a dataset then very quickly delete it, and the messages may be received by the BDS in the wrong order.
It would be good to be alerted to this.
These errors are handled by throwing a BulkDataServiceError, so if the handling code were to catch this exception specifically, and logged an error when so, then once Sentry is added to the BDS, we would get some minimal alerting of such errors. May be good to discuss whether this would be enough.
As a developer,
I want to know if the IATI applications communicating via the MQ get out of sync
so that I can make the system more robust and debug problems.
Think about adding observability / alerts when data consistency issues arise, for example, when the BDS receives a message to update or delete a dataset or reporting org but it doesn't yet have this dataset or reporting org in its database. The update case could be handled by simply creating it. But the delete may indicate something has gone wrong.
In theory, delete messages should only be received for known datasets. But there are two scenarios where this may not hold.
The Registry may fail to send a CREATED message, perhaps due to a temporary failure to contact Azure Service Bus.
Azure Service Bus does not guarantee that messages are always delivered in the order in which they are received. Thus, it is possible given normal operation of the Registry that a user may add a dataset then very quickly delete it, and the messages may be received by the BDS in the wrong order.
It would be good to be alerted to this.
These errors are handled by throwing a
BulkDataServiceError, so if the handling code were to catch this exception specifically, and logged an error when so, then once Sentry is added to the BDS, we would get some minimal alerting of such errors. May be good to discuss whether this would be enough.