Sync Module

I was able to pass a voided identifier across when in 1.5 and the sync module.

Do you have a repeatable workflow to cause this bug Maros?

Saw this happen with edits that were on the main patient admin screen. I made a change in rev:12927 that skips over "deleted" records that didn't exist to begin with. This might mitigate the problem...but the underlying cause is still unknown.

This is frustrating since I can't repeat it reliably, but just now with 1.6 and latest sync I got this payload that throws 'object deleted will be resaved' exception. Looking at the payload, the persistentset 'delete' for uuid is clearly wrong. I am moving it to 'current' release and will continue to search for repeatable workflow.

failing payload example:
<items><SyncItem containedType="org.openmrs.PatientIdentifier" key="25196c84-40b3-4088-ab94-d1b71007101a" state="NEW"><content><Unable to render embedded object: File (00:33.593-0500</dateCreated><uuid type="string">25196c84-40b3-4088-ab94-d1b71007101a</uuid><preferred type="boolean">false</preferred><identifierType type="org.openmrs.PatientIdentifierType">8d79403a-c2cc-11de-8d13-0010c6dffd0f</identifierType><identifier type="string">222222</identifier><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PatientIdentifier>) not found.[type="string">9637a2f0-154c-4836-b4f0-eda36febf604</uuid></org.openmrs.PatientIdentifier>|CDATA[<org.openmrs.PatientIdentifier><uuid]]></content></SyncItem><SyncItem containedType="org.openmrs.Patient" key="2f5b91b4-23bc-4977-ac48-582374cacc87" state="UPDATED"><content><Unable to render embedded object: File (00:33.593-0500</personDateChanged><personVoided type="boolean">false</personVoided><dateChanged type="timestamp">2010-05-04T10:00:33.593-0500</dateChanged><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator><changedBy type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</changedBy><personChangedBy type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</personChangedBy><patient type="boolean">true</patient><personDateCreated type="timestamp">2010-05-04T09:52:23.000-0500</personDateCreated><voided type="boolean">false</voided><birthdate type="timestamp">1977-01-01T00:00:00.000-0600</birthdate><gender type="string">F</gender><dateCreated type="timestamp">2010-05-04T09:52:23.000-0500</dateCreated><uuid type="string">2f5b91b4-23bc-4977-ac48-582374cacc87</uuid><dead type="boolean">false</dead></org.openmrs.Patient>) not found.[type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</changedBy><middleName type="string">patid</middleName><person type="org.openmrs.Person">2f5b91b4-23bc-4977-ac48-582374cacc87</person><voided type="boolean">false</voided><familyName type="string">test2</familyName><dateCreated type="timestamp">2010-05-04T09:52:23.000-0500</dateCreated><givenName type="string">synco</givenName><uuid type="string">79a71053-fee4-4b49-afda-6ec97e93aa96</uuid><preferred type="boolean">true</preferred><dateChanged type="timestamp">2010-05-04T10:00:33.593-0500</dateChanged><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PersonName>|CDATA[<org.openmrs.PersonName><changedBy]]></content></SyncItem><SyncItem containedType="org.openmrs.PatientIdentifier" key="019778a2-03dd-49b4-9ccc-84633ef21f8b" state="UPDATED"><content><Unable to render embedded object: File (00:09.000-0500</dateCreated><uuid type="string">019778a2-03dd-49b4-9ccc-84633ef21f8b</uuid><voidReason type="string">Removed from new patient screen</voidReason><preferred type="boolean">false</preferred><dateVoided type="timestamp">2010-05-04T10:00:33.593-0500</dateVoided><identifierType type="org.openmrs.PatientIdentifierType">8d79403a-c2cc-11de-8d13-0010c6dffd0f</identifierType><identifier type="string">111111</identifier><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PatientIdentifier>) not found.[type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</voidedBy><patient type="org.openmrs.Patient">2f5b91b4-23bc-4977-ac48-582374cacc87</patient><location type="org.openmrs.Location">cd0613cf-c264-46fc-a954-5c849de98ed3</location><voided type="boolean">true</voided><dateCreated type="timestamp">2010-05-04T10:00:09.000-0500</dateCreated><uuid type="string">38009fee-226e-4605-a81a-d488be54c8e7</uuid><voidReason type="string">Removed from new patient screen</voidReason><preferred type="boolean">false</preferred><dateVoided type="timestamp">2010-05-04T10:00:33.593-0500</dateVoided><identifierType type="org.openmrs.PatientIdentifierType">8d79403a-c2cc-11de-8d13-0010c6dffd0f</identifierType><identifier type="string">1111112</identifier><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PatientIdentifier>|CDATA[<org.openmrs.PatientIdentifier><voidedBy]]></content></SyncItem><SyncItem containedType="org.openmrs.PatientIdentifier" key="78729f54-f8b5-4602-aa5b-033eb0cd91ec" state="UPDATED"><content><Unable to render embedded object: File (00:09.000-0500</dateCreated><uuid type="string">78729f54-f8b5-4602-aa5b-033eb0cd91ec</uuid><voidReason type="string">Removed from new patient screen</voidReason><preferred type="boolean">false</preferred><dateVoided type="timestamp">2010-05-04T10:00:33.593-0500</dateVoided><identifierType type="org.openmrs.PatientIdentifierType">8d79403a-c2cc-11de-8d13-0010c6dffd0f</identifierType><identifier type="string">12121-7</identifier><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PatientIdentifier>) not found.[type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</voidedBy><patient type="org.openmrs.Patient">2f5b91b4-23bc-4977-ac48-582374cacc87</patient><location type="org.openmrs.Location">8d6c993e-c2cc-11de-8d13-0010c6dffd0f</location><voided type="boolean">true</voided><dateCreated type="timestamp">2010-05-04T10:00:09.000-0500</dateCreated><uuid type="string">7266d2cf-4003-4922-9d2d-186eb96253e2</uuid><voidReason type="string">Removed from new patient screen</voidReason><preferred type="boolean">false</preferred><dateVoided type="timestamp">2010-05-04T10:00:33.593-0500</dateVoided><identifierType type="org.openmrs.PatientIdentifierType">8d79403a-c2cc-11de-8d13-0010c6dffd0f</identifierType><identifier type="string">333333synco</identifier><creator type="org.openmrs.User">ca836af4-4c1a-11df-a301-6248447b9da3</creator></org.openmrs.PatientIdentifier>|CDATA[<org.openmrs.PatientIdentifier><voidedBy]]></content></SyncItem><SyncItem containedType="org.hibernate.collection.PersistentSortedSet" key="2f5b91b4-23bc-4977-ac48-582374cacc87|identifiers" state="UPDATED"><content><![action="update" properyName="identifiers" type="org.openmrs.Patient" uuid="2f5b91b4-23bc-4977-ac48-582374cacc87"/><entry action="update" type="org.openmrs.PatientIdentifier" uuid="019778a2-03dd-49b4-9ccc-84633ef21f8b"/><entry action="update" type="org.openmrs.PatientIdentifier" uuid="38009fee-226e-4605-a81a-d488be54c8e7"/><entry action="update" type="org.openmrs.PatientIdentifier" uuid="78729f54-f8b5-4602-aa5b-033eb0cd91ec"/><entry action="update" type="org.openmrs.PatientIdentifier" uuid="7266d2cf-4003-4922-9d2d-186eb96253e2"/><entry action="update" type="org.openmrs.PatientIdentifier" uuid="25196c84-40b3-4088-ab94-d1b71007101a"/><entry action="delete" type="org.openmrs.PatientIdentifier" uuid="9637a2f0-154c-4836-b4f0-eda36febf604"/></org.hibernate.collection.PersistentSortedSet>|CDATA[<org.hibernate.collection.PersistentSortedSet><owner]]></content></SyncItem></items>

Hey Maros--

This does seem like a frustrating one... I played around with it for a while this morning and was unable to reproduce (though I managed to discover an unrelated core bug in the process: TRAC-2336).

One thing I did notice, however, if I understand you correctly, is that the payload you posted above is at least internally consistent... patient identifier 9637a2f0-154c-4836-b4f0-eda36febf604 is listed as to be deleted both in the sync record for the identifier, and in the identifier set.

I will try to come back to it later and test it out again.

Finally a reproducible scenario:
1. create new patient on child with two pat ids: pid-1, pid-2. save patient.
2. sync to parent.
3. verify in DB that child and parent have exactly two rows in patient_identifier and uuids are same, note the uuids.
4. on child edit patient short form: add new id pid-3, and remove pid-1. save patient.
5. on child in DB notice that:

• pid-3 was added
• pid-1 was voided and has still the original uuid from step TRAC-3
• pid-2 has same value (i.e pid-2) but new uuid.
  6. now sync

The above steps will break sync in 0.73 rev with error of 'object will be removed by cascade delete, remove from associations'.

Resolved in rev:13689.
The description of the problem:
The patient demographics short form during save rebuilds the identifier collection for any new and updated & existing identifiers. This means that even though in the stated example pid-2 wasn't being modified, it was being in affected deleted and re-inserted. This is supported by fact that it gets new uuid() assigned on child server. As a result, the sync processing for the above example results in: update on pid-1 (to void), delete on pid-2, add on pid-2, and add on pid-3.
The issue in 0.73 is how to process delete of pid-2: sync normally executes direct delete (i.e. sessionFactory.getCurrentSession().delete()). Since patient.identifiers collection is marked in mapping file as all-cascade-orphans this causes the above error.
Resolution:
The problem above should easily solvable by changing logic in SyncUtil.DeleteOpenMRSObject() to pidToRemove.getPatient().getIdentifiers.remove(pidToRemove). This however does not work. As it turns out, removal from identifiers collection does not work in the test case above. This is then a reason hibernate also fails to remove from the collection and properly issue 'delete'. The cause of this is that patient.identifiers collection is TreeSet that is being modified and not properly re-sorted. In the above test case, pid-1 is initially sorted as first, but when voided it is supposed to be last (see patientidentifier.CompareTo() rules). This change in value however does not get reflected in the collection ordering (treesets don't automatically resort when referenced objects change state). Consequently when collection.remove() is executed by hibernate in an attempt to remove pid-2, hibernate actually does not find the pid-2 in the collection: the combination of changed item to voided and structure of the treemap that backs up TreeSet collection is just that treemap search results in the dead end.
To work around this bug in the core, commit rev:13689 changes the ordering of how how we process patient identifiers during ingest: the updates are moved to be last (after inserts and deletes) since any update to existing identifier has a potential of messing up the treeset and render it essentially corrupt.

The SyncOnDeleteTest.shouldDeletePatient unit test is failing in the SyncUtil.deleteOpenmrsObject method.

Fixed patient delete scenario in rev:13792.