Duplicate inverse attributes in JSON serialization

[hlg]

Consider a project modified through the low level interface. For inverse relations in such a project, with higher cardinality, the SharedJsonSerializer delivers multiple instances of each reference - one for each revision in which that references has existed. See the example below, where the inverse relation IfcRelAggregates.IsDecomposedBy has been existing for 5 revisions.

{ "_i": 132903,
  "_u": "76e8cec6-3d9c-48ae-b07a-664fe9787809",
  "_r": "8",
  "_t": "IfcSite",
  "_s": 1,
  "GlobalId": "1cwlDi_hLEvPsClAelBNnz",
  "_rOwnerHistory": {
    "_i": 132401,
    "_t": "IfcOwnerHistory"
  },
  "Name": "Default Site",
  "Description": "Description of Default Site",
  "_rIsDecomposedBy": [
    { "_i": 329053, "_t": "IfcRelAggregates" },
    { "_i": 329053, "_t": "IfcRelAggregates" },
    { "_i": 329053, "_t": "IfcRelAggregates" },
    { "_i": 329053, "_t": "IfcRelAggregates" },
    { "_i": 329053, "_t": "IfcRelAggregates" }
  ], ...
}

I don't understand whether this issue is triggered in the serializer or in the database layer and if the latter whether on storage or on access.
It is easy to work around the issue by eliminating duplicates on the client side.
But obviously it would be better to remove the duplicates before, so they don't consume bandwith, memory and potentially disk space.

[rubendel]

Hmm, could be an a lot of different places. Do you get the same with the SharedJsonStreamingSerializer?

[hlg]

Yes, it is the same with the streaming serializer.

[PavelWhiteTiger]

There is method "readList" in class DatabaseReadingStackFrame , there "long rf = readReference(buffer, feature, referenceClass);" gets the same value every iteration ...

[hlg]

Thanks. You probably mean this:

BIMserver/BimServer/src/org/bimserver/database/queries/DatabaseReadingStackFrame.java

Line 458 in 6daa43a

long rf = readReference(buffer, feature, referenceClass);

Actually, at this point, the iterations seem to be based on the buffer size as read from the database, hence the error likely occurs when writing to the database already.

[hlg]

I think I found the root cause:

BIMserver/BimServer/src/org/bimserver/database/actions/CommitTransactionDatabaseAction.java

Line 207 in 6daa43a

fixInverses(packageMetaData, newRoid, summaryMap.getSummaryMap());

Inverses are fully regenerated and written to the database on each low level commit. They should only be fixed (added and removed references) according to the submitted update changes. This is not absolutely trivial, but I think we might have a fix soon.

[PavelWhiteTiger]

I come back later. We don`t use this method. Commented 206-210 strings.)

[hlg]

If you comment those lines, you will run the risk of dangling inverse references in the database. If you do not remove references with the low level interface you should be fine with this workaround. Added references should be treated already in terms of inverses.