Standalone typed deserializer inconsistent failure

[freakolowsky]

Environment Details

• Eclipse Serializer Version: 4.0.1
• JDK version: Temurin 25
• OS: Linux 6.18.18-1-MANJARO
• Used frameworks: eclipse-store

This might not be a bug but only a PEBKAC situation, i just don't see it ...

I am serializing and offloading data to files for asynchronous processing in a different thread that deserializes and consumes the data. Both threads run in the same VM and each instantiates it's own serializer but using the same code:

        this._foundation = SerializerFoundation.New();
        this._foundation.setFieldEvaluatorPersistable(ESUtils.PFE_NON_PERSISTABLE);
        this._foundation.registerEntityTypes(REGISTERED_TYPES);
        this._foundation.setSerializerTypeInfoStrategyCreator(
            new SerializerTypeInfoStrategyCreator.IncrementalDiff(true)
        );

        this._serializer = TypedSerializer.Bytes(_foundation);

When both threads are running the process works normally.

When i run only the cosumer thread with existing serialized data i get an exception. The weird part is i get a different exception on each run. Below are multiple (partial) stack traces all triggered on the same line of my code. The input data does not change between runs.

Any help would be appreciated ...

Caused by: org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeHandlerConsistencyUnhandledTypeId: No type handler found for type id "1000084".
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.lookupTypeHandler(BinaryLoader.java:206)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.createBuildItem(BinaryLoader.java:214)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.internalReadBinaryEntities(BinaryLoader.java:170)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readBinaryEntities(BinaryLoader.java:158)
	at org.eclipse.serializer.persistence.binary.types.ChunksWrapper.iterateEntityData(ChunksWrapper.java:85)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.addChunks(BinaryLoader.java:805)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readLoadOnce(BinaryLoader.java:713)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:833)
	at org.eclipse.serializer.persistence.types.PersistenceManager$Default.get(PersistenceManager.java:401)
	at org.eclipse.serializer.TypedSerializer$Default.deserialize(TypedSerializer.java:336)

Caused by: java.lang.NullPointerException: Cannot invoke "org.eclipse.serializer.collections.types.XGettingCollection.iterator()" because the return value of "org.eclipse.serializer.collections.BulkList.last()" is null
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.addChunks(BinaryLoader.java:802)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readLoadOidData(BinaryLoader.java:775)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readLoadOnce(BinaryLoader.java:719)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:833)
	at org.eclipse.serializer.persistence.types.PersistenceManager$Default.get(PersistenceManager.java:401)
	at org.eclipse.serializer.TypedSerializer$Default.deserialize(TypedSerializer.java:336)

Caused by: org.eclipse.serializer.persistence.binary.exceptions.BinaryPersistenceExceptionInvalidList: Invalid list data: entityLength = 36, objectId = 1000000000000000002, typeId = 1000082, listStartOffset = 0, listTotalLength = 8059421877986.
	at org.eclipse.serializer.persistence.binary.types.Binary.getBinaryListTotalByteLength(Binary.java:762)
	at org.eclipse.serializer.persistence.binary.types.Binary.getBinaryListElementCountValidating(Binary.java:781)
	at org.eclipse.serializer.persistence.binary.types.Binary.getListElementCountReferences(Binary.java:815)
	at org.eclipse.serializer.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.create(BinaryHandlerNativeArrayObject.java:115)
	at org.eclipse.serializer.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.create(BinaryHandlerNativeArrayObject.java:30)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.createBuildItem(BinaryLoader.java:217)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.internalReadBinaryEntities(BinaryLoader.java:170)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readBinaryEntities(BinaryLoader.java:158)
	at org.eclipse.serializer.persistence.binary.types.ChunksWrapper.iterateEntityData(ChunksWrapper.java:85)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.addChunks(BinaryLoader.java:805)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readLoadOnce(BinaryLoader.java:713)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:833)
	at org.eclipse.serializer.persistence.types.PersistenceManager$Default.get(PersistenceManager.java:401)
	at org.eclipse.serializer.TypedSerializer$Default.deserialize(TypedSerializer.java:336)

Caused by: org.eclipse.serializer.persistence.exceptions.PersistenceExceptionTypeNotPersistable: Type not persistable: "interface java.util.Map".
	at org.eclipse.serializer.persistence.binary.types.BinaryHandlerAbstractType.create(BinaryHandlerAbstractType.java:83)
	at org.eclipse.serializer.persistence.binary.types.BinaryHandlerAbstractType.create(BinaryHandlerAbstractType.java:23)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.createBuildItem(BinaryLoader.java:217)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.internalReadBinaryEntities(BinaryLoader.java:170)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readBinaryEntities(BinaryLoader.java:158)
	at org.eclipse.serializer.persistence.binary.types.ChunksWrapper.iterateEntityData(ChunksWrapper.java:85)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.addChunks(BinaryLoader.java:805)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.readLoadOnce(BinaryLoader.java:713)
	at org.eclipse.serializer.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:833)
	at org.eclipse.serializer.persistence.types.PersistenceManager$Default.get(PersistenceManager.java:401)
	at org.eclipse.serializer.TypedSerializer$Default.deserialize(TypedSerializer.java:336)

[hg-ms]

At first glance, I’d assume that there is problem with missing type information.

The IncrementalDiff type handling, especially if type info is included only once requires the exact same order for deserialization like serialization to keep track of newly added types during serialization to minimize the overhead caused by the type information.

It is also important to know that type must be registered in the same order to ensure matching type ids.
Did you try to use a different type-strategy, like “TypeDictionary”?
This would include all type informations in the serialized data.

[freakolowsky]

As i said both serialization and deserialization use the same initialization code. The order of calls is also identical.

I've tried it again by just leaving out the foundation.registerEntityTypes call and it seems to work, but i get some errors later on which i'm not sure if it's related.

Will try it again over weekend with TypeDirectory strategy.