Deserialization fails with "Buffer underflow" after add a new field to record class (Issue with backward compatibility)

[investr777]

Kryo version 5.5.0
Java 17

When data was serialized and after that a new field was added, deserilization fails with exception: com.esotericsoftware.kryo.kryo5.io.KryoBufferUnderflowException: Buffer underflow
Stacktrace

Exception in thread "main" com.esotericsoftware.kryo.kryo5.io.KryoBufferUnderflowException: Buffer underflow.
Serialization trace:
info (akta.dcs.session.live.hash.Data)
	at com.esotericsoftware.kryo.kryo5.io.Input.require(Input.java:221)
	at com.esotericsoftware.kryo.kryo5.io.Input.readVarIntFlag(Input.java:482)
	at com.esotericsoftware.kryo.kryo5.io.Input.readString(Input.java:777)
	at com.esotericsoftware.kryo.kryo5.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:174)
	at com.esotericsoftware.kryo.kryo5.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:164)
	at com.esotericsoftware.kryo.kryo5.Kryo.readObjectOrNull(Kryo.java:826)
	at com.esotericsoftware.kryo.kryo5.serializers.RecordSerializer.read(RecordSerializer.java:136)
	at com.esotericsoftware.kryo.kryo5.Kryo.readObject(Kryo.java:777)

Code example:

public record Data(
    String id
){}

===============================================
var kryoLocal = ThreadLocal.withInitial(() -> {
    Kryo kryo = new Kryo();
    kryo.setRegistrationRequired(false);
    kryo.setDefaultSerializer(CompatibleFieldSerializer.class);
    return kryo;
});

 var data = new Data("1");

  var output = new Output(1024, -1);
  kryoLocal.get().writeObject(output, data);
  output.flush();
  var serializedData = output.toBytes(); // [-126, 49]
  output.close();

After serialized, new field was added:

public record Data(
    String id,
   String info
){}

=================

var input = new Input(new byte[] {-126, 49});
var deserializedData = kryoLocal.get().readObject(input, Data.class);
input.close();

And eventually an exception is thrown: com.esotericsoftware.kryo.kryo5.io.KryoBufferUnderflowException: Buffer underflow.

But if convert Java record to class, everything works properly:
Code example:

public class Data {
    private String id;

    public Data() {
    }

    public Data(
        String id
    ) {
        this.id = id;
    }

    public String id() {
        return id;
    }
}

=====================================
var kryoLocal = ThreadLocal.withInitial(() -> {
            Kryo kryo = new Kryo();
            kryo.setRegistrationRequired(false);
            kryo.setDefaultSerializer(CompatibleFieldSerializer.class);
            return kryo;
        });

        var data = new Data("1");

        var output = new Output(1024, -1);
        kryoLocal.get().writeObject(output, data);
        output.flush();
        var serializedData = output.toBytes(); // [1, 105, -28, 3, -126, 49]
        output.close();

After serialized, new field was added:

public class Data {
    private String id;
    private String info;

    public Data() {
    }

    public Data(
        String id,
        String info
    ) {
        this.id = id;
        this.info = info;
    }

    public String id() {
        return id;
    }

    public String info() {
        return info;
    }
}

==========================================

var input = new Input(new byte[] {1, 105, -28, 3, -126, 49});
var deserializedData = kryoLocal.get().readObject(input, Data.class);
input.close();

As we can see, that serialized record and class have different bytes.

[theigl]

@investr777: Thanks for the detailed report and sorry for the late answer!

The RecordSerializer currently does not have support for backwards/forwards compatibility. It is a limitation in its design that I realized only after it was originally contributed by Oracle.

See this issue for a more detailed discussion of the problem.

I have created a POC that integrates record serialization back into the default FieldSerializer and its subclasses. I ran your test-case against this PR and it passes. it will be part of the next major release, Kryo 6.

I'm afraid that for Kryo 5 there is not much we can do to fix this. If you need support for this right now and can't switch to normal classes for cases where you need compatibility, you could write your own record serializer that uses the same logic as CompatibleFieldSerializer.

[investr777]

@theigl, thank you so much for your reply.
Will Kryo 6 and Kryo 5 have backward compatibility? Will Kryo 6 be able to deserialized data, which were serialized by Kryo 5?

[theigl]

@investr777: Yes, my goal is that Kryo 6 is backwards compatible (See https://github.com/EsotericSoftware/kryo/wiki/Kryo-v6-Ideas).