[Feature request] Allow sending messages with (nearly) unlimited size

[sudden6]

Currently every Tox client must split long messages into smaller parts. I think it would be beneficial to the user experience and the developer experience if toxcore supported messages with unlimited size or with sizes that are way longer than the current maximum (1372 bytes).

I think this can be implemented either by introducing a flag that indicates a message as part of a multipart message.

[yurivict]

I almost implemented this in qTox.
This requires persistent storage that toxcore doesn't have.

[aaannndddyyy]

+1

[sudden6]

@yurivict I'm interested in how you implemented this in qTox, anything public yet?

Also how do you keep compatibility to other clients?

I also think it can be done without persistent storage, my simple approach would be to only mark a message as confirmed delivered when all parts have arrived, but to retransmit parts as long as both clients are online.

[yurivict]

@sudden6 If the messages aren't too large, it is ok to defragment without persistent storage. But imagine that somebody will send 1GB message? He may need to reboot the computer in between, and previous parts will get lost and he will need to start all over again. It is better to have persistent storage IMO in the general case.

I will release qTox with defragmenter once I will work out few remaining problems.

I also think that it is better to have a mid-level library, xtox, that will contain defragmenter itself to allow other clients to use it. This assumes that all clients use sqlite. I am not sure if this is a reasonable assumption.

This xtox library could export another version of Tox interface that will allow large messages.
Bug #499 is a pre-requisite for such library.

[isotoxin]

Isotoxin has long able to send messages of unlimited size.

I almost implemented this in qTox.

It will be good to make it compatible together, but I understand it is impossible.

[yurivict]

I am going to release the library libtox-defragmenter.so that will do message defragmentation. I am polishing the details now.

This shouldn't be done in the clients, because then defragmentation isn't going to be compatible between clients.

[yurivict]

Tox-defragmenter library: https://github.com/yurivict/tox-defragmenter

It needs:

• Structured API in the tox library: [overridable API] Added Tox API in a structure form. #502
• (for qTox) Wrapped API in the client: Wrapped Tox API in macros. qTox/qTox#4238
• The fix in toxcore: Fixed the bug when receipts for messages sent from the receipt callback never arrived. #500

qTox compatible with defragmenter is in my branch: https://github.com/yurivict/qTox/tree/tox-defragmenter

I will likely update tox-defragmenter based on further testing.

TODO:
Make client aware of the persistent nature of receipts returned by tox-defragmenter. Table faux_offline_pending will need the field persistent_receipt which will make qTox show that the message isn't yet sent, but will prevent it from resending it.

[sudden6]

I don't think tox-defragmenter should come with it's own db, most clients already have a db or something else to store persistent data.

In your use case it fits perfectly for qTox, but what about more lightweight clients that don't use sqlcipher?

@isotoxin can you describe the approach you used?

[yurivict]

I don't think tox-defragmenter should come with it's own db, most clients already have a db or something else to store persistent data.

It doesn't come with db. It uses the db supplied by the client.

In your use case it fits perfectly for qTox, but what about more lightweight clients that don't use sqlcipher?

The persistent storage is needed to handle large messages. It can be made to not explicitly depend on sqlcipher, only require the functions, so that both sqlcipher and SQLite will fit.

[sudden6]

AFAICT your code directly depends on sqlcipher or SQLite, but what if a client wants to use (encrypted) text files as storage?

I didn't look to deep into your code, but what happens if a "normal" client receives a message with your splitting protocol?

[yurivict]

AFAICT your code directly depends on sqlcipher or SQLite, but what if a client wants to use (encrypted) text files as storage?

I added the ability to use the in-memory database: tox_defragmenter_initialize_db_inmemory().

I didn't look to deep into your code, but what happens if a "normal" client receives a message with your splitting protocol?

They will still show it as split, as it happens today.