Migrating custom storage to RC_2

[elgatito]

Hello Arvid,

While trying to migrate to move the code from RC_1_2 to RC_2_0, I've found that I miss a way few items, used from RC_1_2:

1. Method to access disk_thread from session.
2. Way to easily readv/writev using storage methods.

Regarding 1:
I have custom storage interface, but it is needed to be able to use both default and custom interface, depending of user's selection. That was previously easily made by add_torrent_params field.
Now I will probably do a custom lt::disk_interface, which will internally make default_disk_io_constructor, to delegate work with non-custom storages.

Regarding 2:
I have used direct readv calls to storage_interface to properly read downloaded date with a buffer.
Is there an easy example of to calling async_readv to fill in the buffer?

[elgatito]

@arvidn Have you seen such errors from compilers?

This code:

   auto flags = is_delete_data && !is_memory_storage() ? lt::session::delete_files : lt::session::delete_partfile;

It was working fine with RC_1_2, now with RC_2_0 and GCC-6 I get this error:

torrent.cpp:(.text+0xb55): undefined reference to `libtorrent::session_handle::delete_partfile'

Have no error with GCC-7 or GCC-10.
I don't understand why this happen. When I replace delete_partfile with delete_files - compiles fine.

Can I manually set that bit without referencing the variable?

[arvidn]

delete_partfile is defined here.

Are you linking against libtorrent as a shared library? is the symbol exported?

[arvidn]

Regarding 1:
I have custom storage interface, but it is needed to be able to use both default and custom interface, depending of user's selection. That was previously easily made by add_torrent_params field.
Now I will probably do a custom lt::disk_interface, which will internally make default_disk_io_constructor, to delegate work with non-custom storages.

That makes sense. It would be neat if the torrent's user-data would be passed in to new_torrent() on the disk_interface. You may be able to use the storage_mode_t to pass information into your disk subsystem.

Regarding 2:
I have used direct readv calls to storage_interface to properly read downloaded date with a buffer.

It's not generally safe to do that from arbitrary threads. I suppose with your own storage you can make sure it's safe. With the new disk abstraction, the callbacks are run on the network thread, which is risky to run client-code in.

Have you looked into read_piece() on the torrent_handle instead?

Is there an easy example of to calling async_readv to fill in the buffer?

Here's an example:

https://github.com/arvidn/libtorrent/blob/RC_2_0/src/peer_connection.cpp#L5336

[elgatito]

@arvidn

delete_partfile is defined here.

Are you linking against libtorrent as a shared library? is the symbol exported?

Libtorrent built as a static library. With GCC-7 same library, same sources work fine, and with RC_1_2 it worked fine with GCC-6.
It is very strange. delete_files is ok, delete_partfiles is not. Maybe a GCC bug... Is there any way to manually set value as a bit, to avoid using a variable?

[elgatito]

It's not generally safe to do that from arbitrary threads. I suppose with your own storage you can make sure it's safe. With the new disk abstraction, the callbacks are run on the network thread, which is risky to run client-code in.

Have you looked into read_piece() on the torrent_handle instead?

It is an http server, which consumes bytes from libtorrent, so there is a buffered reader, which takes 16kb, per call.
Currently, I ask readv for piece parts, it works fine. read_piece is reading all the piece, so it is returning too much for buffered reading.

[elgatito]

@arvidn

m_nativeSession->remove_torrent(m_nativeHandle, is_delete_data && !is_memory_storage() ? lt::session::delete_files : lt::remove_flags_t(1));

I know that is bad, but is that the same value is delete_partfiles - lt::remove_flags_t(1)? This variant compiles fine with GCC-6.

[elgatito]

@arvidn That is very strange. I have now also these errors:

/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:228: undefined reference to `libtorrent::alert::file_progress_notification'
/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:228: undefined reference to `libtorrent::alert::error_notification'
/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:228: undefined reference to `libtorrent::alert::performance_warning'
/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:228: undefined reference to `libtorrent::alert::status_notification'
/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:228: undefined reference to `libtorrent::alert::tracker_notification'
/home/elgato/Develop/lt2http/src/bittorrent/session.cpp:233: undefined reference to `libtorrent::alert::all_categories'

Now I check names inside static library:

elgato@HP-Laptop:~/Develop/libtorrent-arvidn/build$ nm /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.a | grep delete_partfile
elgato@HP-Laptop:~/Develop/libtorrent-arvidn/build$ nm /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.a | grep delete_files
                 U _ZN10libtorrent7torrent12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEE
0000000000000000 u _ZN10libtorrent14session_handle12delete_filesE
0000000000001370 T _ZN10libtorrent3aux12delete_filesERKNS_12file_storageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESB_NS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
0000000000025d80 T _ZN10libtorrent7torrent12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEE
0000000000000000 W _ZN10libtorrent16disabled_disk_io18async_delete_filesENS_3aux14strong_typedefIjNS_19storage_index_tag_tEvEENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEESt8functionIFvRKNS_13storage_errorEEE
0000000000000000 W _ZN5boost4asio6detail11executor_opINS1_15work_dispatcherIZN10libtorrent16disabled_disk_io18async_delete_filesENS4_3aux14strong_typedefIjNS4_19storage_index_tag_tEvEENS4_5flags13bitfield_flagIhNS4_16remove_flags_tagEvEESt8functionIFvRKNS4_13storage_errorEEEEUlvE_EESaIvENS1_19scheduler_operationEE11do_completeEPvPSN_RKNS_6system10error_codeEm
0000000000000110 T _ZN10libtorrent12mmap_disk_io15do_delete_filesEPNS_3aux11disk_io_jobE
0000000000006d00 T _ZN10libtorrent12mmap_disk_io18async_delete_filesENS_3aux14strong_typedefIjNS_19storage_index_tag_tEvEENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEESt8functionIFvRKNS_13storage_errorEEE
                 U _ZN10libtorrent12mmap_storage12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
0000000000001a80 T _ZN10libtorrent12mmap_storage12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
                 U _ZN10libtorrent3aux12delete_filesERKNS_12file_storageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESB_NS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
0000000000000000 W _ZN10libtorrent13posix_disk_io18async_delete_filesENS_3aux14strong_typedefIjNS_19storage_index_tag_tEvEENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEESt8functionIFvRKNS_13storage_errorEEE
                 U _ZN10libtorrent3aux13posix_storage12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
0000000000000000 W _ZN5boost4asio6detail11executor_opINS1_15work_dispatcherIZN10libtorrent13posix_disk_io18async_delete_filesENS4_3aux14strong_typedefIjNS4_19storage_index_tag_tEvEENS4_5flags13bitfield_flagIhNS4_16remove_flags_tagEvEESt8functionIFvRKNS4_13storage_errorEEEEUlvE_EESaIvENS1_19scheduler_operationEE11do_completeEPvPSN_RKNS_6system10error_codeEm
                 U _ZN10libtorrent3aux12delete_filesERKNS_12file_storageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESB_NS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
0000000000001890 T _ZN10libtorrent3aux13posix_storage12delete_filesENS_5flags13bitfield_flagIhNS_16remove_flags_tagEvEERNS_13storage_errorE
elgato@HP-Laptop:~/Develop/libtorrent-arvidn/build$ 

So it is not in the static library. I don't understand that...

I compile with this:

CC=gcc-7 CXX=g++-7 cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=OFF ..
make

[elgatito]

@arvidn
Shit.. That was caused by adding C++17 when building libtorrent. It caused different behavior to constexpr. Strange that with GCC-10 it was not an issue and everything worked.

[elgatito]

@arvidn I have one more strange thing:

void test(const lt::torrent_handle &th) {
    m_handle = th.native_handle();
    if (m_handle->has_picker()) {
        m_handle->picker().set_piece_priority(pit, lt::download_priority_t(0));
        m_handle->picker().we_dont_have(pit);
    }
}

This was working fine with RC_1_1/RC_1_2.
Here picker() is a nullptr, which is strange, because there are downloaded pieces at that moment.
How is that possible?

[elgatito]

@arvidn One more. I was printing trackers' information, now most of those fields are deprecated, what can I use instead of these fields?

		TORRENT_DEPRECATED time_point32 next_announce = (time_point32::min)();
		TORRENT_DEPRECATED time_point32 min_announce = (time_point32::min)();
		TORRENT_DEPRECATED std::string message;
		TORRENT_DEPRECATED error_code last_error;
		TORRENT_DEPRECATED int scrape_incomplete = -1;
		TORRENT_DEPRECATED int scrape_complete = -1;
		TORRENT_DEPRECATED int scrape_downloaded = -1;
		TORRENT_DEPRECATED std::uint8_t fails : 7;
		TORRENT_DEPRECATED bool updating : 1;
		TORRENT_DEPRECATED bool start_sent : 1;
		TORRENT_DEPRECATED bool complete_sent : 1;

[elgatito]

@arvidn Can't get it. I pause debugger in writev, switch frame to peer_connection, and take t->has_picker() and it returns false, just like my getter.
Local picker variable inside peer_connection is fine. But this, inside peer_connection is returning the same nullptr:

		std::shared_ptr<torrent> t = m_torrent.lock();
                t->picker();

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.