Sync progresses only when doing some action in UI

[ghost]

Operating system

• Windows
• macOS
• [x ] Linux Lubuntu 17.10
• Android
• iOS

Application

• [ x] Desktop Joplin-1.0.78-x86_64.AppImage
• Mobile
• Terminal

I first installed Joplin on Windows, set up the sync target to be my Nextcloud instance, and imported my Evernote enex file. That worked OK. It lost the notebooks I had set up, but I re-created them and moved the notes appropriately (I only have a few hundred at this point).

Then I installed Joplin on one of my Linux machines. When setting the sync target to my Nextcloud instance, it only syncs 1 notebook, and that takes a very long time. That notebook only has 5 entries all of which are text only, no graphics or associated files, and the sync takes at least 5 minutes. Nothing I do gets it to sync the other 3 notebooks.

I tried syncing the Nextcloud directory to my local machine, and pointing to that using the file system target. It syncs quickly and gets all the notebooks, but does not correctly display images.

[laurent22]

Any error in the log? https://github.com/laurent22/joplin/blob/master/CONTRIBUTING.md

[ghost]

My apologies -- I didn't enable debugging. Will do so and report back.

Just as a followup -- I left Joplin up and running overnight. It never progressed past the first entries, but I did note (which I didn't see before) that it was on item 5/247.

I'll look for logs later this afternoon, when I get back to that machine.

[laurent22]

There's some info on how to enable debugging and view logs there - https://github.com/laurent22/joplin/blob/master/readme/debugging.md

Please have a look at it but from the sound of it you simply have many notes and/or a slow connection or large notes or resources, so it takes time. Also keep in mind that there's a throttle mechanism in Nextcloud if you fail to login multiple times, which really slow things down.

[ghost]

Hello again.

I've now followed the instructions for debugging -- created the flags.txt file in ~/.config/Joplin with the line: --open-dev-tools --log-level debug and restarted the application. I don't see any "Developer tools" or a "Console" tab.

As for your other thoughts --

• This is all being done on my home LAN.
• There are fewer than 200 notes total. Some do have images and documents associated, but none more than a few meg at most.
• I've synced the entire folder from Nextcloud to the local machine in less than 1 minute, so network speed isn't the problem.
• Nextcloud is not reporting any failed login attempts -- which I would think should have been a problem when verifying the connection in Joplin.
• I'm not having any difficulty with other WebDAV enabled applications (e.g. KeePass, LibreOffice, Nextcloud Sync, etc.)

Again, I appreciate your time and effort! I'd really like to get this working across my several systems so I can finally eliminate Evernote.

Bill

[jcgerhard]

The flags.txt has to be put in .config/joplin-desktop

[ghost]

OK -- just so you know, here's what it says in the document you linked to:
Desktop application

Add a file named "flags.txt" in the config directory (should be ~/.config/joplin or c:\Users\YOUR_NAME.config\joplin) ...

I've moved it to joplin-desktop, restarted, and am seeing the debugging window... which looks straight out of chrome, and there are log files in the .config/joplin-desktop folder.
Here's the log file. If you need the database log file, let me know so I can remove my credential info from it.

log.txt

[ghost]

More strange behavior... when I click to a different "page" (for example, the Tools->General Options item, and then go back, the sync count increases -- and I'm seeing more items in the "joplin-desktop/resources" directory. It's as if forcing a page reload calls the sync function and pulls the next item.

[edit] Clicking the "layout" link reloads the page and increments the sync counter. Cycling through the layouts looks like it is downloading everything - one item at a time.

[edit 2] I just kept clicking the "layout" link. Eventually everything was synced to the local computer - including notebooks, notes, and tags. After syncing everything, I went to my Windows computer, added a couple of notes with images, let it sync (which works perfectly on Windows). Back to the Linux box, and had to do the "layout" clicking routine again, downloading one item at a time.

[laurent22]

The layout routine, is it on Linux only or does it do it on Windows too?

[blissb]

It was never necessary on Windows, as things sync as expected. Only on the Linux systems did I use that "work around."

[kristog]

Hello, I also have the same issue on linux (archlinux with gnome desktop)

The fetched items count increments only when I do some actions in the joplin interface, it can be switching layout or clicking on tags or notebooks : one click results in one fetched item incrementation.

No problem on windows or android.

log.txt

[wdaniel]

Same issue on OpenSuse Tumbleweed (kernel 4.16) with xfce desktop
no problem on Android, Windows 7, Mac OS X High Sierra

Sending F10 every second speeds syncing up.

[redam]

Same issue here on Ubuntu 18.04, doing some actions (hiding/showing sidebar for example) make joplin finishes the sync while I waited several minutes for it to sync only 1 item.

[Shamp0o]

Same on Mint 19. Works fine on Windows and Android.

[PhilippWoelfel]

Same on OpenSuSE 15.0. With both, Dropbox and Nextcloud sync

[caseyhoward]

Broken for me in Mint 19 and Fedora 28 also. Mounting with davfs2 and using the local directory option seems to work though. The Nextcloud setting is also broken.

[dev4223]

Same issue on OpenSuse Leap 42.3

[dev4223]

This triggering effect has to be done only for the notes. After the notes are synced the other items (tags, ...) sync by joplin itself.

[laurent22]

The fetched items count increments only when I do some actions in the joplin interface, it can be switching layout or clicking on tags or notebooks : one click results in one fetched item incrementation.

Could this be related to some power saving feature of the OS? What if you disable most of them (just for testing)?

[dev4223]

No, I don't think so, because I was constantly working on my laptop during the sync.

[wdaniel]

Attached tshark log, pulled on the server side, for about 3 minutes, in the middle of a sync
Joplin-synclog-104.txt

192.168.0.96 - server
192.168.0.104 - client

[JPvRiel]

Could this be related to some power saving feature of the OS

Just add, fully disabled power saving (Ubuntu LTS 16.04), locked the desktop, left it overnight, and the next morning, it seemed the sync count resumed (no progress). This confirms @dev4223's assumption

No, I don't think so, because I was constantly working on my laptop during the sync.

There's also ani impact on E2EE and setting up other clients. One of the first things is should priority sync is the master key. So far everything else was synced before the master key, so got stalled setting up E2EE on other devices waiting for this to happen. The mobile client event reports "decrypting" while synching without having the master key installed (will need to log a separate bug for that). I see as per Encryption Sync Issues, that doing just a single note setup with E2EE before evernote import a good workaround.

[JPvRiel]

Workaround for slow initial bulk sync after evernote import? is a hack I tried and it worked out well in my case.

TL;DR: Rather use local file system sync pointed at the Dropbox/App/Joplin folder. And for Android, the "Offline files" option in the Dropbox app get stored in an sdcard scratch folder that helps do something similar, but after getting the bulk sync done, switch back to Dropbox API sync method because Dropbox on mobile isn't watching that scratch folder for updates to files.

Instead of 2 days of fighting with sync, I was all done after a few minutes.

I also noticed that local file system sync doesn't seem to stall without the UI being in focus, or at least it's so fast, it doesn't matter that much, so maybe the bug here is quite specific to the Joplin Dropbox sync "driver".

Syncing via the Dropbox native file sync app on the OS goes much faster.

[laurent22]

Idea for a fix: on Linux run forceUpdate() at regular interval while syncing. Also remove energy-saving feature which doesn't help fixing this bug and cause other issues.

[Shamp0o]

I installed Manjaro (xfce) on two machines last week and haven't faced any UI freezes or sync problems. Mint (xfce) had caused those problems before as stated above. Used appimage in both distros as the AUR package failed to build.
Maybe this helps.

[kstevens715]

I'm running Joplin 1.0.114, which I believe includes e4166e9 but I'm still experiencing this issue on Ubuntu 18.04 and syncing to NextCloud 13. I have a couple hundred notes to sync from my laptop to NextCloud but I have to click around in the user interface for progress to be made. Each time I click on something a file will upload. But if I don't click for 5 minutes, it stalls out the entire time until I click on something again. And it's not just the UI not updating as I check in NextCloud and sure enough a file only gets uploaded when I click on something in Joplin.

I've been messing with this all day, but just found this GitHub issue. I'm about to call it a night, but when I get a chance I'm going to do some more digging.

[neolovich]

I'm running Joplin 1.0.114 too on Linux Mint 19. Nextcloud 14.0.3.0, PHP 7.0.32. Syncing takes about a view minutes for one or two notes. The android app on LineageOS 15.1 works well so far. Takes also its time (20 to 30 seconds i guess) but it works.
I tried the workaround JPvRiel suggested (Save the notes local on the computer and synchronize with nextcloud client) but then synchronization of the pictures doesnt work (looks like a "broken link" then).
Is there any possibility to make the synchronization on a Linux machine a bit faster? Or as a alternative: Get the synchronization via nextcloud client working including the pictures?

[laurent22]

Note: for now code in ElectronClient/app/gui/SideBar.jsx has been removed as, in fact, I don't think it was ever enabled due to some logic issue. Since there was no recent report for this bug it's possible that a recent Electron upgrade just fixed it.