Enable trilinear filtering on OpenGL

[rom1v]

This PR aims to improve downscaling quality. See #40 (comment) for more details.

• add an option --render-driver to force a renderer
• enable trilinear filtering if the renderer is OpenGL and if it supports mipmapping
• add an option --no-mipmaps to force-disable trilinear filtering

scrcpy --window-height=500 --window-title=BEFORE --no-mipmaps
scrcpy --window-height=500 --window-title=AFTER

cc @cjxgm

I'm hesitant to set opengl as the default render driver on Windows and macOS. I'm afraid that it could cause problems. What do you think?

Please test with --render-driver=opengl on these platforms, and report any problems :)

Note that this improves quality only for client downscaling (the device sends the video at full definition, and the client resizes the window).
The quality is still bad if the content is resized before encoding (scrcpy -m 500 for example).

[srevinsaju]

@rom1v . I tried playing pixel art games, however the trilinear filter affects the clarity of the pixel art games. I tried this branch out. Tested on Windows 10 20H1 (insiders). It did not happen on 1.12.1

[rom1v]

Thank you for your test. Could you post screenshots showing the difference, please?

[srevinsaju]

Sure, I will post the screenshots in some time. I apologize the delay

[srevinsaju]

@rom1v, after the update, the scrcpy on my desktop has been a lot slower. I am able to act faster on the older scrcpy and not on the mipmaps branch

Before

After

The change is hardly noticeable, but I would like to have triliinear filter as the optional args

Sorry for the bad screenshots, but is there any way to fix the lag on scrcpy now.

These are my logs, if it could help:

INFO: scrcpy 1.12.1 <https://github.com/Genymobile/scrcpy>
x/server/scrcpy-server: 1 file pushed, 0 skipped. 0.0 MB/s (27718 bytes in 0.622s)
[server] INFO: Device: samsung GT-N7100 (Android 9)
INFO: Renderer: opengl
INFO: OpenGL version: 4.6 (Compatibility Profile) Mesa 20.0.4
INFO: Trilinear filtering enabled
INFO: Initial texture: 720x1280

[rom1v]

Sorry for the bad screenshots

Is your screenshot zoomed from what see on your computer, or is it a 1:1 screenshot from your computer screen?

I am able to act faster on the older scrcpy and not on the mipmaps branch

Oh, interesting. Which platform?

Does adding --no-mipmaps solve your issue?

The change is hardly noticeable, but I would like to have triliinear filter as the optional args

It is optional (but enabled by default for now), with --no-mipmaps.

Since it improves text quality, I would like to keep it enabled by default, but it depends on whether it has too many drawbacks.

[srevinsaju]

Yes, both are a 1:1 screenshot taken on the PC.

Does adding --no-mipmaps solve your issue?

Nope, I did a few restarts on my phone, to ensure nothing's hanging around.
To clarify, only scrcpy client is having the lag. My phone is super fast. I am not sure if I did something wrong

Oh, interesting. Which platform?

What is meant by platform

[rom1v]

Nope, I did a few restarts on my phone, to ensure nothing's hanging around.

And on dev, it works fine?

[srevinsaju]

Nope, I did a few restarts on my phone, to ensure nothing's hanging around.

And on dev, it works fine?

I meant stable. (master). I am not on dev. scrcpy 1.12.1 is super fast for me. Just tested after the reboot. Until the reboot, everything was slow. After reboot, everything's fast again. I just tested the mipmap again, and the whole scrcpy becomes slow again.

By slow again, I mean, it becomes slow on all of my scrcpy versions (i.e on mipmap as well as 1.12.1)

PS: Where does scrcpy push the server? Is there any way to kill the server if its still running, I suspect its still running. I have root, and termux, so it might be beneficial

[rom1v]

Could you compare dev (this PR is based on dev) and master, please?

If it's slow on dev but fast on master, we need to find the culprit commit (with git bisect or something).

[srevinsaju]

Ok, will check in 5 min. (My builds are really slow). I guess I should do a reboot before the dev build

[srevinsaju]

@rom1v PS: I tested it again. I guess the slow thing came over in dev branch. I tested it again and again to be sure. The mipmap and the dev are slower than the master branch. As far as the testing I did. I have repeated the tests many times.

[rom1v]

OK, thank you. So could you git bisect to find which commit introduced the slowdown between master and dev, please?

[srevinsaju]

I am not familiar with git bisect. I might take some to learn how it works. My bad 🤕

[rom1v]

git checkout dev
git bisect start
git bisect bad  # slowness problem on dev branch
git checkout master
git bisect good  # works correctly on master
# loop:
# it checkouts some commit between master and dev
# build and run
ninja -Cx && ./run x
# if it's slow: git bisect bad
# if it works correctly: git bisect good
# if it knows which commit introduced the problem, it will tell you
# otherwise, goto loop

[srevinsaju]

woah. cool. thanks for this cool snippet. I will let you know asap.

[srevinsaju]

Ok, my findings seems contradictory.
The lag starts somewhere near ee28947 and 927d655.

But the git diff gives normal variables, and the change you have made in the to commits cannot be expected to create a Lag. I am not good at C, however

[srevinsaju]

Until 1c6207f, it works quite smooth

[rom1v]

So 1c6207f works fine, but from ee28947 it lags? Does fd63e7e make a difference compared to ee28947?

Indeed, it is weird 😕 And I cannot reproduce 😞

[rom1v]

So 1c6207f works fine, but from ee28947 it lags? Does fd63e7e make a difference compared to ee28947?

@srevinsaju Could you please test this if you have time?

[rom1v]

Merged in dev.

[srevinsaju]

@rom1v I apologize. I had a sudden inflow of online educational classes which messed up my open source work temporarily. I am extremely sorry that I couldn't do anything yesterday.
Should I test #1284 (comment) as you have asked? I am now free.

I am sorry again. 😢

[rom1v]

@srevinsaju Don't be sorry, everyone has other things to do 😉

I would like you confirm since which commit between master and dev you encounter slowness problems.

[srevinsaju]

Ok, I will let you know it asap.

[cjxgm]

It might be better if the mipmap is only enabled when the scaling factor is below a threshold (e.g. when computer-window-width < 0.75 * phone-width), this should fix the pixel art being blurry issue.

[srevinsaju]

@rom1v I tested it on three devices. On three devices, I did not get the same results. I did a repeated iteration of git bisect over commits from master and dev. I found that the lag still happens as mentioned here #1284 (comment) on Samsung Galaxy Note 2, but it doesn't happen on Note 5 / Samsung A 70. Maybe the lag might be due hardware issues.

[rom1v]

Sid you find a specific commit where it strt to happen?

Also, is it a lag on touch events, mouse events, keyboard events...?

[srevinsaju]

Its a lag on FRAME_RATE. The touch events works fine.

---

Totally unrelated to the PR; I am building an AppImage for scrcpy, after some trial and error, I have tried to find the libraries which are required by scrcpy.
Out of all of the libraries found by ldd, I have selected these:

This is my shortlist

libSDL2-2.0.so.0     libcrystalhd.so.3  libpanelw.so.5      libva-drm.so.2
libXau.so.6          libffi.so.5        libreadline.so.5    libva-x11.so.2
libXdmcp.so.6        libgdbm.so.2       libshine.so.3       libva.so.2
libavcodec.so.57     libgme.so.0        libsnappy.so.1      libvdpau.so.1
libavformat.so.57    libgsm.so.1        libsndio.so.6.1     libvpx.so.5
libavutil.so.55      liblzma.so.0       libsqlite3.so.0     libwebp.so.6
libbluray.so.2       libncursesw.so.5   libssh-gcrypt.so.4  libx264.so.152
libbz2.so.1          libnsl.so.1        libswresample.so.2  libx265.so.146
libchromaprint.so.1  libopenjp2.so.7    libtcl8.4.so        libxvidcore.so.4
libcrypt.so.2        libopenmpt.so.0    libtk8.4.so         libzvbi.so.0

These were the list given by ldd

libasound.so.2       libfontconfig.so.1      libk5crypto.so.3        libpixman-1.so.0        libtheoradec.so.1       libx265.so.146
libasyncns.so.0      libfreetype.so.6        libkeyutils.so.1        libpng16.so.16          libtheoraenc.so.1       libXau.so.6
libavcodec.so.57     libgcc_s.so.1           libkrb5.so.3            libpthread.so.0         libtwolame.so.0         libxcb-render.so.0
libavformat.so.57    libgcrypt.so.20         libkrb5support.so.0     libpulsecommon-11.1.so  libunistring.so.2       libxcb-shm.so.0
libavutil.so.55      libgdk_pixbuf-2.0.so.0  liblz4.so.1             libpulse.so.0           libuuid.so.1            libxcb.so.1
libblkid.so.1        libgio-2.0.so.0         liblzma.so.5            libresolv.so.2          libva-drm.so.2          libXcursor.so.1
libbluray.so.2       libglib-2.0.so.0        libmount.so.1           librsvg-2.so.2          libva.so.2              libXdmcp.so.6
libbsd.so.0          libgme.so.0             libmp3lame.so.0         librt.so.1              libva-x11.so.2          libXext.so.6
libbz2.so.1.0        libgmodule-2.0.so.0     libmpg123.so.0          libSDL2-2.0.so.0        libvdpau.so.1           libXfixes.so.3
libcairo.so.2        libgmp.so.10            libm.so.6               libselinux.so.1         libvorbisenc.so.2       libXinerama.so.1
libchromaprint.so.1  libgnutls.so.30         libnettle.so.6          libshine.so.3           libvorbisfile.so.3      libXi.so.6
libcom_err.so.2      libgobject-2.0.so.0     libnsl.so.1             libsnappy.so.1          libvorbis.so.0          libxkbcommon.so.0
libcroco-0.6.so.3    libgomp.so.1            libnuma.so.1            libsndfile.so.1         libvpx.so.5             libxml2.so.2
libcrystalhd.so.3    libgpg-error.so.0       libogg.so.0             libsndio.so.6.1         libwavpack.so.1         libXrandr.so.2
libc.so.6            libgraphite2.so.3       libopenjp2.so.7         libsoxr.so.0            libwayland-client.so.0  libXrender.so.1
libdatrie.so.1       libgsm.so.1             libopenmpt.so.0         libspeex.so.1           libwayland-cursor.so.0  libXss.so.1
libdbus-1.so.3       libgssapi_krb5.so.2     libopus.so.0            libssh-gcrypt.so.4      libwayland-egl.so.1     libxvidcore.so.4
libdl.so.2           libharfbuzz.so.0        libp11-kit.so.0         libstdc++.so.6          libwebpmux.so.3         libXxf86vm.so.1
libdrm.so.2          libhogweed.so.4         libpango-1.0.so.0       libswresample.so.2      libwebp.so.6            libz.so.1
libexpat.so.1        libicudata.so.60        libpangocairo-1.0.so.0  libsystemd.so.0         libwrap.so.0            libzvbi.so.0
libffi.so.6          libicuuc.so.60          libpangoft2-1.0.so.0    libtasn1.so.6           libX11.so.6
libFLAC.so.8         libidn2.so.0            libpcre.so.3            libthai.so.0            libx264.so.152

Have I missed out any important libs from my shortlist? I have tested my shortlist on Ubuntu 20.04 and Arch Linux.

[rom1v]

Its a lag on FRAME_RATE.

Oh! If you record with scrcpy -r file mkv, and play the fike, does it still lag?

Did you find a specific commit where it lags, and works fine on its parent commit?

---

Scrcpy only depends on ffmpeg and sdl, the others are transitive depencies, depending on your ffmpeg build configuration. Ideally, you should build your own ffmpeg with only the minimum required for scrcpy (H.264 decoding, and probably other stuff from libavutil...).