Reliable binary data extraction from QR codes

[matheusmoreira]

Currently, ZBar tries to guess the character encoding of binary data streams stored in QR codes even though they might not be encoded text. This unconditional character set conversion makes it impossible to recover other types of binary data stored in the QR code. This is described in issue #55.

This pull request adds a new configuration option that instructs decoders not to convert binary data to text and adds support for it to the QR decoder. Now both zbarimg and zbarcam produce the expected results when given the -Sqr.binary option.

[matheusmoreira]

@mchehab Hello! It's been two weeks since I submitted the PR. What are your thoughts?

I completed the oneshot feature as well: #60, no breaking changes. These two features complement each other: raw oneshot mode makes it possible to pipe binary data to other programs with no intermediary processing.

[peter-moran]

I would love to use this!

[peter-moran]

FWIW I tested this by encoding and decoding a .tar.gz and it worked.

echo "hello.txt" > hello.txt
tar -czvf hello.tar.gz hello.txt 
qrencode -8 -o out.png < hello.tar.gz
./zbarimg -q --raw -Sqr.binary out.png > new_hello.tar.gz
tar -xzvf new_hello.tar.gz -C out
cat out/hello.txt

[matheusmoreira]

@peter-moran Thanks, that is useful. I also made a test script when I created issue #55. Here's a simplified version of it that returns successfully when the binary data was successfully decoded:

dd if=/dev/urandom bs=256 count=1 > test.bin
qrencode -8 -o test.png < test.bin
zbarimg --raw -Sbinary -Sdisable -Sqr.enable test.png | head -c -1 > test.dec
cmp test.bin test.dec

I was unsure whether to commit this code. The test script requires either committing images to the repository or a dependency on qrencode. I didn't know if these were acceptable. I see existing test cases are written in C; perhaps a more sophisticated program should be created?

[matheusmoreira]

Rebased on top of the latest master 9c6f6a1. Added a binary data decoding test.

The temporary fix commit can be deleted if #60 is merged and 259a17a is amended with a --oneshot option.

@peter-moran Can you please verify that the test passes on your computer?

[peter-moran]

Tests pass for me

[mchehab]

FWIW I tested this by encoding and decoding a .tar.gz and it worked.

echo "hello.txt" > hello.txt
tar -czvf hello.tar.gz hello.txt 
qrencode -8 -o out.png < hello.tar.gz
./zbarimg -q --raw -Sqr.binary out.png > new_hello.tar.gz
tar -xzvf new_hello.tar.gz -C out
cat out/hello.txt

I tried the above, but got an error:

$ ./zbarimg/zbarimg -q --raw -Sqr.binary out.png > new_hello.tar.gz
dbus[11204]: arguments to dbus_message_iter_append_basic() were incorrect, assertion "_dbus_check_is_valid_utf8 (*string_p)" failed in file ../../dbus/dbus-message.c line 2754.
This is normally a bug in some application using the D-Bus library.

  D-Bus not built with -rdynamic so unable to print a backtrace
Aborted (core dumped)

It seems that you need also to make sure that dbus will handle raw scans properly.

Also, please add the testing scripting to test/Makefile.am.inc, in order for it to be called when make tests is used.

[mchehab]

This pull request adds a new configuration option that instructs decoders not to convert binary data to text and adds support for it to the QR decoder. Now both zbarimg and zbarcam produce the expected results when given the -Sqr.binary option.

Ah, please also add qr.binary to zbarcam-qt. It should be possible to fully set the ZBar options over there.

[matheusmoreira]

@mchehab Thank you for your review.

---

It seems that you need also to make sure that dbus will handle raw scans properly.

Indeed. It appears ZBar is trying to transfer data via D-Bus using a string type. According to the D-Bus specification, NUL bytes cannot occur in the input:

The value of any string-like type is conceptually 0 or more Unicode codepoints encoded in UTF-8, none of which may be U+0000.

D-Bus also supports byte and array types. After opening an array container for bytes, I believe I can use dbus_message_iter_append_fixed_array() to transfer the binary data via D-Bus. I will start working on this soon.

---

Also, please add the testing scripting to test/Makefile.am.inc, in order for it to be called when make tests is used.

The test I committed is part of test/test_examples.sh.in which is already integrated with test/Makefile.am.inc. It verifies that zbarimg outputs a PNG file after decoding a QR code.

test/test_examples.sh is invoked by the check-images target which is a prerequisite of check-local which is a prerequisite of tests. It can be seen in the continuous integration build output:

/home/travis/build/mchehab/zbar/test/test_examples.sh
zbarimg PASSED.

Would you like me to add more tests? It's possible to add the random data and hello.tar.gz tests to test_examples.sh.

---

Ah, please also add qr.binary to zbarcam-qt. It should be possible to fully set the ZBar options over there.

I added an entry to the settings structure:

struct settings_s {
    QString name;
    zbar::zbar_config_t ctrl;
    bool is_bool;
};

static const struct settings_s settings[] = {
    { "binary",        zbar::ZBAR_CFG_BINARY,        true },
}

I thought that'd cause the option to show up since the code loops over this structure.

For example, the SettingsDialog constructor uses the data in that structure to create the widgets:

for (unsigned i = 0; i < SETTINGS_SIZE; i++) {
    if (settings[i].is_bool) {
        QCheckBox *button = new QCheckBox(settings[i].name, this);
        layout->addWidget(button, i, 0, 1, 2, Qt::AlignTop | Qt::AlignLeft);
    } else {
        QLabel *label = new QLabel(settings[i].name);
        layout->addWidget(label, i, 0, 1, 1, Qt::AlignTop | Qt::AlignLeft);
        IntegerSetting *spin = new IntegerSetting(settings[i].name, value);
        layout->addWidget(spin, i, 1, 1, 1, Qt::AlignTop | Qt::AlignLeft);
    }
}

When the dialog is accepted, it loops over all elements of the structure to apply the options:

void accept() {
    for (unsigned i = 0; i < SETTINGS_SIZE; i++)
        zbar->set_config(sym, settings[i].ctrl, val[i]);
    QDialog::accept();
};

[mchehab]

@mchehab Thank you for your review.

It seems that you need also to make sure that dbus will handle raw scans properly.

Indeed. It appears ZBar is trying to transfer data via D-Bus using a string type. According to the D-Bus specification, NUL bytes cannot occur in the input:

The value of any string-like type is conceptually 0 or more Unicode codepoints encoded in UTF-8, none of which may be U+0000.

D-Bus also supports byte and array types. After opening an array container for bytes, I believe I can use dbus_message_iter_append_fixed_array() to transfer the binary data via D-Bus. I will start working on this soon.

Yeah, using a byte array will work for the raw mode. Yet, we should keep the strings on UTF-8 mode, in order to preserve backward compatibility. So, it sounds that it would need to use a different message for raw mode.

Also, please add the testing scripting to test/Makefile.am.inc, in order for it to be called when make tests is used.

The test I committed is part of test/test_examples.sh.in which is already integrated with test/Makefile.am.inc. It verifies that zbarimg outputs a PNG file after decoding a QR code.

test/test_examples.sh is invoked by the check-images target which is a prerequisite of check-local which is a prerequisite of tests. It can be seen in the continuous integration build output:

/home/travis/build/mchehab/zbar/test/test_examples.sh
zbarimg PASSED.

Would you like me to add more tests?

Yes. It seems that the current test didn't check handling the image via dBus. So, when I did a manual test, it failed.

Also, as we'll need to add dBus raw output message using array of bytes, we'll need to change the dBus testing logic to test both UTF-8 and raw outputs.

Ah, please also add qr.binary to zbarcam-qt. It should be possible to fully set the ZBar options over there.

I added an entry to the settings structure:

struct settings_s {
    QString name;
    zbar::zbar_config_t ctrl;
    bool is_bool;
};

static const struct settings_s settings[] = {
    { "binary",        zbar::ZBAR_CFG_BINARY,        true },
}

I thought that'd cause the option to show up since the code loops over this structure.

For example, the SettingsDialog constructor uses the data in that structure to create the widgets:

for (unsigned i = 0; i < SETTINGS_SIZE; i++) {
    if (settings[i].is_bool) {
        QCheckBox *button = new QCheckBox(settings[i].name, this);
        layout->addWidget(button, i, 0, 1, 2, Qt::AlignTop | Qt::AlignLeft);
    } else {
        QLabel *label = new QLabel(settings[i].name);
        layout->addWidget(label, i, 0, 1, 1, Qt::AlignTop | Qt::AlignLeft);
        IntegerSetting *spin = new IntegerSetting(settings[i].name, value);
        layout->addWidget(spin, i, 1, 1, 1, Qt::AlignTop | Qt::AlignLeft);
    }
}

When the dialog is accepted, it loops over all elements of the structure to apply the options:

void accept() {
    for (unsigned i = 0; i < SETTINGS_SIZE; i++)
        zbar->set_config(sym, settings[i].ctrl, val[i]);
    QDialog::accept();
};

Yeah, the code sounds right. Unfortunately, I could test it. I'm currently on a travel using a low-res monitor, not big enough to show all options. That reminds that we should likely add a scroll-bar for the options with low-res monitors.

For the next changes, I'll double check it on a hi-res monitor.

[matheusmoreira]

Now the image scanner outputs a byte array to D-Bus. It does this only if the user set the binary decoding option, so this should not conflict with existing users. I replaced the Data key which had a variant UTF-8 text value with a BinaryData key that has a variant byte array value. Is this an adequate solution that ensures backwards compatibility? Should I keep the Data key?

The code builds with D-Bus enabled and all tests pass, including the current D-Bus checkers. I don't know how to test the correctness of this output though. I get another type of error when I run zbarimg:

$ ./zbarimg/zbarimg --raw --oneshot -Sbinary examples/qr-code-binary.png > qr-code-binary.png.out
Name Release Error (Connection was disconnected before a reply was received)
scanned 1 barcode symbols from 1 images in 0.01 seconds

I know about test/test_dbus.c but I'm currently not familiar enough with D-Bus to modify it. Please give me some time.

[matheusmoreira]

Modified the D-Bus testing code: it now writes the BinaryData to a separate file which the test script checksums. The non-binary Data test passes and then times out while waiting for the BinaryData message. The first invocation of zbarimg produces no error messages but the second prints the same error I got earlier:

Name Release Error (Connection was disconnected before a reply was received)

It seems the data is not being written to the bus. I'm not sure why yet but I suspect the problem is in the code that writes the array to the bus:

dbus_message_iter_open_container(&dict_entry, DBUS_TYPE_VARIANT, "ay", &dict_val);
dbus_message_iter_append_fixed_array(&dict_val, DBUS_TYPE_BYTE, &value, value_length);

Is there a problem with it?

[mchehab]

Now the image scanner outputs a byte array to D-Bus. It does this only if the user set the binary decoding option, so this should not conflict with existing users. I replaced the Data key which had a variant UTF-8 text value with a BinaryData key that has a variant byte array value. Is this an adequate solution that ensures backwards compatibility?

It looks sane to me.

Should I keep the Data key?

No. If the data is not UTF-8 encoded, the best is to only output the raw message.

The code builds with D-Bus enabled and all tests pass, including the current D-Bus checkers. I don't know how to test the correctness of this output though.

Based on your next patch, it seems that you already figured it out.

I get another type of error when I run zbarimg:

$ ./zbarimg/zbarimg --raw --oneshot -Sbinary examples/qr-code-binary.png > qr-code-binary.png.out
Name Release Error (Connection was disconnected before a reply was received)
scanned 1 barcode symbols from 1 images in 0.01 seconds

I know about test/test_dbus.c but I'm currently not familiar enough with D-Bus to modify it. Please give me some time.

Ok. D-Bus is a new feat to ZBar. So, this code is new even to me ;-)

[mchehab]

Modified the D-Bus testing code: it now writes the BinaryData to a separate file which the test script checksums. The non-binary Data test passes and then times out while waiting for the BinaryData message. The first invocation of zbarimg produces no error messages but the second prints the same error I got earlier:

Name Release Error (Connection was disconnected before a reply was received)

It seems the data is not being written to the bus. I'm not sure why yet but I suspect the problem is in the code that writes the array to the bus:

dbus_message_iter_open_container(&dict_entry, DBUS_TYPE_VARIANT, "ay", &dict_val);
dbus_message_iter_append_fixed_array(&dict_val, DBUS_TYPE_BYTE, &value, value_length);

Is there a problem with it?

Didn't test, but in principle looks ok to me. What I do recommend you is to open two terminals. At the first one, you run just:

$ ./test/test_dbus

On the second one, parse different images, on raw and UTF-8 mode, for example running something like:

$ zbarimg/zbarimg examples/<image1>.png
$ zbarimg/zbarimg --raw examples/<image2>.png
...

This way, you can see what each different image (with or with and without --raw) will output.

You may add some additional prints at test_dbus.c,for example to double-check if the test_dbus will not be on some endless waiting loop.

[mchehab]

It sounds hacky to test for a property here. I mean, the D-Bus listener should not rely on anything except on the received messages. See, the dbus testing code: it is possible to test if the received message is a string or not:

if (DBUS_TYPE_STRING != dbus_message_iter_get_arg_type(&dict))

The right way seems to have an else clause there for the bytearray type.

[matheusmoreira]

This line is checking the type of the key. Strings are the only valid keys. The org.linuxtv.Zbar1.Code signal is defined as a dictionary with string keys and variant values:

dbus_message_iter_open_container(&args, DBUS_TYPE_ARRAY, "{sv}", &dict);

The types of the values are currently not checked. I wrote my code following the pattern of the surrounding code. So currently the code checks the key and recurses into the variant value without checking its type. Before this patch, only string values were possible. I agree that it'd be wise to add some type checks.

Since the nature of the data can be distinguished by checking its type, do we need a new BinaryData key? It should be possible to have Data contain both strings and byte arrays and clients should be able to handle both types.

[matheusmoreira]

Is there a problem with it?

Indeed, the problem was there. It's not possible to simply write the array into the variant. An array container must be opened and recursed into:

dbus_message_iter_open_container(&dict_entry, DBUS_TYPE_VARIANT, "ay", &dict_val);
dbus_message_iter_open_container(&dict_val, DBUS_TYPE_ARRAY, "y", &array_val);
dbus_message_iter_append_fixed_array(&array_val, DBUS_TYPE_BYTE, &value, value_length);

After updating the test_dbus.c code with another level of recursion, it works!

$ make check-dbus
  CC       test/test_dbus.o
  CCLD     test/test_dbus
/home/matheus/dev/test-zbarimg/zbar/test/check_dbus.sh
Waiting for Zbar events
Type = CODE-128
Value = https://github.com/mchehab/zbar
Type = QR-Code
BinaryData[210]
D-Bus PASSED.

dbus-monitor output:

array [
   dict entry(
      string "Type"
      variant             string "CODE-128"
   )
   dict entry(
      string "Data"
      variant             string "https://github.com/mchehab/zbar"
   )
]

array [
   dict entry(
      string "Type"
      variant             string "QR-Code"
   )
   dict entry(
      string "BinaryData"
      variant             array of bytes [
            89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 64 00
            00 00 64 01 00 00 00 00 58 99 a8 f9 00 00 00 99 49 44 41 54 38
            4f 9d d3 b1 91 04 01 08 c4 40 65 a0 fc b3 9c 0c f4 c6 de 9d bd
            3c 5e 3b 14 0c 05 80 98 f1 a9 97 b2 2c d2 3a 09 dd 30 f1 ac c0
            ff c8 79 56 39 b0 df 0e ef 04 a8 ea 2f b3 77 aa 72 ac 6a 07 31
            1d f1 a4 7b 90 6c 8c 06 3b a8 50 da b6 38 68 5b 18 31 2e 42 5c
            26 9f a9 df 8a e7 2c cb ed a2 f6 f8 73 96 b7 52 e7 9c cc 93 06
            7c 63 3e a8 ca 9e 7d 2f 02 dc a6 1a 07 59 16 d8 f3 94 af 85 9a
            4f b3 ab 16 d8 38 aa d9 82 93 4a 27 9b 27 01 4f 42 3e 7d 5e ea
            0f 1b 6c f4 b0 5e 84 51 65 00 00 00 00 49 45 4e 44 ae 42 60 82
         ]
   )
]

Perhaps Data can be used for both outputs. Since it's a variant type, there is no guarantee that it will be a string and clients should be checking the type.

[matheusmoreira]

@mchehab Hello! Have you had an opportunity to test the new D-Bus code?

[jerabaul29]

@matheusmoreira do you expect that this feature is fully available now? If you confirm I will give it a try here too :)

[matheusmoreira]

@jerabaul29 Yes. I added a binary decoding option to the ZBar API and implemented support for it in the QR decoder. The zbarimg and zbarcam tools set decoder options according to the API so they have automatically gained support for the binary option. Passing -Sbinary applies the option to all decoders. I also implemented D-Bus support as requested by the maintainer.

I have added automated tests which use these new features to decode a QR code which contains a PNG file. They pass on my machine. Please clone my branch, build the project and run the tests. It'd be greatly appreciated if you could test the Qt GUI as well, I added the binary option to the GUI but I wasn't able to build it.

[jerabaul29]

@matheusmoreira I have some small problems here looks like (probably doing something stupid somewhere). It looks like I successfully compiled:

jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> autoreconf -vfi && ./configure
...
please verify that the detected configuration matches your expectations:
------------------------------------------------------------------------
X                      --with-x=yes
pthreads               --enable-pthread=yes
doc                    --enable-doc=yes
v4l                    --enable-video=yes
jpeg                   --with-jpeg=yes
Python                 --with-python=python2    python2.7
GTK                    --with-gtk=gtk2          Gtk2.24.32
GObject introspection  --with-gir=no
Qt                     --with-qt=no            Qt
Java                   --with-java=no
Dbus                   --with-dbus=no
ImageMagick            --with-imagemagick=check
Enabled codes:         ean databar code128 code93 code39 codabar i25 qrcode sqcode
Disabled codes:        pdf417

        => libv4l will *NOT* be used
        => the Qt widget will *NOT* be built
        => the Java interface will *NOT* be built
        => the Java unit test will *NOT* be enabled
✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> make
make  all-recursive
make[1]: Entering directory '/home/jrlab/Desktop/Git/zbar'
Making all in zbar
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar/zbar'
  CC       libzbar_la-config.lo
  CC       libzbar_la-error.lo
  CC       libzbar_la-symbol.lo
  CC       libzbar_la-image.lo
  CC       libzbar_la-convert.lo
  CC       libzbar_la-processor.lo
  CC       processor/libzbar_la-lock.lo
  CC       libzbar_la-refcnt.lo
  CC       libzbar_la-window.lo
  CC       libzbar_la-video.lo
  CC       libzbar_la-img_scanner.lo
  CC       libzbar_la-scanner.lo
  CC       libzbar_la-decoder.lo
  CC       libzbar_la-misc.lo
  CC       decoder/libzbar_la-ean.lo
  CC       decoder/libzbar_la-databar.lo
  CC       decoder/libzbar_la-code128.lo
  CC       decoder/libzbar_la-code93.lo
  CC       decoder/libzbar_la-code39.lo
  CC       decoder/libzbar_la-codabar.lo
  CC       decoder/libzbar_la-i25.lo
  CC       decoder/libzbar_la-qr_finder.lo
  CC       qrcode/libzbar_la-qrdec.lo
  CC       qrcode/libzbar_la-qrdectxt.lo
  CC       qrcode/libzbar_la-rs.lo
  CC       qrcode/libzbar_la-isaac.lo
  CC       qrcode/libzbar_la-bch15_5.lo
  CC       qrcode/libzbar_la-binarize.lo
  CC       qrcode/libzbar_la-util.lo
  CC       libzbar_la-sqcode.lo
  CC       decoder/libzbar_la-sq_finder.lo
  CC       processor/libzbar_la-posix.lo
  CC       video/libzbar_la-v4l.lo
  CC       video/libzbar_la-v4l2.lo
  CC       libzbar_la-jpeg.lo
  CC       processor/libzbar_la-x.lo
  CC       window/libzbar_la-x.lo
  CC       window/libzbar_la-ximage.lo
  CCLD     libzbar.la
copying selected object files to avoid basename conflicts...
ar: `u' modifier ignored since `D' is the default (see `U')
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar/zbar'
Making all in gtk
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar/gtk'
glib-genmarshal --g-fatal-warnings --prefix=zbar_marshal \
    --body zbarmarshal.list > zbarmarshal.c
INFO: Reading zbarmarshal.list...
glib-genmarshal --g-fatal-warnings --prefix=zbar_marshal \
    --header zbarmarshal.list > zbarmarshal.h
INFO: Reading zbarmarshal.list...
make  all-am
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar/gtk'
  CC       libzbargtk_la-zbargtk.lo
  CC       libzbargtk_la-zbarmarshal.lo
  CCLD     libzbargtk.la
ar: `u' modifier ignored since `D' is the default (see `U')
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar/gtk'
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar/gtk'
Making all in .
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar'
  CC       python/python_zbar_la-zbarmodule.lo
  CC       python/python_zbar_la-enum.lo
  CC       python/python_zbar_la-exception.lo
  CC       python/python_zbar_la-symbol.lo
  CC       python/python_zbar_la-symbolset.lo
  CC       python/python_zbar_la-symboliter.lo
  CC       python/python_zbar_la-image.lo
  CC       python/python_zbar_la-processor.lo
  CC       python/python_zbar_la-imagescanner.lo
  CC       python/python_zbar_la-decoder.lo
  CC       python/python_zbar_la-scanner.lo
  CCLD     python/zbar.la
  CC       zbarimg/zbarimg_zbarimg-zbarimg.o
  CCLD     zbarimg/zbarimg
  CC       zbarcam/zbarcam_zbarcam-zbarcam.o
  CCLD     zbarcam/zbarcam
  CC       zbarcam/zbarcam_zbarcam_gtk-zbarcam-gtk.o
  CC       zbarcam/zbarcam_zbarcam_gtk-scan_video.o
  CCLD     zbarcam/zbarcam-gtk
: --searchpath /home/jrlab/Desktop/Git/zbar/doc -m /home/jrlab/Desktop/Git/zbar/doc/style.xsl --skip-validation  -o doc/man man doc/manual.xml
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
: --searchpath /home/jrlab/Desktop/Git/zbar/doc -m /home/jrlab/Desktop/Git/zbar/doc/style.xsl --skip-validation  -o doc/man man doc/manual.xml
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
: --searchpath /home/jrlab/Desktop/Git/zbar/doc -m /home/jrlab/Desktop/Git/zbar/doc/style.xsl --skip-validation  -o doc/man man doc/manual.xml
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make[1]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> make check
make  check-recursive
make[1]: Entering directory '/home/jrlab/Desktop/Git/zbar'
Making check in zbar
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar/zbar'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar/zbar'
Making check in gtk
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar/gtk'
make  check-am
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar/gtk'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar/gtk'
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar/gtk'
Making check in .
make[2]: Entering directory '/home/jrlab/Desktop/Git/zbar'
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
: --searchpath /home/jrlab/Desktop/Git/zbar/doc -m /home/jrlab/Desktop/Git/zbar/doc/style.xsl --skip-validation  -o doc/man man doc/manual.xml
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
: --searchpath /home/jrlab/Desktop/Git/zbar/doc -m /home/jrlab/Desktop/Git/zbar/doc/style.xsl --skip-validation  -o doc/man man doc/manual.xml
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make  test/test_decode test/test_convert test/test_video test/test_proc test/test_cpp test/test_cpp_img test/test_jpeg 
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
  CC       test/test_test_decode-test_decode.o
  CCLD     test/test_decode
  CC       test/test_convert.o
  CC       test/test_images.o
  CCLD     test/test_convert
  CC       test/test_video.o
  CCLD     test/test_video
  CC       test/test_proc.o
  CCLD     test/test_proc
  CXX      test/test_cpp.o
  CXXLD    test/test_cpp
  CXX      test/test_cpp_img.o
  CXXLD    test/test_cpp_img
  CC       test/test_jpeg.o
  CCLD     test/test_jpeg
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make  check-local
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
/usr/bin/python2 /home/jrlab/Desktop/Git/zbar/test/barcodetest.py
F
======================================================================
FAIL: runTest (__main__.BuiltinTestCase)
/home/jrlab/Desktop/Git/zbar/examples/ean-13.png
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 168, in runTest
    compare_sources(expect, actual[0])
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 248, in compare_sources
    compare_maps(map_source(expect), map_source(actual), compare_indices)
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 217, in compare_maps
    assert not errors, '\n'.join(errors)
AssertionError: Traceback (most recent call last):
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 200, in compare_maps
    compare_func(iexp, iact)
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 269, in compare_indices
    compare_maps(map_index(expect), map_index(actual), compare_symbols)
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 217, in compare_maps
    assert not errors, '\n'.join(errors)
AssertionError: Traceback (most recent call last):
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 200, in compare_maps
    compare_func(iexp, iact)
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 278, in compare_symbols
    assert actual.get('orientation') == orient
AssertionError



----------------------------------------------------------------------
Ran 1 test in 0.019s

FAILED (failures=1)
Makefile:2361: recipe for target 'check-images-py' failed
make[3]: *** [check-images-py] Error 1
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:2121: recipe for target 'check-am' failed
make[2]: *** [check-am] Error 2
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:1831: recipe for target 'check-recursive' failed
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:2124: recipe for target 'check' failed
make: *** [check] Error 2
✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding| …2]> ls
data.bin  data.png  Makefile.am.inc  zbarimg  zbarimg.c  zbarimg.rc  zbarimg_zbarimg-zbarimg.o
✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding| …2]> ./zbarimg -q --raw -Sqr.binary data.png | xxd
ERROR: invalid configuration setting: qr.binary

usage: zbarimg [options] <image>...

[jerabaul29]

PS: I am on Ubuntu 18.04 if this is helpful to know.

[matheusmoreira]

@jerabaul29 That's strange.

I do not know why the Python tests are failing. It seems it didn't make it to the check-images target. Does make check-images fail as well? Those are the tests I modified.

The tools use the zbar_processor_parse_config ZBar API function in order to apply options. The fact it can't recognize the binary option suggests that zbarimg is somehow being linked to an incorrect version of the library. Can you confirm that the binary is using the library that was built from source?

[jerabaul29]

Mmmh, I am probably doing something stupid somewhere, sorry for that, I am not used to build stuff...

The tests failing to build:

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding| …2]> make check-images
/home/jrlab/Desktop/Git/zbar/test/test_examples.sh
FAILED: /home/jrlab/Desktop/Git/zbar/examples/code-93.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of f8f55b828eb7d0400f300be021d29293bd4a3191)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/code-93.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/code-39.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 7537d593ea42393a43bc0eda0a896c0e31017dd8)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/code-39.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/codabar.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of a56811d078ea5cfac9be5deb4b6796177763e152)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/codabar.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/databar.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of aebbdbed0b32d7fd72f1245e3fb384822d492062)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/databar.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/databar-exp.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 9e245874d3229a575eabfdba1c668369c55960e3)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/databar-exp.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/ean-2.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 5ab2b518e2c9d827cedc5825d2e3c9646d43713a)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus -Sean2.enable '/home/jrlab/Desktop/Git/zbar/examples/ean-2.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/ean-5.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 668fef8cb9caac34df8cb8564c2cde62e4af5e65)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus -Sean5.enable '/home/jrlab/Desktop/Git/zbar/examples/ean-5.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/qr-code-inverted.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of b350ca7efad7a50c5ac082d5c683a8e8d8d380a7)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus -Stest-inverted '/home/jrlab/Desktop/Git/zbar/examples/qr-code-inverted.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/qr-code-binary.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of df896e459e47a7d392031a7d4962722a143e276b)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus --raw --oneshot -Sbinary '/home/jrlab/Desktop/Git/zbar/examples/qr-code-binary.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/sqcode1-generated.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 84c0ce7072e2227073dc8bd1e5f4518d8f42ae3d)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/sqcode1-generated.png'
	results: FAILED: /home/jrlab/Desktop/Git/zbar/examples/sqcode1-scanned.png (da39a3ee5e6b4b0d3255bfef95601890afd80709 instead of 84c0ce7072e2227073dc8bd1e5f4518d8f42ae3d)
	cmd: /home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/sqcode1-scanned.png'
	results: Makefile:2364: recipe for target 'check-images' failed
make: *** [check-images] Error 1

[jerabaul29]

I think I am just confused somewhere. I tried to compile locally by following the instructions here:

https://github.com/mchehab/zbar/blob/master/INSTALL.md

then following the Basic Installation section. This is where things fail (doing the tests). Then I simply try to launch the zbarimg script generated by the make command. I suppose it is there some linking etc may be wrong / I should not launch it directly / something like that?

[matheusmoreira]

@jerabaul29 Every test command is outputting the same data. They all have the same checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709. I wonder what's going on there. Please run one of the test commands manually and post the output:

/home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus --raw --oneshot -Sbinary '/home/jrlab/Desktop/Git/zbar/examples/qr-code-binary.png'

[jerabaul29]

(note that I have not run the make install command since the make check failed. I guess this may explain for the failed tests? I do not want to break some packages on my machine by installing some broken packages :( ).

[jerabaul29]

✘-1 jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding|✔]> ls
Makefile.am.inc  zbarimg  zbarimg.c  zbarimg.rc  zbarimg_zbarimg-zbarimg.o
✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding|✔]> ./zbarimg  --nodbus --raw --oneshot -Sbinary '/home/jrlab/Desktop/Git/zbar/examples/qr-code-binary.png'
ERROR: invalid configuration setting: binary

usage: zbarimg [options] <image>...

scan and decode bar codes from one or more image files

options:
    -h, --help      display this help text
    --version       display version information and exit
    -q, --quiet     minimal output, only print decoded symbol data
    -v, --verbose   increase debug output level
    --verbose=N     set specific debug output level
    -d, --display   enable display of following images to the screen
    -D, --nodisplay disable display of following images (default)
    --xml, --noxml  enable/disable XML output format
    --raw           output decoded symbol data without symbology prefix
    -1, --oneshot   exit after scanning one QR code
    -S<CONFIG>[=<VALUE>], --set <CONFIG>[=<VALUE>]
                    set decoder/scanner <CONFIG> to <VALUE> (or 1)

[jerabaul29]

I deeply apologize for my n00bness @matheusmoreira in the very likely case that I am doing something obviously wrong in the process for somebody with more experience but that I am missing...

Any other way for your to share a 'built' version of the package that I can test, or some containerized version, or something in this kind?

[matheusmoreira]

@jerabaul29 The following test should be working since I did not modify it in any way:

/home/jrlab/Desktop/Git/zbar/zbarimg/zbarimg --nodbus '/home/jrlab/Desktop/Git/zbar/examples/code-93.png'

Does it output anything?

I built my copy of ZBar by executing autoreconf -vfi && ./configure --with-dbus && make on a freshly cloned repository. I can use the new features I developed by executing zbarimg/zbarimg instead of the packaged binary I have installed on my system. I did not run make install. I'm not sure if the distribution could be the cause of this but in any case I am using Arch Linux.

---

Turns out da39a3ee5e6b4b0d3255bfef95601890afd80709 is the SHA-1 of the empty string. For some reason, zbarimg is outputting zero bytes to standard output even in the case of previously working tests...

[jerabaul29]

Aah, I think that this is slightly different from the command I used to configure following what was in the INSTALL.MD. Running the commands suggested in this file work fine, but adding your --with-dbus option breaks things as it is now on my computer:

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> git reset --hard HEAD && git clean -fd
HEAD is now at 1a203a4 test: verify types of data sent over D-Bus

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> git remote -v
origin	[email protected]:matheusmoreira/zbar.git (fetch)
origin	[email protected]:matheusmoreira/zbar.git (push)

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> git status
On branch binary-decoding
Your branch is up to date with 'origin/binary-decoding'.
nothing to commit, working tree clean

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> autoreconf -vfi && ./configure --with-dbus && make
...
checking for DBUS... no
configure: error: in `/home/jrlab/Desktop/Git/zbar':
configure: error: DBus development libraries not found
See `config.log' for more details

• maybe the INSTALL.MD should be updated if necessary?
• I install the dbus stuff and let you know.

[jerabaul29]

Ok, installing dbus-dev:

sudo apt install libdbus-1-dev

Lets me run your command for compilation without error.

But when running make check I still get some errors:

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar [binary-decoding|✔]> make check
...
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
make  check-local
make[3]: Entering directory '/home/jrlab/Desktop/Git/zbar'
/usr/bin/python2 /home/jrlab/Desktop/Git/zbar/test/barcodetest.py
F
======================================================================
FAIL: runTest (__main__.BuiltinTestCase)
/home/jrlab/Desktop/Git/zbar/examples/ean-13.png
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 163, in runTest
    actual = run_zbarimg((expect.get('href'),))
  File "/home/jrlab/Desktop/Git/zbar/test/barcodetest.py", line 137, in run_zbarimg
    'zbarimg returned error status (%d)\n' % rc + err
AssertionError: zbarimg returned error status (127)
/home/jrlab/Desktop/Git/zbar/zbarimg/.libs/zbarimg: symbol lookup error: /home/jrlab/Desktop/Git/zbar/zbarimg/.libs/zbarimg: undefined symbol: zbar_processor_request_dbus


----------------------------------------------------------------------
Ran 1 test in 0.011s

FAILED (failures=1)
Makefile:2361: recipe for target 'check-images-py' failed
make[3]: *** [check-images-py] Error 1
make[3]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:2121: recipe for target 'check-am' failed
make[2]: *** [check-am] Error 2
make[2]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:1831: recipe for target 'check-recursive' failed
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory '/home/jrlab/Desktop/Git/zbar'
Makefile:2124: recipe for target 'check' failed
make: *** [check] Error 2

[jerabaul29]

(@matheusmoreira @mchehab if you think I am polluting your push request thread let me know, and I can move / clean my posts to whatever location you think is better suited).

[jerabaul29]

Note: in the current state I cannot do anything:

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding|✔]> ./zbarimg  --nodbus --raw --oneshot -Sbinary '/home/jrlab/Desktop/Git/zbar/examples/qr-code-binary.png'
/home/jrlab/Desktop/Git/zbar/zbarimg/.libs/zbarimg: symbol lookup error: /home/jrlab/Desktop/Git/zbar/zbarimg/.libs/zbarimg: undefined symbol: zbar_processor_request_dbus

Guess this means I am using the right binary?

[matheusmoreira]

symbol lookup error

Seems to be some kind of linking issue. Looks like zbarimg can't find the correct ZBar library. Do these errors happen when building and testing the master branch?

[joernheissler]

Hello,
try setting LD_LIBRARY_PATH and/or LD_PRELOAD.
On my system, the system-wide lib was preferred.

Btw, thanks for this patch. Need it for a project of mine.

[jerabaul29]

@matheusmoreira fails in the exact same way on the master branch.

[jerabaul29]

@joernheissler if I understand well the point with setting a LD_PRELOAD is to force the system to use a given .so object when executing the next command, right? Problem is, I do not see any .so after having run the make command (autoreconf -vfi && ./configure --with-dbus && make), I get:

✔ jrlab-ThinkPad-T490:~/Desktop/Git/zbar/zbarimg [binary-decoding|✔]> ls
Makefile.am.inc  zbarimg  zbarimg.c  zbarimg.rc  zbarimg_zbarimg-zbarimg.o

while I guess I should see and set LD_PRELOAD to a .so present here, right? Possible I am completely wrong and lost, sorry.

[matheusmoreira]

@matheusmoreira fails in the exact same way on the master branch.

Looks like this issue is not specific to my pull request. The build system itself seems to be running into problems on your system. I don't understand autotools well enough to debug this issue. Perhaps there will be people who faced similar issues in a Ubuntu or Debian channel.

Problem is, I do not see any .so after having run the make command

I remember seeing a hidden .libs directory in zbarimg's directory. Try ls -la and if anything turns up try setting LD_LIBRARY_PATH to that directory before running zbarimg so that the dynamic linker will look there first. But honestly, the build system should be handling this in some way.

[mchehab]

Applied.

[ashwanirathee]

@matheusmoreira thank you so much for the feature,@jerabaul29 thank you for discussion on installation in ubuntu 18.04,I was having some of those issues on my ubuntu 20.04,now they're resolved and all checks/tests work.

[matheusmoreira]

@ashwani-rathee You're welcome. Thank the original developers as well. It turns out the binary decoding capability was always there... I merely provided an API for it.

If you managed to resolve the Ubuntu build issues, please provide details on how you did it. Could help future readers!