Skip to content

error: bad char in struct format in read_plain_int96 #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mdemoss opened this issue Dec 14, 2016 · 2 comments
Closed

error: bad char in struct format in read_plain_int96 #49

mdemoss opened this issue Dec 14, 2016 · 2 comments

Comments

@mdemoss
Copy link

mdemoss commented Dec 14, 2016

"<qi" * count produces something like <qi<qi<qi

The docs indicate that the first character of the format string can be used to indicate the byte order, size and alignment.

I've tested potential fixes for this but I suspect the results may be incorrect because the values aren't what I expected. I tried calling an old version of read_plain_int96 multiple times, but that didn't produce values I expected either.

Does anybody have a good test case for this or know what the format ought to be?

(most recent call last):
  File "/usr/lib64/python2.7/pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib64/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/usr/lib64/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "transformParquet.py", line 1, in <module>
    import parquet
  File "/home/ec2-user/poll-pull-transform-parquet/pptp/local/lib/python2.7/site-packages/parquet/__init__.py", line 379, in DictReader
    for row in reader(fo, columns):
  File "/home/ec2-user/poll-pull-transform-parquet/pptp/local/lib/python2.7/site-packages/parquet/__init__.py", line 433, in reader
    dict_items = read_dictionary_page(fo, ph, cmd)
  File "/home/ec2-user/poll-pull-transform-parquet/pptp/local/lib/python2.7/site-packages/parquet/__init__.py", line 359, in read_dictionary_page
    page_header.dictionary_page_header.num_values)
  File "/home/ec2-user/poll-pull-transform-parquet/pptp/local/lib/python2.7/site-packages/parquet/encoding.py", line 88, in read_plain
    return conv(fo, count)
  File "/home/ec2-user/poll-pull-transform-parquet/pptp/local/lib/python2.7/site-packages/parquet/encoding.py", line 46, in read_plain_int96
    items = struct.unpack("<qi" * count, fo.read(12) * count)
error: bad char in struct format
jcrobak added a commit that referenced this issue Dec 18, 2016
Previously, an incorrect format string was used and only the first
12 bytes were read from the stream.

Refs #49.
@jcrobak
Copy link
Owner

jcrobak commented Dec 18, 2016

Thanks for the report. Looks like there was more than one bug in that parsing code. Putting together a fix now.

jcrobak added a commit that referenced this issue Dec 18, 2016
Previously, an incorrect format string was used and only the first
12 bytes were read from the stream.

Refs #49.
jcrobak added a commit that referenced this issue Dec 18, 2016
Previously, an incorrect format string was used and only the first
12 bytes were read from the stream.

Refs #49.
@jcrobak
Copy link
Owner

jcrobak commented Dec 18, 2016

Should be fixed by #50 . Please reopen if you're still seeing issues!

@jcrobak jcrobak closed this as completed Dec 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants