read_Sav regression "Unable to convert string to the requested encoding (invalid byte sequence)"

[pdbailey0]

In haven 2.4.0, 2.4.1 (And 2.4.1.9000) I get an error when reading in the International Association for the Evaluation of Educational Achievement's ePIRLS data available here.

After unzipping that to ~/ePIRLS/2016/ I do

x <- haven::read_sav("~/ePIRLS/2016//asausae1.sav")
# Error: Failed to parse [snip]/ePIRLS/2016/asausae1.sav: Unable to convert string to the requested encoding (invalid byte sequence).

however, with haven 2.3.1 it reads in without errors. If I write it out with the haven 2.3.1 write_sav, then haven 2.4.1 can read in that file cleanly.

[pdbailey0]

Maybe this is addressed in the read_dta/read_stata documentation which reads,

If you encounter an error such as "Unable to convert string to the requested encoding", try encoding = "latin1"

and this runs cleanly (no error on exit)

x <- haven::read_sav("~/ePIRLS/2016//asausae1.sav", encoding="latin1")

[pdbailey0]

Nevertheless, it's odd that there is nothing in the NEWS since 2.1.0 about read_sav but changed behavior. Is there a way to see what encoding is being used?

[hadley]

Unfortunately there's no easy way to track down what went wrong here 😞

[Deleetdk]

The file here https://worldsofjournalism.org/data-d79/data-and-key-tables-2012-2016/ produces the same error, and it solved by @pdbailey0's suggestion above.

> woj2 = read_sav("data/WJS2 open V4-02 030517.sav")
Error: Failed to parse /science/projects/world of journalism/data/WJS2 open V4-02 030517.sav: Unable to convert string to the requested encoding (invalid byte sequence).
> woj2 = read_sav("data/WJS2 open V4-02 030517.sav", encoding = "latin1")

[pdbailey0]

It would be really great to be able to see what encoding is being used when one is not set.

[sam-crawley]

Hi,

The problem with the suggested workaround is that it can break encoding. e.g. loading Afrobarometer Wave 6 (available here: https://afrobarometer.org/data/merged-round-6-data-36-countries-2016), in haven 2.3.1:

afb6 <- read_sav("merged_r6_data_2016_36countries2.sav")

levels(haven::as_factor(unique(afb6$COUNTRY)))[[25]]
[1] "São Tomé and Príncipe"

But in haven 2.4.3:

afb6 <- read_sav("merged_r6_data_2016_36countries2.sav", encoding = "latin1")

levels(haven::as_factor(unique(afb6$COUNTRY)))[[25]]
[1] "São Tomé and Príncipe"

The file is UTF-8, and may have something broken about it, but loading it as latin1 may not be the solution?

[hadley]

@gorcha does this ring any bells with you? I don't think anything has changed in haven relating to this, so it might be a readstat bug?

[pdbailey0]

haven only started documenting readStat version numbers in the NEWS in haven 2.4.0 (which uses readSav 1.1.5), but the 2.3.0 NEWS mentions sas "any" encoding, so that was probably readStat 1.1.2.

[gorcha]

Hey @hadley, definitely nothing haven related.

I've done a bit of digging (thanks @pdbailey0 for the version tip!) and it's because of this change in ReadStat 1.1.4 WizardMac/ReadStat@a8b0466 - reverting this line to the old code loads this file successfully with the default encoding.

So it looks like something is falling over in iconv, but not sure what exactly. I'll have a poke around and see if I can find something definitive.

[gorcha]

This is pretty obscure, but the short version is SPSS is probably not our friend and doesn't encode UTF-8 properly.

The issue is that SPSS (or at least the version that produced the offending files) appears to store multi-byte unicode characters using the code point (a single byte) instead of code units (which can be 1 to 4 bytes).

The string that's causing the issue in the AfroBarometer file is "VOTAÇÃO", which shows up in row 39619.
Having a look at the raw SPSS file, the hex representation is 56 4f 54 41 c7 c3 4f - the c7 and c3 represent the Ç and à characters respectively.

The problem is that, using à as an example, C3 is the "code point" representation, but the correct UTF-8 encoding is two bytes - C3 83 (see https://en.wikipedia.org/wiki/%C3%83).

So SPSS uses the correct "code point" representations of these two characters, but they're not the correct binary encoding for UTF-8. They should both be stored as multi-byte characters in a correct UTF-8 encoding.

I'm not deep enough in the ReadStat code to know why it was working fine, but it's failing now because it's being forced through iconv (for other very necessary reasons) under the totally fair but incorrect assumption that SPSS was encoding things in the way that it said it was.

You can get it down to the exact offending cell using:

read_sav("~/Downloads/merged_r6_data_2016_36countries2.sav", col_select = "Q29B", skip = 39618, n_max = 1)

@evanmiller can you please have a look?

[gorcha]

@pdbailey0, any idea what version of SPSS produced these files? It could be a version specific thing

[evanmiller]

Just FYI what you are calling the "code point" representation is actually Latin-1, see https://en.wikipedia.org/wiki/ISO/IEC_8859-1

Is SPSS producing files containing both UTF-8 and Latin-1 data?

[evanmiller]

@pdbailey0 If you download the standalone readstat utility, it will report the file's self-reported encoding.

$ readstat binlfp2.sav
Format: SPSS binary file (SAV)
Columns: 8
Rows: 753
Table label: binlfp2
Format version: 2
Text encoding: UTF-8
Byte order: little-endian
Timestamp: 28 Oct 2015 14:34