SAS doesn't support byte-compress SPSS files

[Robert-Krajcik]

I had a process in place to send R data to SAS by way of an SPSS dataset, using the "inefficient uncompressed format."
With Haven 2.4.0 and above, the data read into SAS is now garbled.

Why wasn't there an option for doing the byte compressed format? Or at least keeping the uncompressed format?
SAS Proc Import cannot read this new format.
This is not something the authors of "The Pragmatic Programmer" would approve of -- blindly making changes to code and having something downstream you are not aware of break.

sample R script:

library(dplyr)
library(haven)

data("iris")
set.seed(34567)
sampl_iris <- iris %>% 
  group_by(Species) %>% 
  slice_sample(n = 5) 

xdir <- "."
write_sav(sampl_iris,file.path(xdir, "irisdata.sav"))

Sample SAS code:

filename sav_file "./irisdata.sav";
proc import out = irisdata
            datafile = sav_file
            dbms = SAV replace;
run;

proc print data=irisdata;
title1 "sample iris data";
run;

[hadley]

Actual link to change: #544

@oliverbock it seems like SAS doesn't support the byte code compressed spss format. Any thoughts?

[oliverbock]

It surprises me that SAS does not support byte-code compressed .sav files, but I guess it is a consequence of this format not being publicly documented. (It is not much more effort to code.) I looked at SAS documentation and unfortunately it does not contain any details on which .sav variants it supports.

Note that my experience comes mainly from the market research world, where SPSS is the dominant platform for exchanging survey data, because SPSS itself is/was widely used.

The problem as I see it is that we have three SPSS file variants, but only two boolean values for the 'compress' parameter to write_sav().

1. Uncompressed .sav files. Uncommon in my experience, but definitely seen.
2. Byte-compressed .sav files. This is the default produced by SPSS and is by far the most common variant, in my experience.
3. The new zlib-compressed .zsav files. I have never seen these coming from my customers, and it won't be supported by most non-SPSS .sav readers, but it offers substantial file size advantages, and may become more popular.

Perhaps the 'compress' parameter could be allowed to take:

• FALSE (READSTAT_COMPRESS_NONE)
• TRUE (READSTAT_COMPRESS_ROWS)
• "zsav" (READSTAT_COMPRESS_BINARY)

I would change the default from FALSE to TRUE because of the dominance of the byte-compressed format, but I can see that others might disagree.

@hadley , would that work for you?

[hadley]

Maybe we should switch the allowed values to three strings allowing FALSE and TRUE for backward compatibility? I’m fine with continuing to use rows as default since it only seems to have affected one person.

Something like this:

read_spss <- function(compress = c("rows", "none", "binary")) {
  
  if (isTRUE(compress)) {
    compress <- "rows"
  } else if (isFALSE(compress)) {
    compress <- "none"
  } else {
    compress <- arg_match(compress)  
  }
}

[gorcha]

Hey @hadley, that sounds good to me. I'll chuck together a PR.

[gorcha]

I'm thinking options should be "byte", "none" and "zsav" (since rows and binary are ReadStat specific terminology), with "byte" as default.

For backwards compatibility FALSE = "none" and TRUE = "zsav", which is consistent with haven pre-2.4.0.

@oliverbock does that sound good to you?

For the record, I'm extremely surprised that SAS can't read the byte compressed format. It's been the default output format from SPSS for as long as I can remember and I've never had any complaints from SAS users after sending them byte compressed files output from SPSS. Maybe it's a ReadStat thing?

[oliverbock]

Hi @gorcha, yes that sounds perfect.

[gorcha]

Hey @hadley I think we're good to close this unless you have any feedback on the changes (51649c9)?

[hadley]

@gorcha LGTM

[gorcha]

Thanks!