Cannot load string streams with more than one channel

[lgarbar]

The read_xdf function seems to work without problem for some .xdf files and not others. Listed below is the full error message from my Pluto notebook. Importantly, for the files that fail and the ones that succeed don't seem to have any reliable difference in size (given that they're from the same task performed by different participants) and compile the same number of streams and headers. Any insight would be appreciated.

Error message:
MethodError: no method matching read(::IOStream, ::Nothing)
Closest candidates are:
read(::IOStream)
@ Base iostream.jl:519
read(::IOStream, !Matched::Type{Float64})
@ Base iostream.jl:422
read(::IOStream, !Matched::Type{Float32})
@ Base iostream.jl:421
...
(::Main.var"workspace#18".myXDF.var"#1#3"{Dict{Int64, Int64}, Dict{Int64, Any}})(::IOStream)@my_XDF.jl:130
var"#open#424"(::Base.Pairs{Symbol, Union{}, Tuple{}, @NamedTuple{}}, ::typeof(open), ::Main.var"workspace#18".myXDF.var"#1#3"{Dict{Int64, Int64}, Dict{Int64, Any}}, ::String)@io.jl:396
open(::Function, ::String)@io.jl:393
read_xdf(::String, ::Bool)@my_XDF.jl:36
read_xdf(::String)@my_XDF.jl:33
top-level scope@Local: 8

[cbrnr]

Can you provide a minimal reproducible example with a file that does not work (i.e. no notebook) and then post the stack trace?

Did you try if e.g. pyxdf can read your files?

[lgarbar]

Hello,

The datasets that I've seen fail are participant data, so I won't be able to share any of those data and certainly not through a public forum. Perhaps I'll check with my supervisor if it's acceptable to share fully de-identified and encrypted data with you privately so you can take a look. Otherwise, the issue appears for some files and not others, so I can try to record some dummy data to see if I can reproduce the error and share that instead.

Otherwise, I have had no issues with pyxdf, but it is significantly slower in my experience than unpacking with Julia, so we're looking to potentially make a switch.

[cbrnr]

It would be great if you recorded some dummy data to reproduce the problem. Otherwise, it will be almost impossible to determine what the issue is.

[lgarbar]

How might you suggest sharing the data file? The dummy data I have here is >100mb, so I can't attach it here nor push it to a repo.

[cbrnr]

Do you have Google Drive, OneDrive, Dropbox, etc.? If not, I can share an upload link.

[lgarbar]

Sorry, I guess I meant if you had a preference. Here's a onedrive link with the data, jl script, and stack trace listed in a txt file

[cbrnr]

This link doesn't work, it seems to be restricted for internal use.

[lgarbar]

Makes sense. Try this link

[cbrnr]

Yes, that link worked!

[cbrnr]

Something's wrong when reading chunk 9 (stream ID 5, chunk type 3). Somehow, there's a length of 17 bytes where there should be either 1, 4, or 8 bytes. This could be due to two reasons (I can think of): (1) it's a bug or (2) string streams are not supported yet. I'm still trying to figure it out...

[lgarbar]

Hmm that's interesting. I imagine the issue itself isn't with the datatype being strings since I'm able to unpack our other data of the same format and with string streams.

[cbrnr]

It definitely has to do with your data containing a regularly sampled string stream. I have never seen this in the wild (and hence I don't have any testing data), so I did not implement support for it (yet). Of course, since these are perfectly valid in XDF, these should be supported, but someone will need to implement it. Currently, I don't have the time to do that, but I'll at least mention this limitation in the README.

[lgarbar]

Interesting. I'll try collecting a set of dummy data that can be successfully read to see if you can identify any systemic differences between the two datasets.

[abcsds]

Oh, I'm just reading this now. Hi @lgarbar I implemented the string markers and can help you debug this if you're still interested.

Let me tell you how I'm picking up this issue: I downloaded the file you provide, a 481.1 Mb xdf. A stream, FaceVideo, with 25344 channels of type float32. Could you please confirm this is the file? I have a sha256 of "836ef458ba94bfa95a5fc1f42093cf671e012632435e5159cb12853335962dbe".

Upon reading chunk by chunk with the debugger I found no issue, but integrating it to tests I get weird error messages about the read() function call in L36

The function `read` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  read(::Any, ::Type{Union{}}, Any...; kwargs...)
   @ Base io.jl:236
  read(::IO, ::Type{Char})
   @ Base io.jl:962
  read(::IO, ::Type{T}) where T<:Enum
   @ Base Enums.jl:22
  ...

These type of errors are common on my linux ditsros, and I usually solve them with a container. I'll test again the file you sent, but It would be of much help if you can tell us more about the origin of the data: was it created by a provider? the LibLSL libraries? Is it working if you use the pyxdf?

Thanks for letting us know about the issue.

@cbrnr It might be better to pause #11 if this is a bug in the code. I hadn't seen this issue. Next time please feel free to tag me about bugs, specially if you think they might be in code I implemented. Still, I don't see string streams in the file, just the float32 stream. Is it the same file you revised?

[cbrnr]

@abcsds could it be because you are assuming that there is only one channel? Stream 5 has 22 channels and dtype string...