A problem with idlneturl__define.pro (spawn, cmd, result, blahblah)

[Albom]

An error occurs in idlneturl__define.pro file (line 134).

Trying:

spawn,'curl -LI --silent --show-error --include --write-out "\n%{size_header}\n%{content_type}\n%{response_code}\n%{size_download}" http://www.http2demo.io/',result,eee

Get in eee:

curl: (3) URL using bad/illegal format or missing URL

Get in result:

0 000 00

Running in a command line (cmd):
curl -LI --silent --show-error --include --write-out "\n%{size_header}\n%{content_type}\n%{response_code}\n%{size_download}" http://www.http2demo.io/

Get:

HTTP...
....

392
text/html
200
0

curl 7.78.0 (x86_64-pc-win32) is in PATH. Another version of curl was also tried. The result is the same.
OS is Windows 11 Home 21H2.

[GillesDuvert]

Hi thanks for the report.
This procedure has never been tested under Windows before and the fact that spawning curl does work is already a feat! 😄
Probably the spawn C++ code does unfortunate conversions of the 'sent' string, like replacing / by \ etc on this platform.

[Albom]

Thank you! Yes, really. In basic_pro.cpp (Line 1835), there is a such conversion:

std::replace(cmd.begin(), cmd.end(), '/', '\\');

Therefore, the string http://www.http2demo.io/ is converted to http:\\www.http2demo.io\.

Trying curl in a command line:

curl -LI --silent --show-error --include --write-out "\n%{size_header}\n%{content_type}\n%{response_code}\n%{size_download}" http:\\www.http2demo.io\

And get:

curl: (3) URL using bad/illegal format or missing URL

0

000
0

Is this conversion required? Users should know what kind of slash symbols ('/' or '') to use...

[GillesDuvert]

Yes the call passes OK on unix/linux and probably OSX as well.
GDL is supposed to behave as IDL (or better :smile) so in the particular case of idlneturl:: use GDL must provide a call to curl (external or, in the future, internally using libcurl) that does not make such conversion on windows.
This opens a greater question: does the windows version of gdl's 'spawn' need to make this translation at all? I doubt IDL does it.

[Albom]

Tested in IDL Version 8.5.1, Microsoft Windows (Win32 x86_64 m64):

IDL> spawn,'curl -LI --silent --show-error --include --write-out "\n%{size_header}\n%{content_type}\n%{response_code}\n%{size_download}" http://www.http2demo.io/',result,eee
IDL> print,result
HTTP/1.1 200 OK Date: Wed, 11 Jan 2023 14:07:06 GMT Content-Type: text/html Content-Length: 105216
Connection: keep-alive ETag: "5aa91b6d-19b00" Cache-Control: no-cache Access-Control-Allow-Origin: *
X-Accel-Expires: @1680730520 Server: CDN77-Turbo X-77-NZT: AVm7qxmlHhD/cgxyAQ
X-77-NZT-Ray: 0497bb32127d82118ac2be6343a22b35 X-Cache: HIT X-Age: 24251506 Accept-Ranges: bytes 392 text/html 200 0
IDL> print,eee

IDL>

Works perfectly.

[Albom]

Thank you! I've tested the build after branches merging and execution of the command works now. Nevertheless, n_bytes function (idlneturl__define.pro, Line 49) is missing in GDL standard library. I used it from https://idlastro.gsfc.nasa.gov/ftp/pro/misc/n_bytes.pro

P.S. I can't fully test all idlneturl__define module now due to errors in the program I'm trying to run. :(

Upd.:
The problem was not in my program but in idlneturl__define.pro again.
I've changed quotes and moved forward.

Line 99 should be (curl under Windows accept only "-quotes):

if strlen((*self.headers)[i]) gt 0 then curl_cmd+='-H "'+strtrim((*self.headers)[i],2)+'" '

Line 178 should be (failed when output file name contained spaces):

curl_asyn_get_all='curl -L --progress-bar -o "'+filename+'" ' ; will use progressbar values, but alas need output in external file.

Line 209 should be (like in a previous case):

curl_syn='curl -L --silent --show-error -o "'+filename+'" '

P.P.S.
I have an error in FILE_CHMOD procedure now...

FILE_CHMOD is only available on Unix Operating Systems (help!)
Should I open a new issue?

[GillesDuvert]

@Albom yes open a new issue for FILE_CHMOD, please.
As FILE_CHMOD is a procedure in GDL (contrary to IDL) you probably can solve the FILE_CHMOD problem locally (even if it is just to do nothing), but we'll be interested if you provide a wrking version of FILE_CHMOD for Windows.

Besides, thanks for the tips above for a better idlneturl__define.pro

Concerning n_bytes, many gdl users have added idlastro to their !PATH, including the gdl maintainers, so this dependency passed unnoticed. The procedure should be modified as the use of n_bytes is not really needed there.

[Albom]

One more line (131) in idlneturl__define.pro should be changed to:

curl_asyn_get_headers+='--write-out "\n%{size_header}\n%{content_type}\n%{response_code}\n%{size_download}" '    ; we get some useful values

[GillesDuvert]

changes done. thanks!

[slayoo]

@GillesDuvert @Albom let's add a test so that this is actually ensured on our Windows CI builds!
@Albom, what would be a minimal reproducer for this issue? Thanks