fix: populate timezone data when formatting time

[marcin-serwin]

The jv2tm function was zeroing fields of struct tm that were not specified by the standard. However, depending on the libc this produced incorrect timezone data when used together with formatting functions. This change tries to fill the timezone data using either mktime, timegm, or manually.

fixes #2475; fixes #2429; fixes #1912.

Example:

TZ=Asia/Tokyo jq -rn '1731627341 | strflocaltime("%F %T %z %Z"), strftime("%F %T %z %Z")'

Before:

2024-11-15 08:35:41 +0000 JST
2024-11-14 23:35:41 +0000 JST

After:

2024-11-15 08:35:41 +0900 JST
2024-11-14 23:35:41 +0000 GMT

[itchyny]

I'm afraid the timezone offset is not fixed even with this patch.

# 1.7.1
 $ TZ=Asia/Tokyo jq -rn '1731627341 | strflocaltime("%F %T %z %Z"), strftime("%F %T %z %Z")'  
2024-11-15 08:35:41 +0900 JST
2024-11-14 23:35:41 +0900 JST

# marcin-serwin:strftime-fix
 $ TZ=Asia/Tokyo ./jq -rn '1731627341 | strflocaltime("%F %T %z %Z"), strftime("%F %T %z %Z")'
2024-11-15 08:35:41 +0900 JST
2024-11-14 23:35:41 +0900 UTC

# 1.7.1
 $ TZ=Europe/Paris jq -rn '1731627341 | strflocaltime("%F %T %z %Z"), strftime("%F %T %z %Z")'
2024-11-15 00:35:41 +0100 CET
2024-11-14 23:35:41 +0100 CET

# marcin-serwin:strftime-fix
 $ TZ=Europe/Paris ./jq -rn '1731627341 | strflocaltime("%F %T %z %Z"), strftime("%F %T %z %Z")'
2024-11-15 00:35:41 +0100 CET
2024-11-14 23:35:41 +0100 UTC

 $ uname -v
Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103

[marcin-serwin]

This issue is caused by a bug in the Apple's Libc implementation. Specifically in this line the __DARWIN_UNIX03 macro is negated which means that the strftime is ignoring the time offset data present in the tm struct in favor of the older heuristic needed by legacy standards. Looking at other uses of this macro, it seems to enable non-legacy behavior so the negation here is likely a bug since this means that the presence of this macro opts into the legacy behavior. The FreeBSD libc, from which the Apple's one is forked, does not have this issue.

I've added a workaround in the second commit that temporarily sets the global timezone to UTC if it is compiled for apple platforms.

[wader]

Nice digging 👍 jq can be used as a library does that make it a bad idea to use setenv? 🤔

Possible to add some tests for this?

[marcin-serwin]

jq can be used as a library does that make it a bad idea to use setenv? 🤔

setenv is already used in the my_mktime function for a similar purpose. It can cause issues in multi-threaded programs but I don't know how to avoid that.

Possible to add some tests for this?

I've added some tests in tests/shtest, is this the right place?

[marcin-serwin]

I've disabled the tests on windows. The ucrt on Windows always uses the global settings for determining the timezone, which is documented in a note near the end of the strftime docs. I tried to make them work similarly to Apple workaround by using setenv and _tzset but I couldn't make it work and I gave up trying to figure out why. Given that it's a documented behavior for Windows and the jq docs warn that strftime output depends on the operating system I think it's ok to leave it as is.

[wader]

jq can be used as a library does that make it a bad idea to use setenv? 🤔

setenv is already used in the my_mktime function for a similar purpose. It can cause issues in multi-threaded programs but I don't know how to avoid that.

I see 👍

Possible to add some tests for this?

I've added some tests in tests/shtest, is this the right place?

Is fine i think

About the apple libc bug: Will things break once/if apple fix it?

Otherwise i think this looks fine but would be great with some more eyes on it

[marcin-serwin]

About the apple libc bug: Will things break once/if apple fix it?

It will still work. With this bug fixed, the code will ignore the TZ variable and just look inside the struct for the timezone thus behaving like glibc.

[itchyny]

I think this is okay to merge, but sorry for shtest conflict.

[itchyny]

LGTM, thank you.

[itchyny]

@marcin-serwin
Oh, we have to copy the string (just like how my_mktime does), otherwise we end up failing to restore TZ.
See the following code.

#include <stdio.h>
#include <stdlib.h>

int main(void) {
  // Set to UTC and restore.
  char *tz = getenv("TZ");
  printf("%s\n", tz); // Etc/GMT+7
  setenv("TZ", "UTC", 1);
  printf("%s\n", tz); // Etc/GMT+7
  setenv("TZ", tz, 1);

  // Set to UTC and try to restore but fail.
  tz = getenv("TZ");
  printf("%s\n", tz); // Etc/GMT+7
  setenv("TZ", "UTC", 1);
  printf("%s\n", tz); // UTC (expected)
  setenv("TZ", tz, 1);

  tz = getenv("TZ");
  printf("%s\n", tz); // UTC
}

 $ TZ=Etc/GMT+7 ./a.out
Etc/GMT+7
Etc/GMT+7
Etc/GMT+7
UTC
UTC

This failure appears in the following interesting jq query (of course on macOS).

 $ TZ=Etc/GMT+7 ./jq -nc '1731627341 | .,. | [strftime("%FT%T"),strflocaltime("%FT%T%z")]'
["2024-11-14T23:35:41","2024-11-14T16:35:41-0700"]
["2024-11-14T23:35:41","2024-11-14T23:35:41+0000"]

[marcin-serwin]

You're right, I've opened #3264 with the fix.