Wi-Fi network with SSID containing underscore or hyphen cannot be connected

[cima]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

Wi-Fi accesspoint with SSID containing hyphen "-" or underscore "_" can be connected by ESP32 in a station mode. E.g. There is a router with network named "my-pizza" or "great_pizza" and esp microcontroller can connect it in station mode.

Actual behavior (suspected bug)

The ESP always gets kicked out with reason 201 (AP not found) despite network is present, accesible by other devices such as Cell phones, laptop or even ESP8266 devices.

Error logs or terminal output

I (18140)
: Setup to connect to AP SSID:cima-gopro password:************
D (18150) nvs: nvs_get opmode 1
D (18150) wifi:clear blacklist
W (18150) wifi:Password length matches WPA2 standards, authmode threshold changes from OPEN to WPA2
D (18160) wifi:filter: set rx policy=0
I (18170) wifi:mode : sta (94:b5:55:25:64:08)
I (18170) wifi:enable tsf
D (18170) wifi:filter: set rx policy=1
D (18180) wifi:connect status 0 -> 0
I (18180)
: Wifi station started.
I (18180)
: Retrying to connect to the AP
D (18190) event: running post WIFI_EVENT:3 with handler 0x400e9860 and context 0x3ffca908 on loop 0x3ffc12fc
--- 0x400e9860: cima::system::network::WifiManager::wifiEventHandlerWrapper(void*, char const*, long, void*) at C:/src/cima/cima-esp32-gopro/components/cima-system/network/WifiManager.cpp:142

D (18200) event: running post WIFI_EVENT:2 with handler 0x400e9860 and context 0x3ffca908 on loop 0x3ffc12fc
--- 0x400e9860: cima::system::network::WifiManager::wifiEventHandlerWrapper(void*, char const*, long, void*) at C:/src/cima/cima-esp32-gopro/components/cima-system/network/WifiManager.cpp:142

D (18210) wifi:Start wifi connect
D (18210) wifi:connect status 0 -> 0
D (18210) wifi:connect chan=0
D (18220) wifi:first chan=11
D (18220) wifi:connect status 0 -> 1
D (18220) wifi:filter: set rx policy=3
D (18230) wifi:clear scan ap list
D (18230) wifi:start scan: type=0x50f, priority=2, cb=0x4011c9b0, arg=0x0, ss_state=0x1, time=235434, index=0
--- 0x4011c9b0: cnx_start_handoff_cb at ??:?

D (18240) wifi:perform scan: ss_state=0x9, chan<11,0>, dur<0,120>
I (18250)
: Connection attempt 1 ended with code: 0
D (18250)
: Network is down - exiting status request
D (18260)
: Network is down - not stopping recording
D (18370) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18370) wifi:perform scan: ss_state=0x9, chan<1,0>, dur<0,120>
D (18490) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18490) wifi:perform scan: ss_state=0x9, chan<2,0>, dur<0,120>
D (18610) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18610) wifi:perform scan: ss_state=0x9, chan<3,0>, dur<0,120>
D (18730) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18730) wifi:perform scan: ss_state=0x9, chan<4,0>, dur<0,120>
D (18850) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18860) wifi:perform scan: ss_state=0x9, chan<5,0>, dur<0,120>
D (18980) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (18980) wifi:perform scan: ss_state=0x9, chan<6,0>, dur<0,120>
D (19100) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19100) wifi:perform scan: ss_state=0x9, chan<7,0>, dur<0,120>
D (19220) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19220) wifi:perform scan: ss_state=0x9, chan<8,0>, dur<0,120>
D (19260) 
: Network is down - exiting status request
D (19260) 
: Network is down - not stopping recording
D (19340) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19340) wifi:perform scan: ss_state=0x9, chan<9,0>, dur<0,120>
D (19470) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19470) wifi:perform scan: ss_state=0x9, chan<10,0>, dur<0,120>
D (19590) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19590) wifi:perform scan: ss_state=0x9, chan<12,0>, dur<360,360>
D (19950) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (19950) wifi:perform scan: ss_state=0x9, chan<13,0>, dur<360,360>
D (20270) 
: Network is down - exiting status request
D (20270)
: Network is down - not stopping recording
D (20310) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (20310) wifi:perform scan: ss_state=0x9, chan<14,0>, dur<360,360>
D (20670) wifi:scan end: arg=0x3ffb7218, status=0, ss_state=0x3
D (20670) wifi:filter: set rx policy=4
D (20670) wifi:first chan=1
D (20670) wifi:handoff_cb: status=0
D (20680) wifi:clear rc list
D (20680) wifi:clear blacklist
D (20680) wifi:Send disconnect event, reason=201
D (20690) wifi:connect status 1 -> 3
D (20690) wifi:disable connect timer
D (20690) wifi:clear scan ap list
D (20700) event: running post WIFI_EVENT:5 with handler 0x400e9860 and context 0x3ffca908 on loop 0x3ffc12fc
--- 0x400e9860: cima::system::network::WifiManager::wifiEventHandlerWrapper(void*, char const*, long, void*) at C:/src/cima/cima-esp32-gopro/components/cima-system/network/WifiManager.cpp:142

D (20710) wifi:Start wifi connect
D (20710) wifi:connect status 3 -> 0
D (20710) wifi:connect chan=0
D (20720) wifi:first chan=11
D (20720) wifi:connect status 0 -> 1
D (20720) wifi:filter: set rx policy=3
D (20730) wifi:clear scan ap list
D (20730) wifi:start scan: type=0x50f, priority=2, cb=0x4011c9b0, arg=0x0, ss_state=0x1, time=2734639, index=0
--- 0x4011c9b0: cnx_start_handoff_cb at ??:?

D (20740) wifi:perform scan: ss_state=0x9, chan<11,0>, dur<0,120>
I (20750) 
: Connection attempt 1 ended with code: 0
I (20750)
: Kick reason: 201
I (20750)
: Network cima-gopro not found. Trying next one.

Steps to reproduce the behavior

Steps

1. Have a wi-fi access point with 2.4 GHz network which SSID contain hyphen "-" or underscore "_"
2. Have a basic esp-idf project that connects to wi-fi (station)
3. Configure the project to connect to AP from step 1
4. Compile & upload the code to ESP32 (not ESP8266)

Alternative No-code steps

1. Have a wi-fi access point with 2.4 GHz network which SSID contains hyphen "-" or underscore "_"
2. Have ESP32 dev board (not ESP8266)
3. Connect it to PC via USB cable (assuming UART drivers are installed)
4. Use browser based flashing from https://esphome.io/projects/
5. Select Empty ESPHome device ->ESP32
6. Click connect
7. Select COM port
8. Install basic ESP home firmware
9. Once done select connect to wifi
10. Choose the Wi-fi from step 1
11. Type password

Project release version

release/v5.4, commit hash 206fe3e

System architecture

ESP-WROOM-32

Operating system

ESP32, ESP-IDF 5.4

Operating system version

esp-idf 5.4

Shell

CMD

Additional context

I have recently updated my project https://github.com/cima/cima-esp-gopro from esp-idf 4.4 (my branch develop) to 5.4 (my branch update_to_esp-idf-5.4) and it can no longer connect to Go Pro's wi-fi which I named "cima-gopro" (contains hyphen). It worked earlier.

I have other ESPHome projects that connects to my home wi-fi "atSomeone_WLAN" (contains underscore) I compiled for ESP8266 and those work fine. I have created new one for ESP32 that works with wi-fis without underscore or hypen in SSID and fail to connect to my home wi-fi with underscore.

There are complains around the internet that those two characters cause problems in seemingly unrelated projects

• Esphome : ESP32 unable to connect wifi
• No connection with underscore in SSID

I spent quite some time browsing through esp-idf source code to spot how SSID is being procesed but the only suspicious place is function esp_err_t esp_wifi_set_config(wifi_interface_t interface, wifi_config_t *conf); from esp_wifi.h that can only be found in precompiled libraries of libnet_80211.a and libmesh.a

[cima]

Dump of one of the pages of the partition where SSID and other stuff are being written upon each connection.

Page no. 29, Status: Full, Version: 2, CRC32: fcff957f, Page address: 0x3000
 Entry state bitmap: 00 00 00 00 aa aa a2 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0
 000. Erased , Namespace Index: 002, Type: blob_data , Span: 002, Chunk Index: 128, CRC32: 6459b59d | sta.apsw: Size=2, CRC32=82880bba
      0x000  05 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ..ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 002. Erased , Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: 7fa29633 | sta.apsw: Size=2, ChunkCount=1, ChunkStart=128
 003. Erased , Namespace Index: 002, Type: blob_data , Span: 003, Chunk Index: 128, CRC32: a5cfea91 | sta.ssid: Size=36, CRC32=848812b9
      0x000  06 00 00 00 47 69 7a 6d 6f 73 00 70 72 6f 00 00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ....Gizmos.pro.. ................
      0x020  00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ....ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 006. Erased , Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: b108fd8c | sta.ssid: Size=36, ChunkCount=1, ChunkStart=128
 007. Erased , Namespace Index: 002, Type: blob_data , Span: 004, Chunk Index: 128, CRC32: c2cbfe09 | sta.pswd: Size=65, CRC32=c2f23193
      0x000  XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  **************** ................
      0x020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................ ................
      0x040  00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  .ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 011. Erased , Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: b1245008 | sta.pswd: Size=65, ChunkCount=1, ChunkStart=128
 012. Erased , Namespace Index: 002, Type: blob_data , Span: 002, Chunk Index: 000, CRC32: f413ad9e | sta.apsw: Size=2, CRC32=f58f3b2c
      0x000  05 01 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ..ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 014. Erased , Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: 9e978db3 | sta.apsw: Size=2, ChunkCount=1, ChunkStart=0
 015. Erased , Namespace Index: 002, Type: uint8_t   , Span: 001, Chunk Index: 255, CRC32: f7bfe5bf | sta.chan: 1
 016. Written, Namespace Index: 002, Type: blob_data , Span: 003, Chunk Index: 000, CRC32:  2ae4ec9 | sta.ssid: Size=36, CRC32=2cf8f2d3
      0x000  0a 00 00 00 63 69 6d 61 2d 67 6f 70 72 6f 00 00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ....cima-gopro.. ................
      0x020  00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ....ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 019. Written, Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: 503de60c | sta.ssid: Size=36, ChunkCount=1, ChunkStart=0
 020. Written, Namespace Index: 002, Type: blob_data , Span: 004, Chunk Index: 000, CRC32: d6c7b327 | sta.pswd: Size=65, CRC32=30ae8cd1
      0x000  XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX 00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ***************. ................
      0x020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................ ................
      0x040  00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  .ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 024. Written, Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: 50114b88 | sta.pswd: Size=65, ChunkCount=1, ChunkStart=0
 025. Erased , Namespace Index: 002, Type: uint8_t   , Span: 001, Chunk Index: 255, CRC32: 3b15e521 | sta.chan: 0
 026. Written, Namespace Index: 002, Type: blob_data , Span: 002, Chunk Index: 128, CRC32: 6459b59d | sta.apsw: Size=2, CRC32=82880bba
      0x000  05 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ..ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 028. Written, Namespace Index: 002, Type: blob_index, Span: 001, Chunk Index: 255, CRC32: 7fa29633 | sta.apsw: Size=2, ChunkCount=1, ChunkStart=128

The slight padding of SSID by 4 bytes seems to be an integer value of string length. It is unexpected that password uses different style of string serialization. Overall it seems not malformed. If so the problem must be in runtime code when handling the ssid.