view OTA update status

[lucho512]

Hi, how can I get the update status, in the program loop

I need to be able to disable interrupts since they block the update

Thanks!!!

[Hieromon]

There is no proper way to know that in the current AutoConnect version.
It will be enhanced with v1.3.0.

[lucho512]

oks, thanks Hieromon

[Hieromon]

@lucho512 I have staged the changes to enhance/v130 branch to help with your wishes, and You can start the evaluation. Also, PageBuilder v1.5.0 is required for evaluation, and I hope you will update the PageBuilder library from the development branch.

The sequence for checking OTA status is following:

1. Supported API summary

   • AutoConnectOTA* AutoConnect::getOTA(void)
     Returns a pointer of AutoConnectOTA instance. This function is valid after the AutoConnect::begin call and returns a pointer to AutoConnectOTA class for OTA due to set AC_OTA_BUILTIN with AutoConnectConfig::ota.

   • AutoConnectOTA::AC_OTAStatus_t AutoConnectOTA::status(void)
     Returns an enumerated value that represents the OTA status.

     typedef enum  {
       OTA_IDLE,               /**< Update process has not started */
       OTA_START,              /**< Update process has started */
       OTA_PROGRESS,           /**< Update process in progress */
       OTA_SUCCESS,            /**< A binary updater has uploaded fine */
       OTA_RIP,                /**< Ready for module restart */
       OTA_FAIL                /**< Failed to save binary updater by Update class */
     } AC_OTAStatus_t;

2. Typical API sequence

   AutoConnect portal;
   AutoConnectConfig config:
   ...
   config.ota = AC_OTA_BUILTIN;
   portal.begin();
   ...
   // In the interrupt handler
   if (!portal.getOTA()) {
     AutoConnectOTA* ota = portal.getOTA();
     if (ota->status() != AutoConnectOTA::OTA_PROGRESS) {
       // Here, something to handle the interrupt
     }
   }

Addition:

Once OTA is started, control does not return from AutoConnect::handleClient to the user sketch until the update is complete. The update process is blocked by the requestHandler of the WebServer class, so the start of the upload process cannot be detected in the usual way.
Therefore, the API shown above is a non-preemption approach and will have limited uses.

I have another specific proposals. The AutoConnectOTA class is a derivative of AutoConnectAUX, so it is itself a requestHandler for the WebServer class. Since the AutoConnectOTA class can detect the status change of the upload process by the WebServer class, it is possible to call back the function of the user sketch. It is expected to have the following specifications.

1. Subscribe to an event of OTA status change

AutoConnect::onOTAStatusChange(std::function<void(const AutoConnectOTA::AC_OTAStatus_t)>);

2. Declare Call-back function

void OTAStatusChanged(const AutoConnectOTA::AC_OTAStatus_t status) {
  if (status == AutoConnectOTA::OTA_START) {
      // Mask interruption
  }
  if (status == AutoConnectOTA::OTA_SUCCESS || status == AutoConnectOTA::OTA_FAIL) {
      // Unmask interruption
  }
}

3. Typical sequence

AutoConnect portal;
AutoConnectConfig config;
...
void OTAStatusChanged(const AutoConnectOTA::AC_OTAStatus_t status) {
  if (status == AutoConnectOTA::OTA_START) {
      // Mask interruption
  }
  if (status == AutoConnectOTA::OTA_SUCCESS || status == AutoConnectOTA::OTA_FAIL) {
      // Unmask interruption
  }
}

void setup() {
  config.ota = AC_OTA_BUILTIN;
  portal.config(config);
  portal.onOTAStatusChange(OTAStatusChanged);
  portal.begin();
}

void loop() {
  portal.handleClient();
}  

The API for this callback technique has not yet been implemented, but it may be a more effective approach than the AutoConnectOTA::getOTA mentioned above.
I look forward to your suggestions.

[lucho512]

Thank you very much Hieromon, I will try it ...

Congratulations on the excellent work

[Hieromon]

@lucho512 In addition to the getOTA function mentioned above, I have added a new API for callbacks that notify OTA status. You can evaluate it in the development branch as enhance/v130.

1. OTA status change notification callback interface

   typedef std::function<bool(const AC_OTAStatus_t)> OTAStatusChangeExit_ft;

   Parameter

   AC_OTAStatus_t: A value that indicates the OTA status. An enumeration with the following values:

   • AC_OTA_IDLE Update process has not started.
   • AC_OTA_START Update process has started.
   • AC_OTA_PROGRESS Update process in progress.
   • AC_OTA_SUCCESS A binary updater has uploaded fine.
   • AC_OTA_RIP Ready for module restart.
   • AC_OTA_FAIL Failed to save binary updater by Update class.

   Return

   • true: Continue OTA processing.
   • false: Cancel OTA processing.

2. Register the callback function to AutoConnect

   AutoConnect::onOTAStatusChange(OTAStatusChangeExit_ft fn)

3. Usage (Example)

   AutoConnect portal;
   AutoConnectConfig config;
   
       // OTA status change callback
   bool OTAStatus(const AC_OTAStatus_t status) {
     switch (status) {
     case AC_OTA_START:
       Serial.println("OTA started");
       break;
     case AC_OTA_PROGRESS:
       Serial.println("OTA in progress");
       break;
     case AC_OTA_SUCCESS:
       Serial.println("OTA ended");
       break;
     case AC_OTA_FAIL:
       Serial.println("OTA fails");
       break;
     default:
       Serial.printf("OTA status change: %d\n", (int)status);
       break;
     }
     return true;
   }
   
   void setup() {
     portal.onOTAStatusChange(OTAStatus);  // Register OTA status change callback
     config.ota = AC_OTA_BUILTIN;
     portal.config(config);
     portal.begin();
   }
   
   void loop() {
     portal.handleClient();
   }

If you have no objection to this specification, I will officially release it in AutoConnect v1.3.0.

[Hieromon]

@lucho512 I have changed the API for OTA status notifications. It is formed similar to the ArduinoOTA class and has the following interfaces:

  typedef std::function<void(void)>       OTAStartExit_ft;
  typedef std::function<void(void)>       OTAEndExit_ft;
  typedef std::function<void(uint8_t)>    OTAErrorExit_ft;
  typedef std::function<void(unsigned int, unsigned int)> OTAProgressExit_ft;

  void  AutoConnect::onOTAStart(OTAStartExit_ft fn);
  void  AutoConnect::onOTAEnd(OTAEndExit_ft fn);
  void  AutoConnect::onOTAError(OTAErrorExit_ft fn);
  void  AutoConnect::onOTAProgress(OTAProgressExit_ft fn);

Typical sequence

AutoConnet  portal;
AutoConnectConfig config;
...

void OTAStart() {
  Serial.println("Start updating");
}

void OTAEnd() {
  Serial.println("\nEnd");
}

void OTAProgess(unsigned int progress, unsigned int total) {
  Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
}

void OTAError(unsigned int error) {
  Serial.printf("Error[%u]: ", error);
}

void setup() {
  config.ota = AC_OTA_BUILTIN;
  portal.config(config);
  portal.onOTAStart(OTAStart);
  portal.onOTAEnd(OTAEnd);
  portal.onOTAProgress(OTAProgress);
  portal.onOTAError(OTAError);
  portal.begin();
}

void loop() {
  portal.handleClient();
}

You can evaluate new API with enhance/v130 branch. (Need to PageBuilder v1.5.0)

[HankLloydRight]

Is there a trick or other setting to get portal.onOTAProgress(OTAProgress); to work?

I have it set up exactly like you have in the example above, but the OTA... callbacks are never called during a firmware update.

Thank you.

[Hieromon]

@HankLloydRight The implementation of the exit routine call for OTA in AutoConnect V130 is not sufficient. Until v134, the exit routine is not called during the OTA update. It is a bug.

This problem is resolved in AutoConnect v135. I will release it soon, and you can evaluate it with the development branch as enhance/v135.

[HankLloydRight]

Hi @Hieromon,
You've been so helpful, and I've been anxiously waiting for v1.3.5.. andI hesitate posting this. But I still can't get the OTA update functions to work. I have the code setup exactly like you outlined above in #325 (comment) above.
I am specifying 1.3.5 in platform.io:

lib_deps =hieromon/[email protected]

Also, even more unfortunate, when I compile with 1.3.5, the OTA updates no longer work. At the very end of the update, I get an error message on the HTML update page:

Failed to update: 0

I've attached the DCORE_DEBUG_LEVEL=9 log below at the end of the update. You can see at line 32 (not sure if this is the problem).

[D][HTTPUpdate.cpp:339] handleUpdate(): Update ok
[D][WiFiClient.cpp:514] connected(): Disconnected: RES: 0, ERR: 128

I have to do an over-the-wire USB update back to v.1.3.4 to get OTA updates to work again. It's entirely the same code, just one version has
lib_deps =hieromon/[email protected]
and the other
lib_deps =hieromon/[email protected]

Thank you!

ac135.log

[Hieromon]

Hello @HankLloydRight ,

Failed to update: 0

Ahh, so the phenomenon has occurred...Sorry to disappoint you. I experienced this phenomenon too during the v1.3.5 test. But, the test location had a weak wireless signal (RSSI about 40%), and when we changed access points, it stopped the phenomenon. Then I stopped finding the cause.
But you've encountered the phenomenon too, and it's clear that v1.3.5 has a problem.

I begin a rigorous investigation. Thank you for notifying me. So I have one favor to ask: the arduino core release is 2.0.4, which is ahead, but the latest release of PlatformIO's framework-espressif32, 5.0.0, is still arduino core 2.0.3. The most significant difference between the two is that the revision of ESP-IDF has increased. So, can you verify the same situation where you encountered the problem using arduino core 2.0.4 in the Arduino IDE environment?

[HankLloydRight]

Hi @Hieromon,
I'm still using platform = [email protected], and I'm actually locked into that version since you so clearly explained in another post how 5.0.0 was totally refactored, and it's messing up my interrupt timings for my project. I've included below the output when compiling my project -I don't see the Arduino core version being used.
I hope this helps, let me know if anything else you might need.
Also, my Wifi is very strong here, I'm right next to the transmitter base (Google Nest mesh network).
-Hank

Processing lolin32 (platform: [email protected]; board: lolin_d32; framework: arduino)
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin_d32.html
PLATFORM: Espressif 32 (3.5.0) > WEMOS LOLIN D32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:

• framework-arduinoespressif32 @ 3.10006.210326 (1.0.6)
• tool-esptoolpy @ 1.30100.210531 (3.1.0)
• tool-mkspiffs @ 2.230.0 (2.30)
• toolchain-xtensa32 @ 2.50200.97 (5.2.0)

[Hieromon]

@HankLloydRight Thanks for the detailed log.
I could probably reach the cause. AutoConnect has been degraded.

The update process in v1.3.4 was redundant and I thought I had optimized it for the v1.3.5 enhancement. At this time, I had forgotten one thing: the ESP32 TCP stack sometimes behaves strangely. It may be escaped from the event loop with the packet still in the buffer when an incoming interrupt occurs. In that case, the HTTP request handler behaves as if it received empty content. Then, AutoConnect will assume that the update data was interrupted and immediately begin error handling. It was a mistake.

I staged a patch to patches/v136 branch. This fix is common to AutoConnectOTA and AutoConnectUpdate.

Perhaps you see the consequences of this fix and will feel that the update has slowed down. But it is normal behavior. I would like you to evaluate this fix. In that case, both AutoConnectOTA and AutoConnectUpdate use cases will be more effective for verification. If the results are good, I will release it as v1.3.6.

Thank you for your contribution.

[HankLloydRight]

I’m happy to test the v136 patch, but could you please let me know how to tell Platform.io how to select that version? Thank you.

[Hieromon]

@HankLloydRight

Platform.io how to select that version?

The following lib_deps directive should get it directly from the GitHub branch... maybe.

lib_deps = https://github.com/{OWNER}/{REPOSITORY}.git#{BRANCH}

platform = [email protected]
framework = arduino
board = lolin_d32
lib_extra_dirs = ~/Documents/Arduino/libraries
lib_deps = https://github.com/Hieromon/AutoConnect.git#patches/v136
lib_ldf_mode = chain+
build_flags =
    -DAC_DEBUG
    -DAC_USE_SPIFFS
    -DPB_USE_SPIFFS
    -DCORE_DEBUG_LEVEL=4
board_build.filesystem = spiffs
board_build.partitions = min_spiffs.csv
upload_port = COMn
monitor_port = COMn
upload_speed = 921600
monitor_speed = 115200
monitor_filters = esp32_exception_decoder, colorize

[HankLloydRight]

lib_deps = https://github.com/{OWNER}/{REPOSITORY}.git#{BRANCH}
lib_deps = https://github.com/Hieromon/AutoConnect.git#patches/v136

That worked, thank you.

And now the OTA firmware updates also are working again with v1.3.6

I still can not get the OTA update/processing functions to work, but that's a super minor, non-critical, issue.

Thank you.

[Hieromon]

@HankLloydRight Thank you for testing.

I still can not get the OTA update/processing functions to work

Which one still does not work AutoConnectOTA or AutoConnectUpdate?
You can identify a type of OTA that AutoConnectConfig::ota specifies AutoConnectOTA, and the AutoConnectUpdate class is used with the Update server.

[HankLloydRight]

I'm not sure how to answer your question. I'm using the code exactly as posted in #325 (comment)

[Hieromon]

@HankLloydRight, Roger that. My hasty mistake. I assumed you were using the Update Server together.
The OTA exit seems to be triggering fine, so I will release v1.3.6.
Thank you for your cooperation.

[HankLloydRight]

By "update server" I assume you mean the python script? Yes I am totally using that!

edit: Ok, totally my misunderstanding. I am using AutoConnectUpdate update() and not AutoConnectOTA

I now understand the difference between the two. And now see the code sample above is only for AutoConnectOTA and not AutoConnectUpdate. I'm very sorry.

Is there a way to get this same function, but with AutoConnectUpdate instead?

void UpdateProgess(unsigned int progress, unsigned int total) {
  Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
}

[Hieromon]

@HankLloydRight

Is there a way to get this same function, but with AutoConnectUpdate instead?

Well...I just checked the AutoConnect documentation and it doesn't mention the description of the AutoConnectUpdate exit routine. Oops.

AutoConnect portal(server);
AutoConnectUpdate update;

void exitOTAStart() {
  Serial.println("OTA started");
}

void exitOTAProgress(unsigned int amount, unsigned int sz) {
  Serial.printf("OTA in progress: received %d bytes, total %d bytes\n", sz, amount);
}

void exitOTAEnd() {
  Serial.println("OTA ended");
}

void exitOTAError(uint8_t err) {
  Serial.printf("OTA error occurred %d\n", err);
}

void setup() {
  Serial.begin(115200);
  
  // Register OTA exit routines
  update.onOTAStart(exitOTAStart);
  update.onOTAEnd(exitOTAEnd);
  update.onOTAProgress(exitOTAProgress);
  update.onOTAError(exitOTAError);

  update.attach(portal);
  portal.begin();
}

void loop() {
}

NOTES
Some features of AutoConnectUpdate are missing in the above method. That is, the progress bar on the web page is not updated during updates. However, the OTA process proceeds normally.
This is because AutoConnectUpdate itself uses the OTA exit to update the progress bar, which conflicts with the onProgress exit in the user sketch.
This limitation could be improved by devising a better implementation of AutoConnectUpdate. Should I do it?

[HankLloydRight]

As long as I can use
exitOTAStart And exitOTAEnd, that should address my needs to change the OLED screen with a message when doing an http firmware update.

But if you do decide to make the exitOTAProgress function usable without disabling the progress bar on the web page, that would be icing on the cake, but not necessary.

Thank you.

[HankLloydRight]

 // Register OTA exit routines
  update.onOTAStart(exitOTAStart);
  update.onOTAEnd(exitOTAEnd);
  update.onOTAProgress(exitOTAProgress);
  update.onOTAError(exitOTAError);

Hi @Hieromon,
I tried to use the code you posted, but those functions do not appear to be available. I searched the source code for them, but came up empty. I'm still using lib_deps = https://github.com/Hieromon/AutoConnect.git#patches/v136
Thank you.

[Hieromon]

Oh, sorry, @HankLloydRight. It slipped my mind.
OTA exit for AutoConnectUpdate was enabled starting with ESP32 Arduino Core 2.0.0. espressif/arduino-esp32#5408

The AutoConnectUpdate class was implemented by inheriting the HTTPUpdate class from the ESP32 Arduino core; it is HTTPUpdate library that actually calls back the OTA exit routine. AutoConnectUpdate does not trigger a callback.

AutoConnectUpdate can support its own exit routine for users like you who stay on ESP Core 1.0.6. But conversely, it is an unintended overridden function for users who have already migrated to ESP Core 2.0.0. It is possible to use the C++ SNIFAE mechanism to determine if the HTTPUpdate class owns members for the OTA exit function and suppress the override by the AutoConnectUpdate class at compile time. This would be my work.

Another way is for you to modify the HTTPUpdate library in ESP32 Arduino Core 1.0.6 to implement the equivalent of the callback on OTA exit that the HTTPUpdate class in ESP32 2.0.0 or later has. The implementation of the HTTPUpdate class is not too difficult and I am sure you are skilled enough to make it work.
You can go to the PR that supported OTA exit for HTTPUpdate class and see the modified diff. The required changes are HTTPUpdate.h and HTTPUpdate.cpp. You can ignore changes to other files. The Expand all icon can show the whole code with differences.

Of course, you could clone and modify the espressif/ardiuno-esp32 repository, but I would not recommend it; it is more reliable to modify the package source carried by PlatformIO directly. They are usually located on the following path:
C:\Users\{USER}\.platformio\packages\[email protected]\libraries\HTTPUpdate\src

[HankLloydRight]

Ok, I now see the issue, and yeah, I can't upgrade to Arduino core 2.0.0.

I'll dive into this to modify the code when I want a break from "real work" LOL. I'd like to get it working, but as I mentioned, it's not critical.

All the best.

[Hieromon]

@HankLloydRight I found one mistake too.
The APIs for registering exit routines are slightly different between AutoConnectOTA and AutoConnectUpdate; for AutoConnectUpdate it is onStart, onEnd, onProgress, onError.

update.onOTAStart(exitOTAStart);
update.onOTAEnd(exitOTAEnd);
update.onOTAProgress(exitOTAProgress);
update.onOTAError(exitOTAError);