OTA from ESPHome and improv wifi credentials

[PaulWieland]

Starting Jan 1, all ratgdo32 control boards will be shipped with a skeleton ESPHome firmware to make setup and configuration easier. The firmware has esp32_improv over BLE and captive portal enabled. The Improv Web Serial process should be unaffected.

The skeleton looks like this:

More testing and instructions for how to onboard HK firmware users from this skeleton is required.

Uploading firmware/homekit-ratgdo32-v3.0.5.firmware.bin seems to boot successfully but the previous configured Improv WiFi credentials were lost and it went back into AP mode.

[dkerr64]

I agree that it makes sense for you to ship with firmware already installed.

There are a lot of things to worry about when it comes to uploading a different firmware. Most important is that the partitions on the flash are the same... an OTA update only updates (one of) the OTA partitions... they alternate so that if a flash fails there is a good one to revert to. The other partitions are not touched.

A flash over USB / serial will update the other partitions, and/or erase one or more partitions. If the partition layout changes between firmware then an OTA update is unwise. Here is the HomeKit partitions.csv file

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1F0000,
app1,     app,  ota_1,   0x200000,0x1F0000,
coredump, data, coredump,0x3F0000,0x10000,

This layout was selected to maximize space for the application (firmware.bin). I do not know what the partition layout is on ESPhome firmware, but an OTA update will only write into whatever partitions it has for applications. Partition layout can not be changed by OTA, only by a USB serial flash.

The HomeKit firmware saves persistent data in the NVS partition (nvram), including WiFi credentials. We use HomeSpan and it is handling the WiFi connection for us... it saves credentials to NVS rather than lean on the Arduino WiFi class to save the credentials (which is what we did in the ESP8266 version).

HomeSpan uses multiple namespaces in NVS, I have not looked into what they all are. HomeKit for ratgdo uses "ratgdo" namespace... when I selected that, I had not considered possibility of having ESPhome data also in the NVS partition. Obviously it is critical to ensure no use of the same namespace between firmwares.

There is also limited space in NVS... my partition table allows for only 20KB, the maximum possible size given my allocation for the application partitions... unless I get rid of the coredump partition.

I'd really like to allow for OTA update to switch from ESPhome to HomeKit. At a minimum we need.

1. Identical partition table
2. Coordinate use of NVS storage/namespaces.

[dkerr64]

I just noticed that HomeSpan have published a new release, in which they have made a number of changes to WiFi connection. It may be possible for me to override HomeSpan's default WiFi.begin() such that we can save and use credentials same way as ESPhome (which I am assuming uses Arduino's built in credential store?).

We still need same partition table and to coordinate NVS namespace use.

[PaulWieland]

which I am assuming uses Arduino's built in credential store?

I wouldn't assume that since you can also build with ESP-IDF.

When i get a chance I'll try and see if I can figure out where the credentials are stored.

In the meantime, the wifi component.h is here: https://github.com/esphome/esphome/blob/dev/esphome/components/wifi/wifi_component.h

And the improv serial component might be worth checking too:
https://github.com/esphome/esphome/blob/dev/esphome/components/improv_serial/improv_serial_component.h

[dkerr64]

Yup... ESPhome has its own preferences methods. WiFi credentials are saved here which then has a key added here. But unfortunately the whole ESPhome preferences code relies on using filesystem, setup here.

I removed the filesystem partition to allow me to grow the application partition to maximum possible size. You and I would have to coordinate a way to hold SSID/password outside of both ESPhome and HomeSpan if we want to allow setting credentials on one to work in the other.

Personally, I don't think it is worth it. It's a one-time setup step.

But it also highlights the problem we will have with partition table... ESPhome must have set aside a partition for a filesystem. If I adopt the same partition table for HomeKit, then the available space for application binary will be reduced... which I am nervous about doing.