Firmware Update

Overview

The OTA firmware update feature built into Mongoose Library uses binary images. Check here for more information, or here to see how to generate these files in your IDE.

This tutorial shows how to use Mongoose firmware update support functions in the context of a device dashboard. The UI runs on a web browser and interacts with the device via a RESTful API.

The device dashboard UI, including user authentication, is explained in its own tutorial. To build and try the example, please follow it. Selecting Firmware Update in the menu on the left, you'll access the corresponding screen for this tutorial

You basically upload a firmware file, then the device resets to boot from it.
The MQTT dashboard UI is explained in its own tutorial. To build and try the example, please follow it. It works in a pretty similar way to what we previously described.

API

bool mg_ota_begin(size_t new_firmware_size);     // Start writing
bool mg_ota_write(const void *buf, size_t len);  // Write chunk, aligned to 1k
bool mg_ota_end(void);                           // Stop writing

Example RESTful API

For this application, the RESTful server API in the Device Dashboard tutorial provides the following authenticated endpoints:

/api/firmware/upload - handles the process of uploading a file to the device and saving it as binary data

Each endpoint basically calls its respective backend API function for the OTA process. The update process itself requires three function calls: ota_begin(), ota_write() for each data chunk received, and ota_end() to finish the transaction.

Example RPC API

For this application, the RPC API in the MQTT Dashboard tutorial provides the following methods:

ota.upload - instructs the device to get and save new firmware

In this example, devices will publish their status, which includes firmware status

Compile options

The implementation is driven by the build option MG_OTA:

MG_OTA_NONE - No OTA support, uses a stub implementation that does nothing: src/ota_dummy.c
MG_OTA_CUSTOM - Custom implementation, roll your own set of mg_ota_* functions.
MG_OTA_<device name> - OTA for that specific device, see below.

Supported devices

Check src/ota.h for supported devices; e.g.:

Basic usage

The OTA API functions are usually called from:

a RESTful API, as used in the Device Dashboard tutorial:
an RPC API, as used in the MQTT Dashboard tutorial:

Firmare Image formats

There are a number of ways to deliver a firmware image. The OTA firmware update feature built into Mongoose Library uses binary images, here's a short description of the other image types and how to use your tools to get a binary image.

BIN

A binary file contains data for a block of correlative addresses, that is: a 1K file contains 1K bytes that belong to 1K contiguous positions in memory, starting at some address. This address is not specified in the file itself, so it must be known beforehand, usually based on the microcontroller or development board architecture and design. E.g.: flash memory in most STM32 microcontrollers starts at 0x08000000. It is the raw content to be placed in a microcontroller flash memory.

HEX

A hex file, or Intel HEX file, is a text file containing a hexadecimal representation of memory. Data is preceded by a record-type identificator and the memory address where the following data will be placed. This allows to specify non-contiguous blocks of memory, and, as the format is text-based, can be transferred in a lot of different ways; this format has been developed in the 70's and since then used by EPROM and flash programmers.

https://en.wikipedia.org/wiki/Intel_HEX

Integrated Development Environments (IDEs) usually do not generate hex files unless instructed to do so. If you are developing your own firmware, you are most likely to get the output in one of the other formats below. If you get some random hex file, each contiguous block in it can be extracted to a binary file, there are numerous hex2bin utilities around.

To generate hex output, see objcopy below for your IDE

There are other hex file formats, among them Motorola S records; these are usually variations on the same concept with different internal formats.

ELF

An ELF file contains object code and debugging information, they are also known as ELF/DWARF, in reference to the code and debug records. ELF files are usually the output of a linker after successful compilation. These files can include different memory blocks, and instructions to place them in memory.

https://en.wikipedia.org/wiki/Executable_and_Linkable_Format

In workstation operating systems, executable programs are usually one type of ELF file or equivalent, so the OS can load each block at its place in memory.

In microcontrollers, an ELF file can be opened by a debugger and/or a programmer. To work with ELF files, there are tools equivalent to those in the binutils package, most notably some form of readelf or objdump. To get a plain binary image out of an ELF file, use the equivalent of objcopy in your environment.

AXF

AXF files are a type of ELF file generated by some versions of ARM compilers; they are usually found in Keil and MCUXpresso projects, for example. Treat them as ELF files.

Getting the proper image format

The OTA firmware update feature built into Mongoose Library uses binary images. If you need to deliver an image to your customers, you'll usually also deliver a binary image, an ELF/AXF file, or a hex file to be loaded by a programming tool, depending on your vendor.

GCC

In workstations, you'll have the binutils tools:

objcopy -O binary firmware.elf firmware.bin
objcopy -O ihex firmware.elf firmware.hex

For microcontrollers, depending on your target, you'll have a flavor of the GNU C Compiler; for example the GNU Arm Embedded Toolchain:

arm-none-eabi-objcopy -O binary firmware.elf firmware.bin
arm-none-eabi-objcopy -O ihex firmware.elf firmware.hex

STM32CubeIDE

Open Project Properties -> C/C++ Build -> Settings, then select the Build steps tab and edit the Post-build steps field; there, select the proper command:

arm-none-eabi-objcopy -O binary ${ProjName}.elf ${ProjName}.bin

Should you need to [also] generate a hex file:

arm-none-eabi-objcopy -O ihex ${ProjName}.elf ${ProjName}.hex

MCUXpresso

Open Project Properties -> C/C++ Build -> Settings, then select the Build steps tab and go to the Post-build steps field; there, click on Edit and un-comment the proper command:

arm-none-eabi-objcopy -v -O binary "${BuildArtifactFileName}" "${BuildArtifactFileBaseName}.bin"

Should you need to [also] generate a hex file, add the following command:

arm-none-eabi-objcopy -v -O ihex "${BuildArtifactFileName}" "${BuildArtifactFileBaseName}.hex"

Pico-SDK

With Pico-SDK there is usually a step indicated in the CMake file to generate an UF2 file, that you will then copy to the disk that is mounted in your workstation when you plug the Pico with the Boot button pressed:

pico_add_extra_outputs(firmware)

The very same directory where your UF2 file is, will also contain BIN, ELF, and HEX files.

$ ls -1 build/firmware.*
build/firmware.bin
build/firmware.dis
build/firmware.elf
build/firmware.elf.map
build/firmware.hex
build/firmware.uf2

ESP-IDF

esptool.py generates three binary objects: the bootloader, the partition table, and the application binary. Use the application binary, Mongoose knows where to flash it.

If you happen to have a hex file, you can convert it to a binary file with esputil

VSCode

If your IDE is based on Visual Studio Code, you will likely be using a compiler provided by some extension, or one you installed yourself. Even you might be using a whole environment provided by extensions (Keil MDK6, PlatformIO, and others, with traditional vendors moving to it). For Arm targets, chances are the compiler is either GCC or Arm Compiler, so you will, manually or script-based, run a flavor of objcopy; check GCC above

PlatformIO IDE extension

This IDE will use the respective vendor SDK for your hardware. At the end of the build process, it can call an extra_script (Python code) where you can add your required post-build actions for your target. See PlatformIO docs for script file location and details.

For example, for a GCC-based environment, the script will call a flavor of objcopy for you.

Import("env")
...
# Custom BIN from ELF
env.AddPostAction(
  "$BUILD_DIR/${PROGNAME}.elf",
  env.VerboseAction(" ".join([
      "$OBJCOPY", "-O",	"binary",	"$TARGET", "$BUILD_DIR/${PROGNAME}.bin"
  ]), "Building $BUILD_DIR/${PROGNAME}.bin"))

In some platforms, you may need to add more arguments (-R) to get rid of some sections in the ELF file.

Keil uVision

Keil MDK includes a proprietary conversion tool: fromelf. It can be found with the other tools in ARM/ARMCC/bin under the installation directory.

fromelf --bin --output firmware.bin firmware.axf

You can also add a call to it as a post build step: open Options for Target, select the User tab, there, in Command Items -> After Build/Rebuild, check "Run #1"; enter in the User Command path:

fromelf.exe --bin --output=firmware.bin firmware.axf

If you are using a separate path for object files, then prepend your file names with that path.

Zephyr

The Zephyr Docker image builds using their west tool, based on ninja. The build directory under your project directory will contain both the ELF and BIN files.

Arduino IDE

If you enable verbose output during compilation:

You'll be able to see what is going on under the hood:

For Arm targets, chances are it is using GCC

The directory where your sketches are stored, will show both ELF and BIN files:

$ cd .cache/arduino/sketches/<hash> && ls -1 w5500-http.ino.*
w5500-http.ino.bin
w5500-http.ino.elf
w5500-http.ino.hex
w5500-http.ino.map

For the ESP32, it will use esptool.py t ogenerate the binary from the ELF file:

Linking everything together...
/home/username/.arduino15/.../xtensa-esp32-elf-g++ ... -o /home/username/.cache/arduino/sketches/hash/esp32-http.ino.elf
python3 /home/username/.arduino15/.../esptool.py --chip esp32 elf2image --flash_mode dio --flash_freq 80m --flash_size 4MB --elf-sha256-offset 0xb0 -o /home/username/.cache/arduino/sketches/hash/esp32-http.ino.bin /home/username/.cache/arduino/sketches/hash/esp32-http.ino.elf
esptool.py v4.6
Creating esp32 image...
Merged 27 ELF sections
Successfully created esp32 image.